2024-04-26T23:37:24.301 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T23:37:24.301 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} --verbose --archive /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327 --name rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi --owner scheduled_rishabh@teuthology -- /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327/orig.config.yaml 2024-04-26T23:37:24.359 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T23:37:24.449 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327 branch: wip-rishabh-testing-20240426.111959 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} email: null first_in_suite: false job_id: '7675327' kernel: client: branch: testing flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240426.111959 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt dry_run: true install: ceph: flavor: default sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240426.111959 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 816 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 sleep_before_teardown: 0 subset: 166/512 suite: fs suite_branch: wip-rishabh-testing-20240426.111959 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwSeJ17OMvP14ak7Z8vDvzpYZZ5APsroIk596/OJ+6gCqVuzhHhFuEKIlZWxTnRb4t2zljNy4cG/6ZYqRZlzI8= smithi080.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGG4qydJcaDfLB1hJL+lnmNljOis90+30mSwEmpNjpHXURuJIVwWBXmlPfRTjCk1VpTZnC1aceb6zc0HC08hQ6M= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLlUMjdQkK+WFIUS6/T926oCkzIXL0fltx68SaxLU1RFQ7tnwTG1oGF5I3WTGLNbzO2iw4+Bg5sKhiwMGPuX5+U= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/test_o_trunc.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_19:30:57 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T23:37:24.449 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa; will attempt to use it 2024-04-26T23:37:24.450 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks 2024-04-26T23:37:24.450 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T23:37:24.452 INFO:teuthology.task.internal:Checking packages... 2024-04-26T23:37:24.475 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc612ffb9c1d5bc27536bc917a705262a36d4387' 2024-04-26T23:37:24.475 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T23:37:24.475 INFO:teuthology.packaging:ref: None 2024-04-26T23:37:24.475 INFO:teuthology.packaging:tag: None 2024-04-26T23:37:24.475 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240426.111959 2024-04-26T23:37:24.475 INFO:teuthology.packaging:sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:37:24.475 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240426.111959 2024-04-26T23:37:24.694 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3306.gbc612ffb 2024-04-26T23:37:24.695 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T23:37:24.702 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T23:37:24.702 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T23:37:24.712 INFO:teuthology.task.internal:Saving configuration 2024-04-26T23:37:24.733 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T23:37:24.738 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T23:37:24.763 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 23:31:53.364035', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwSeJ17OMvP14ak7Z8vDvzpYZZ5APsroIk596/OJ+6gCqVuzhHhFuEKIlZWxTnRb4t2zljNy4cG/6ZYqRZlzI8='} 2024-04-26T23:37:24.784 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 23:31:53.367297', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGG4qydJcaDfLB1hJL+lnmNljOis90+30mSwEmpNjpHXURuJIVwWBXmlPfRTjCk1VpTZnC1aceb6zc0HC08hQ6M='} 2024-04-26T23:37:24.808 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 23:31:53.365865', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLlUMjdQkK+WFIUS6/T926oCkzIXL0fltx68SaxLU1RFQ7tnwTG1oGF5I3WTGLNbzO2iw4+Bg5sKhiwMGPuX5+U='} 2024-04-26T23:37:24.808 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T23:37:24.814 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-26T23:37:24.814 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-26T23:37:24.814 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-26T23:37:24.815 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T23:37:24.906 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7ebc06a160>, signals=[15]) 2024-04-26T23:37:24.906 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T23:37:24.911 INFO:teuthology.task.internal:Opening connections... 2024-04-26T23:37:24.911 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:37:24.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:37:24.991 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi080.front.sepia.ceph.com 2024-04-26T23:37:24.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:37:25.067 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-04-26T23:37:25.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:37:25.140 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T23:37:25.147 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-04-26T23:37:25.165 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-04-26T23:37:25.165 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-04-26T23:37:25.224 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-04-26T23:37:25.224 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2024-04-26T23:37:25.224 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-04-26T23:37:25.224 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-04-26T23:37:25.224 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2024-04-26T23:37:25.224 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2024-04-26T23:37:25.225 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T23:37:25.225 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-04-26T23:37:25.225 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2024-04-26T23:37:25.225 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T23:37:25.225 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-04-26T23:37:25.225 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T23:37:25.225 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T23:37:25.225 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T23:37:25.226 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-04-26T23:37:25.250 DEBUG:teuthology.orchestra.run.smithi080:> uname -m 2024-04-26T23:37:25.270 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64 2024-04-26T23:37:25.270 DEBUG:teuthology.orchestra.run.smithi080:> cat /etc/os-release 2024-04-26T23:37:25.327 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="CentOS Stream" 2024-04-26T23:37:25.327 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="9" 2024-04-26T23:37:25.327 INFO:teuthology.orchestra.run.smithi080.stdout:ID="centos" 2024-04-26T23:37:25.327 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE="rhel fedora" 2024-04-26T23:37:25.327 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="9" 2024-04-26T23:37:25.328 INFO:teuthology.orchestra.run.smithi080.stdout:PLATFORM_ID="platform:el9" 2024-04-26T23:37:25.328 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T23:37:25.328 INFO:teuthology.orchestra.run.smithi080.stdout:ANSI_COLOR="0;31" 2024-04-26T23:37:25.328 INFO:teuthology.orchestra.run.smithi080.stdout:LOGO="fedora-logo-icon" 2024-04-26T23:37:25.328 INFO:teuthology.orchestra.run.smithi080.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T23:37:25.328 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://centos.org/" 2024-04-26T23:37:25.328 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T23:37:25.328 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T23:37:25.328 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T23:37:25.328 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2024-04-26T23:37:25.472 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-04-26T23:37:25.490 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-04-26T23:37:25.490 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-04-26T23:37:25.547 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-04-26T23:37:25.547 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-04-26T23:37:25.547 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-04-26T23:37:25.547 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-04-26T23:37:25.547 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-04-26T23:37:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-04-26T23:37:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T23:37:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-04-26T23:37:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-04-26T23:37:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T23:37:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-04-26T23:37:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T23:37:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T23:37:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T23:37:25.548 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-04-26T23:37:25.575 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T23:37:25.585 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T23:37:25.591 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T23:37:25.592 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-04-26T23:37:25.594 DEBUG:teuthology.orchestra.run.smithi080:> test '!' -e /home/ubuntu/cephtest 2024-04-26T23:37:25.596 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-04-26T23:37:25.612 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T23:37:25.618 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T23:37:25.618 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-04-26T23:37:25.652 DEBUG:teuthology.orchestra.run.smithi080:> test -z $(ls -A /var/lib/ceph) 2024-04-26T23:37:25.654 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-04-26T23:37:25.742 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T23:37:25.819 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T23:37:25.833 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-26T23:37:25.833 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-26T23:37:25.833 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-26T23:37:25.834 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T23:37:25.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:37:25.851 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-26T23:37:26.091 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T23:37:26.091 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-26T23:37:26.091 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-04-26T23:37:26.111 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-437.el9.x86_64 2024-04-26T23:37:26.111 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T23:37:26.111 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-26T23:37:26.112 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum upgrade -y linux-firmware 2024-04-26T23:37:28.468 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-26T23:37:47.968 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 19 MB 00:18 2024-04-26T23:37:52.128 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-26T23:37:53.594 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 16 kB 00:00 2024-04-26T23:37:55.978 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 21 MB 00:02 2024-04-26T23:38:01.228 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-26T23:38:03.290 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-26T23:38:03.291 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-04-26T23:38:03.291 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-26T23:38:03.391 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T23:38:03.391 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T23:38:03.531 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-26T23:38:03.531 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-26T23:38:03.534 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi040.front.sepia.ceph.com to complete... 2024-04-26T23:38:15.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26 23:38:15 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-26T23:38:15.870 INFO:teuthology.orchestra.run.smithi040.stderr:FINISHED --2024-04-26 23:38:15-- 2024-04-26T23:38:15.871 INFO:teuthology.orchestra.run.smithi040.stderr:Total wall clock time: 12s 2024-04-26T23:38:15.871 INFO:teuthology.orchestra.run.smithi040.stderr:Downloaded: 1 files, 341M in 12s (28.5 MB/s) 2024-04-26T23:38:15.873 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T23:38:15.873 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-26T23:38:18.072 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying... ######################################## 2024-04-26T23:38:18.209 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing... ######################################## 2024-04-26T23:38:18.336 INFO:teuthology.orchestra.run.smithi040.stdout:Updating / installing... 2024-04-26T23:38:26.607 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-26T23:39:06.852 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-26T23:39:06.853 DEBUG:teuthology.orchestra.run.smithi040:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-26T23:39:06.934 INFO:teuthology.orchestra.run.smithi040.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-26T23:39:06.934 INFO:teuthology.orchestra.run.smithi040.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-26T23:39:06.934 INFO:teuthology.orchestra.run.smithi040.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-26T23:39:06.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-26T23:39:06.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-26T23:39:06.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-26T23:39:06.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-26T23:39:06.934 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-26T23:39:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-26T23:39:06.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-26T23:39:06.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-26T23:39:06.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-26T23:39:06.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-26T23:39:06.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-26T23:39:06.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-26T23:39:06.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-26T23:39:06.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-26T23:39:06.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-26T23:39:06.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-26T23:39:06.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-26T23:39:06.936 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-26T23:39:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-26T23:39:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-26T23:39:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-26T23:39:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-26T23:39:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-26T23:39:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-26T23:39:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-26T23:39:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-26T23:39:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-26T23:39:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-26T23:39:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-26T23:39:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-26T23:39:06.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-26T23:39:06.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-26T23:39:06.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-26T23:39:06.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-26T23:39:06.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-26T23:39:06.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-26T23:39:06.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-26T23:39:06.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-26T23:39:06.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-26T23:39:06.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-26T23:39:06.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-26T23:39:06.938 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-26T23:39:06.939 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-26T23:39:06.940 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-26T23:39:06.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-26T23:39:06.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-26T23:39:06.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-26T23:39:06.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-26T23:39:06.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-26T23:39:06.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-26T23:39:06.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-26T23:39:06.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-26T23:39:06.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-26T23:39:06.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-26T23:39:06.941 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-26T23:39:06.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-26T23:39:06.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-26T23:39:06.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-26T23:39:06.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-26T23:39:06.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-26T23:39:06.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-26T23:39:06.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-26T23:39:06.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-26T23:39:06.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-26T23:39:06.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-26T23:39:06.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-26T23:39:06.942 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-26T23:39:06.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-26T23:39:06.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-26T23:39:06.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-26T23:39:06.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-26T23:39:06.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-26T23:39:06.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-26T23:39:06.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-26T23:39:06.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-26T23:39:06.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-26T23:39:06.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-26T23:39:06.943 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-26T23:39:06.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-26T23:39:06.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-26T23:39:06.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-26T23:39:06.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-26T23:39:06.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-26T23:39:06.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-26T23:39:06.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-26T23:39:06.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-26T23:39:06.944 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-26T23:39:06.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-26T23:39:06.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-26T23:39:06.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-26T23:39:06.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-26T23:39:06.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-26T23:39:06.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-26T23:39:06.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-26T23:39:06.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-26T23:39:06.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-26T23:39:06.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-26T23:39:06.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-26T23:39:06.945 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-26T23:39:06.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-26T23:39:06.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-26T23:39:06.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-26T23:39:06.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-26T23:39:06.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-26T23:39:06.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-26T23:39:06.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-26T23:39:06.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-26T23:39:06.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-26T23:39:06.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-26T23:39:06.946 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-26T23:39:06.947 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-26T23:39:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-26T23:39:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-26T23:39:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-26T23:39:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-26T23:39:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-26T23:39:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-26T23:39:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-26T23:39:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-26T23:39:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-26T23:39:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-26T23:39:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-26T23:39:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-26T23:39:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-26T23:39:06.950 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-26T23:39:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-26T23:39:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-26T23:39:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-26T23:39:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-26T23:39:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-26T23:39:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-26T23:39:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-26T23:39:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-26T23:39:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-26T23:39:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-26T23:39:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-26T23:39:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-26T23:39:06.952 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-26T23:39:06.952 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-26T23:39:06.952 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-26T23:39:06.952 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-26T23:39:06.952 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-26T23:39:06.952 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-26T23:39:06.952 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-26T23:39:06.952 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-26T23:39:06.952 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-26T23:39:06.952 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-26T23:39:06.952 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-26T23:39:06.952 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-26T23:39:06.953 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-26T23:39:06.953 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-26T23:39:06.953 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-26T23:39:06.953 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-26T23:39:06.953 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-26T23:39:06.953 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-26T23:39:06.953 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-26T23:39:06.953 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-26T23:39:06.953 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-26T23:39:06.953 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-26T23:39:06.953 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-26T23:39:06.954 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-26T23:39:06.954 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-26T23:39:06.954 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-26T23:39:06.954 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-26T23:39:06.954 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-26T23:39:06.954 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-26T23:39:06.954 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-26T23:39:06.954 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-26T23:39:06.954 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-26T23:39:06.954 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-26T23:39:06.954 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-26T23:39:06.954 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-26T23:39:06.955 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-26T23:39:06.955 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-26T23:39:06.955 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-26T23:39:06.955 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-26T23:39:06.955 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-26T23:39:06.955 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-26T23:39:06.955 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-26T23:39:06.955 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-26T23:39:06.955 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-26T23:39:06.955 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-26T23:39:06.955 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-26T23:39:06.955 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-26T23:39:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-26T23:39:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-26T23:39:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-26T23:39:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-26T23:39:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-26T23:39:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-26T23:39:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-26T23:39:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-26T23:39:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-26T23:39:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-26T23:39:06.956 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-26T23:39:06.957 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-26T23:39:06.958 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-26T23:39:06.958 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-26T23:39:06.958 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-26T23:39:06.958 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-26T23:39:06.958 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-26T23:39:06.958 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-26T23:39:06.958 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-26T23:39:06.958 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-26T23:39:06.958 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-26T23:39:06.958 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-26T23:39:06.958 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-26T23:39:06.958 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-26T23:39:06.959 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-26T23:39:06.959 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-26T23:39:06.959 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-26T23:39:06.959 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-26T23:39:06.959 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-26T23:39:06.959 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-26T23:39:06.959 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-26T23:39:06.959 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-26T23:39:06.959 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-26T23:39:06.959 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-26T23:39:06.959 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-26T23:39:06.960 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-26T23:39:06.960 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-26T23:39:06.960 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-26T23:39:06.960 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-26T23:39:06.960 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-26T23:39:06.960 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-26T23:39:06.960 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-26T23:39:06.960 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-26T23:39:06.960 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-26T23:39:06.960 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-26T23:39:06.960 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-26T23:39:06.960 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-26T23:39:06.961 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-26T23:39:06.962 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-26T23:39:06.963 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-26T23:39:06.964 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-26T23:39:06.965 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-26T23:39:06.965 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-26T23:39:06.965 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-26T23:39:06.965 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-26T23:39:06.965 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-26T23:39:06.965 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-26T23:39:06.965 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-26T23:39:06.965 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-26T23:39:06.965 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-26T23:39:06.965 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-26T23:39:06.965 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-26T23:39:06.965 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-26T23:39:06.966 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-26T23:39:06.966 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-26T23:39:06.966 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-26T23:39:06.966 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-26T23:39:06.966 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-26T23:39:06.966 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-26T23:39:06.966 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-26T23:39:06.966 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-26T23:39:06.966 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-26T23:39:06.966 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-26T23:39:06.966 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-26T23:39:06.967 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-26T23:39:06.967 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-26T23:39:06.967 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-26T23:39:06.967 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-26T23:39:06.967 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-26T23:39:06.967 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-26T23:39:06.967 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-26T23:39:06.967 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-26T23:39:06.967 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-26T23:39:06.967 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-26T23:39:06.967 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-26T23:39:06.967 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-26T23:39:06.968 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-26T23:39:06.968 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-26T23:39:06.968 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-26T23:39:06.968 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-26T23:39:06.968 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-26T23:39:06.968 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-26T23:39:06.968 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-26T23:39:06.968 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-26T23:39:06.968 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-26T23:39:06.968 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-26T23:39:06.968 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-26T23:39:06.968 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-26T23:39:06.969 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-26T23:39:06.969 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-26T23:39:06.969 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-26T23:39:06.969 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-26T23:39:06.969 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-26T23:39:06.969 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-26T23:39:06.969 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-26T23:39:06.969 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-26T23:39:06.969 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-26T23:39:06.969 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-26T23:39:06.969 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-26T23:39:06.969 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-26T23:39:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-26T23:39:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-26T23:39:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-26T23:39:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-26T23:39:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-26T23:39:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-26T23:39:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-26T23:39:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-26T23:39:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-26T23:39:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-26T23:39:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-26T23:39:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-26T23:39:06.971 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-26T23:39:06.971 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-26T23:39:06.971 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-26T23:39:06.971 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-26T23:39:06.971 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-26T23:39:06.971 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-26T23:39:06.971 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-26T23:39:06.971 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-26T23:39:06.971 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-26T23:39:06.971 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-26T23:39:06.971 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-26T23:39:06.971 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-26T23:39:06.972 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-26T23:39:06.973 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-26T23:39:06.974 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-26T23:39:06.974 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-26T23:39:06.974 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-26T23:39:06.974 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-26T23:39:06.974 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-26T23:39:06.974 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-26T23:39:06.974 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-26T23:39:06.974 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-26T23:39:06.974 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-26T23:39:06.974 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-26T23:39:06.974 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-26T23:39:06.974 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-26T23:39:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-26T23:39:06.976 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-26T23:39:06.977 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-26T23:39:06.978 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-26T23:39:06.978 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-26T23:39:06.978 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-26T23:39:06.978 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-26T23:39:06.978 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-26T23:39:06.978 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-26T23:39:06.978 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-26T23:39:06.978 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-26T23:39:06.978 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-26T23:39:06.978 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-26T23:39:06.978 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-26T23:39:06.979 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-26T23:39:06.979 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-26T23:39:06.979 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-26T23:39:06.979 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-26T23:39:06.979 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-26T23:39:06.979 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-26T23:39:06.979 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-26T23:39:06.979 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-26T23:39:06.979 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-26T23:39:06.979 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-26T23:39:06.979 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-26T23:39:06.979 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-26T23:39:06.980 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-26T23:39:06.980 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-26T23:39:06.980 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-26T23:39:06.980 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-26T23:39:06.980 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-26T23:39:06.980 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-26T23:39:06.980 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-26T23:39:06.980 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-26T23:39:06.980 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-26T23:39:06.980 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-26T23:39:06.980 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-26T23:39:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-26T23:39:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-26T23:39:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-26T23:39:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-26T23:39:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-26T23:39:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-26T23:39:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-26T23:39:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-26T23:39:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-26T23:39:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-26T23:39:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-26T23:39:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-26T23:39:06.982 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-26T23:39:06.982 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-26T23:39:06.982 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-26T23:39:06.982 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-26T23:39:06.982 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-26T23:39:06.982 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-26T23:39:06.982 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-26T23:39:06.982 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-26T23:39:06.982 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-26T23:39:06.982 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-26T23:39:06.982 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-26T23:39:06.983 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-26T23:39:06.983 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-26T23:39:06.983 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-26T23:39:06.983 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-26T23:39:06.983 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-26T23:39:06.983 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-26T23:39:06.983 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-26T23:39:06.983 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-26T23:39:06.983 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-26T23:39:06.983 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-26T23:39:06.983 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-26T23:39:06.983 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-26T23:39:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-26T23:39:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-26T23:39:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-26T23:39:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-26T23:39:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-26T23:39:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-26T23:39:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-26T23:39:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-26T23:39:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-26T23:39:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-26T23:39:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-26T23:39:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-26T23:39:06.985 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-26T23:39:06.986 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-26T23:39:06.986 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-26T23:39:06.986 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-26T23:39:06.986 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-26T23:39:06.986 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-26T23:39:06.986 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-26T23:39:06.986 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-26T23:39:06.986 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-26T23:39:06.986 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-26T23:39:06.986 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-26T23:39:06.986 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-26T23:39:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-26T23:39:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-26T23:39:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-26T23:39:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-26T23:39:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-26T23:39:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-26T23:39:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-26T23:39:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-26T23:39:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-26T23:39:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-26T23:39:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-26T23:39:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-26T23:39:06.988 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-26T23:39:06.988 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-26T23:39:06.988 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-26T23:39:06.988 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-26T23:39:06.988 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-26T23:39:06.988 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-26T23:39:06.988 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-26T23:39:06.988 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-26T23:39:06.988 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-26T23:39:06.988 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-26T23:39:06.988 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-26T23:39:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-26T23:39:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-26T23:39:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-26T23:39:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-26T23:39:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-26T23:39:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-26T23:39:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-26T23:39:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-26T23:39:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-26T23:39:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-26T23:39:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-26T23:39:06.989 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-26T23:39:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-26T23:39:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-26T23:39:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-26T23:39:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-26T23:39:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-26T23:39:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-26T23:39:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-26T23:39:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-26T23:39:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-26T23:39:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-26T23:39:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-26T23:39:06.991 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-26T23:39:06.991 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-26T23:39:06.991 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-26T23:39:06.991 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-26T23:39:06.991 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-26T23:39:06.991 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-26T23:39:06.991 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-26T23:39:06.991 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-26T23:39:06.991 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-26T23:39:06.991 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-26T23:39:06.991 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-26T23:39:06.992 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-26T23:39:06.992 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-26T23:39:06.992 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-26T23:39:06.992 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-26T23:39:06.992 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-26T23:39:06.992 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-26T23:39:06.992 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-26T23:39:06.992 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-26T23:39:06.992 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-26T23:39:06.992 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-26T23:39:06.992 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-26T23:39:06.993 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-26T23:39:06.993 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-26T23:39:06.993 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-26T23:39:06.993 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-26T23:39:06.993 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-26T23:39:06.993 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-26T23:39:06.993 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-26T23:39:06.993 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-26T23:39:06.993 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-26T23:39:06.993 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-26T23:39:06.993 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-26T23:39:06.993 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-26T23:39:06.994 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-26T23:39:06.994 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-26T23:39:06.994 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-26T23:39:06.994 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-26T23:39:06.994 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-26T23:39:06.994 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-26T23:39:06.994 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-26T23:39:06.994 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-26T23:39:06.994 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-26T23:39:06.994 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-26T23:39:06.994 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-26T23:39:06.994 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-26T23:39:06.995 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-26T23:39:06.995 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-26T23:39:06.995 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-26T23:39:06.995 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-26T23:39:06.995 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-26T23:39:06.995 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-26T23:39:06.995 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-26T23:39:06.995 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-26T23:39:06.995 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-26T23:39:06.995 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-26T23:39:06.995 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-26T23:39:06.996 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-26T23:39:06.996 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-26T23:39:06.996 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-26T23:39:06.996 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-26T23:39:06.996 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-26T23:39:06.996 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-26T23:39:06.996 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-26T23:39:06.996 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-26T23:39:06.996 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-26T23:39:06.996 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-26T23:39:06.996 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-26T23:39:06.996 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-26T23:39:06.997 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-26T23:39:06.998 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-26T23:39:06.998 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-26T23:39:06.998 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-26T23:39:06.998 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-26T23:39:06.998 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-26T23:39:06.998 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-26T23:39:06.998 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-26T23:39:06.998 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-26T23:39:06.998 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-26T23:39:06.998 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-26T23:39:06.998 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-26T23:39:06.999 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-26T23:39:07.000 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-26T23:39:07.000 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-26T23:39:07.000 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-26T23:39:07.000 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-26T23:39:07.000 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-26T23:39:07.000 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-26T23:39:07.000 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-26T23:39:07.000 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-26T23:39:07.000 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-26T23:39:07.000 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-26T23:39:07.000 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-26T23:39:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-26T23:39:07.002 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-26T23:39:07.003 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-26T23:39:07.004 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-26T23:39:07.004 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-26T23:39:07.004 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-26T23:39:07.004 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-26T23:39:07.004 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-26T23:39:07.004 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-26T23:39:07.004 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-26T23:39:07.004 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-26T23:39:07.004 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-26T23:39:07.004 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-26T23:39:07.004 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-26T23:39:07.004 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-26T23:39:07.005 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-26T23:39:07.006 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-26T23:39:07.006 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-26T23:39:07.006 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-26T23:39:07.006 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-26T23:39:07.006 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-26T23:39:07.006 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-26T23:39:07.006 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-26T23:39:07.006 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-26T23:39:07.006 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-26T23:39:07.006 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-26T23:39:07.006 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-26T23:39:07.006 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-26T23:39:07.007 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-26T23:39:07.008 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-26T23:39:07.009 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-26T23:39:07.010 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-26T23:39:07.011 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-26T23:39:07.012 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-26T23:39:07.012 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-26T23:39:07.012 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-26T23:39:07.012 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-26T23:39:07.012 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-26T23:39:07.012 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-26T23:39:07.012 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-26T23:39:07.012 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-26T23:39:07.012 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-26T23:39:07.012 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-26T23:39:07.012 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-26T23:39:07.013 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-26T23:39:07.014 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-26T23:39:07.015 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-26T23:39:07.015 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-26T23:39:07.015 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-26T23:39:07.015 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-26T23:39:07.015 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-26T23:39:07.015 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-26T23:39:07.015 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-26T23:39:07.015 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-26T23:39:07.015 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-26T23:39:07.015 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-26T23:39:07.015 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-26T23:39:07.015 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-26T23:39:07.016 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-26T23:39:07.016 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-26T23:39:07.016 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-26T23:39:07.016 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-26T23:39:07.016 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-26T23:39:07.016 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-26T23:39:07.016 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-26T23:39:07.016 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-26T23:39:07.016 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-26T23:39:07.016 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-26T23:39:07.016 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-26T23:39:07.017 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-26T23:39:07.018 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-26T23:39:07.018 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-26T23:39:07.018 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-26T23:39:07.018 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-26T23:39:07.018 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-26T23:39:07.018 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-26T23:39:07.018 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-26T23:39:07.018 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-26T23:39:07.018 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-26T23:39:07.019 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-26T23:39:07.020 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-26T23:39:07.021 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-26T23:39:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-26T23:39:07.023 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-26T23:39:07.024 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-26T23:39:07.025 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-26T23:39:07.026 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-26T23:39:07.026 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-26T23:39:07.026 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-26T23:39:07.026 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-26T23:39:07.026 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-26T23:39:07.026 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-26T23:39:07.026 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-26T23:39:07.026 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-26T23:39:07.026 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-26T23:39:07.026 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-26T23:39:07.026 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-26T23:39:07.027 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-26T23:39:07.027 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-26T23:39:07.027 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-26T23:39:07.027 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-26T23:39:07.027 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-26T23:39:07.027 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-26T23:39:07.027 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-26T23:39:07.027 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-26T23:39:07.027 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-26T23:39:07.027 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-26T23:39:07.027 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-26T23:39:07.027 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-26T23:39:07.028 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-26T23:39:07.028 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-26T23:39:07.028 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-26T23:39:07.028 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-26T23:39:07.028 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-26T23:39:07.028 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-26T23:39:07.028 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-26T23:39:07.028 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-26T23:39:07.028 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-26T23:39:07.028 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-26T23:39:07.028 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-26T23:39:07.029 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-26T23:39:07.029 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-26T23:39:07.029 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-26T23:39:07.029 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-26T23:39:07.029 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-26T23:39:07.029 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-26T23:39:07.029 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-26T23:39:07.029 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-26T23:39:07.029 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-26T23:39:07.029 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-26T23:39:07.029 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-26T23:39:07.029 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-26T23:39:07.030 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-26T23:39:07.030 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-26T23:39:07.030 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-26T23:39:07.030 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-26T23:39:07.030 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-26T23:39:07.030 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-26T23:39:07.030 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-26T23:39:07.030 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-26T23:39:07.030 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-26T23:39:07.030 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-26T23:39:07.030 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-26T23:39:07.030 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-26T23:39:07.031 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-26T23:39:07.032 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-26T23:39:07.033 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-26T23:39:07.033 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-26T23:39:07.033 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-26T23:39:07.033 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-26T23:39:07.033 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-26T23:39:07.033 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-26T23:39:07.033 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-26T23:39:07.033 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-26T23:39:07.033 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-26T23:39:07.033 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-26T23:39:07.033 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-26T23:39:07.033 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-26T23:39:07.034 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-26T23:39:07.035 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-26T23:39:07.035 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-26T23:39:07.035 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-26T23:39:07.035 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-26T23:39:07.035 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-26T23:39:07.035 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-26T23:39:07.035 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-26T23:39:07.035 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-26T23:39:07.035 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-26T23:39:07.035 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-26T23:39:07.035 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-26T23:39:07.035 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-26T23:39:07.036 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-26T23:39:07.037 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-26T23:39:07.037 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-26T23:39:07.037 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-26T23:39:07.037 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-26T23:39:07.037 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-26T23:39:07.037 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-26T23:39:07.037 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-26T23:39:07.037 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-26T23:39:07.037 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-26T23:39:07.037 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-26T23:39:07.037 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-26T23:39:07.037 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-26T23:39:07.038 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-26T23:39:07.038 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-26T23:39:07.038 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-26T23:39:07.038 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-26T23:39:07.038 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-26T23:39:07.038 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-26T23:39:07.038 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-26T23:39:07.038 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-26T23:39:07.038 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-26T23:39:07.038 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-26T23:39:07.038 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-26T23:39:07.039 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-26T23:39:07.039 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-26T23:39:07.039 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-26T23:39:07.039 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-26T23:39:07.039 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-26T23:39:07.039 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-26T23:39:07.039 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-26T23:39:07.039 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-26T23:39:07.039 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-26T23:39:07.039 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-26T23:39:07.039 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-26T23:39:07.040 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-26T23:39:07.040 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-26T23:39:07.040 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-26T23:39:07.040 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-26T23:39:07.040 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-26T23:39:07.040 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-26T23:39:07.040 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-26T23:39:07.040 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-26T23:39:07.040 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-26T23:39:07.040 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-26T23:39:07.040 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-26T23:39:07.040 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-26T23:39:07.041 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-26T23:39:07.042 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-26T23:39:07.042 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-26T23:39:07.042 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-26T23:39:07.042 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-26T23:39:07.042 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-26T23:39:07.042 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-26T23:39:07.042 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-26T23:39:07.042 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-26T23:39:07.042 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-26T23:39:07.042 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-26T23:39:07.042 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-26T23:39:07.042 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-26T23:39:07.043 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-26T23:39:07.043 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-26T23:39:07.043 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-26T23:39:07.043 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-26T23:39:07.043 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-26T23:39:07.043 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-26T23:39:07.043 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-26T23:39:07.043 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-26T23:39:07.043 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-26T23:39:07.043 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-26T23:39:07.043 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-26T23:39:07.044 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-26T23:39:07.044 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-26T23:39:07.044 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-26T23:39:07.044 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-26T23:39:07.044 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-26T23:39:07.044 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-26T23:39:07.044 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-26T23:39:07.044 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-26T23:39:07.044 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-26T23:39:07.044 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-26T23:39:07.044 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-26T23:39:07.044 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-26T23:39:07.045 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-26T23:39:07.045 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-26T23:39:07.045 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-26T23:39:07.045 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-26T23:39:07.045 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-26T23:39:07.045 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-26T23:39:07.045 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-26T23:39:07.045 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-26T23:39:07.045 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-26T23:39:07.045 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-26T23:39:07.045 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-26T23:39:07.046 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-26T23:39:07.046 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-26T23:39:07.046 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-26T23:39:07.046 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-26T23:39:07.046 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-26T23:39:07.046 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-26T23:39:07.046 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-26T23:39:07.046 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-26T23:39:07.046 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-26T23:39:07.046 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-26T23:39:07.046 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-26T23:39:07.046 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-26T23:39:07.047 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-26T23:39:07.047 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-26T23:39:07.047 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-26T23:39:07.047 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-26T23:39:07.047 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-26T23:39:07.047 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-26T23:39:07.047 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-26T23:39:07.047 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-26T23:39:07.047 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-26T23:39:07.047 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-26T23:39:07.047 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-26T23:39:07.047 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-26T23:39:07.048 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-26T23:39:07.048 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-26T23:39:07.048 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-26T23:39:07.048 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-26T23:39:07.048 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-26T23:39:07.048 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-26T23:39:07.048 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-26T23:39:07.048 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-26T23:39:07.048 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-26T23:39:07.048 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-26T23:39:07.048 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-26T23:39:07.048 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-26T23:39:07.049 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-26T23:39:07.050 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-26T23:39:07.050 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-26T23:39:07.050 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-26T23:39:07.050 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-26T23:39:07.050 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-26T23:39:07.050 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-26T23:39:07.050 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-26T23:39:07.050 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-26T23:39:07.050 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-26T23:39:07.050 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-26T23:39:07.050 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-26T23:39:07.050 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-26T23:39:07.051 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-26T23:39:07.051 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-26T23:39:07.051 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-26T23:39:07.051 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-26T23:39:07.051 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-26T23:39:07.051 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-26T23:39:07.051 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-26T23:39:07.051 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-26T23:39:07.051 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-26T23:39:07.051 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-26T23:39:07.051 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-26T23:39:07.052 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-26T23:39:07.053 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-26T23:39:07.053 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-26T23:39:07.053 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-26T23:39:07.053 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-26T23:39:07.053 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-26T23:39:07.053 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-26T23:39:07.053 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-26T23:39:07.053 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-26T23:39:07.053 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-26T23:39:07.053 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-26T23:39:07.053 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-26T23:39:07.054 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-26T23:39:07.054 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-26T23:39:07.054 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-26T23:39:07.054 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-26T23:39:07.054 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-26T23:39:07.054 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-26T23:39:07.054 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-26T23:39:07.054 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-26T23:39:07.054 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-26T23:39:07.054 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-26T23:39:07.054 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-26T23:39:07.054 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-26T23:39:07.055 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-26T23:39:07.055 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-26T23:39:07.055 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-26T23:39:07.055 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-26T23:39:07.055 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-26T23:39:07.055 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-26T23:39:07.055 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-26T23:39:07.055 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-26T23:39:07.055 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-26T23:39:07.055 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-26T23:39:07.055 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-26T23:39:07.056 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-26T23:39:07.056 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-26T23:39:07.056 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-26T23:39:07.056 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-26T23:39:07.056 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-26T23:39:07.056 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-26T23:39:07.056 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-26T23:39:07.056 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-26T23:39:07.056 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-26T23:39:07.056 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-26T23:39:07.057 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-26T23:39:07.057 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-26T23:39:07.057 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-26T23:39:07.057 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-26T23:39:07.057 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-26T23:39:07.057 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-26T23:39:07.057 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-26T23:39:07.057 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-26T23:39:07.057 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-26T23:39:07.057 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-26T23:39:07.057 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-26T23:39:07.058 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-26T23:39:07.058 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-26T23:39:07.058 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-26T23:39:07.058 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-26T23:39:07.058 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-26T23:39:07.058 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-26T23:39:07.058 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-26T23:39:07.058 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-26T23:39:07.058 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-26T23:39:07.058 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-26T23:39:07.058 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-26T23:39:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-26T23:39:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-26T23:39:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-26T23:39:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-26T23:39:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-26T23:39:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-26T23:39:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-26T23:39:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-26T23:39:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-26T23:39:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-26T23:39:07.060 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-26T23:39:07.060 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-26T23:39:07.060 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-26T23:39:07.060 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-26T23:39:07.060 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-26T23:39:07.060 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-26T23:39:07.060 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-26T23:39:07.060 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-26T23:39:07.060 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-26T23:39:07.060 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-26T23:39:07.060 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-26T23:39:07.060 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-26T23:39:07.061 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-26T23:39:07.061 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-26T23:39:07.061 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-26T23:39:07.061 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-26T23:39:07.061 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-26T23:39:07.061 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-26T23:39:07.061 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-26T23:39:07.061 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-26T23:39:07.061 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-26T23:39:07.061 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-26T23:39:07.061 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-26T23:39:07.061 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-26T23:39:07.062 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-26T23:39:07.062 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-26T23:39:07.062 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-26T23:39:07.062 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-26T23:39:07.062 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-26T23:39:07.062 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-26T23:39:07.062 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-26T23:39:07.062 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-26T23:39:07.062 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-26T23:39:07.062 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-26T23:39:07.062 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-26T23:39:07.062 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-26T23:39:07.063 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-26T23:39:07.064 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-26T23:39:07.064 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-26T23:39:07.064 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-26T23:39:07.064 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-26T23:39:07.064 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-26T23:39:07.064 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-26T23:39:07.064 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-26T23:39:07.064 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-26T23:39:07.064 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-26T23:39:07.064 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-26T23:39:07.064 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-26T23:39:07.065 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-26T23:39:07.065 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-26T23:39:07.065 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-26T23:39:07.065 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-26T23:39:07.065 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-26T23:39:07.065 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-26T23:39:07.065 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-26T23:39:07.065 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-26T23:39:07.065 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-26T23:39:07.065 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-26T23:39:07.065 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-26T23:39:07.065 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-26T23:39:07.066 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-26T23:39:07.066 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-26T23:39:07.066 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-26T23:39:07.066 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-26T23:39:07.066 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-26T23:39:07.066 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-26T23:39:07.066 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-26T23:39:07.066 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-26T23:39:07.066 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-26T23:39:07.066 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-26T23:39:07.066 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-26T23:39:07.067 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-26T23:39:07.067 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-26T23:39:07.067 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-26T23:39:07.067 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-26T23:39:07.067 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-26T23:39:07.067 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-26T23:39:07.067 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-26T23:39:07.067 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-26T23:39:07.067 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-26T23:39:07.067 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-26T23:39:07.067 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-26T23:39:07.067 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-26T23:39:07.068 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-26T23:39:07.068 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-26T23:39:07.068 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-26T23:39:07.068 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-26T23:39:07.068 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-26T23:39:07.068 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-26T23:39:07.068 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-26T23:39:07.068 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-26T23:39:07.068 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-26T23:39:07.068 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-26T23:39:07.068 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-26T23:39:07.069 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-26T23:39:07.069 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-26T23:39:07.069 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-26T23:39:07.069 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-26T23:39:07.069 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-26T23:39:07.069 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-26T23:39:07.069 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-26T23:39:07.069 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-26T23:39:07.069 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-26T23:39:07.069 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-26T23:39:07.069 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-26T23:39:07.069 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-26T23:39:07.070 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-26T23:39:07.070 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-26T23:39:07.070 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-26T23:39:07.070 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-26T23:39:07.070 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-26T23:39:07.070 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-26T23:39:07.070 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-26T23:39:07.070 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-26T23:39:07.070 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-26T23:39:07.070 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-26T23:39:07.070 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-26T23:39:07.071 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-26T23:39:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-26T23:39:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-26T23:39:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-26T23:39:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-26T23:39:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-26T23:39:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-26T23:39:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-26T23:39:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-26T23:39:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-26T23:39:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-26T23:39:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-26T23:39:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-26T23:39:07.073 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-26T23:39:07.073 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-26T23:39:07.073 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-26T23:39:07.073 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-26T23:39:07.073 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-26T23:39:07.073 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-26T23:39:07.073 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-26T23:39:07.073 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-26T23:39:07.073 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-26T23:39:07.073 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-26T23:39:07.073 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-26T23:39:07.074 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-26T23:39:07.074 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-26T23:39:07.074 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-26T23:39:07.074 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-26T23:39:07.074 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-26T23:39:07.074 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-26T23:39:07.074 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-26T23:39:07.074 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-26T23:39:07.074 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-26T23:39:07.074 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-26T23:39:07.074 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-26T23:39:07.074 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-26T23:39:07.075 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-26T23:39:07.076 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-26T23:39:07.077 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-26T23:39:07.078 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-26T23:39:07.078 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-26T23:39:07.078 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-26T23:39:07.078 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-26T23:39:07.078 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-26T23:39:07.078 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-26T23:39:07.078 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-26T23:39:07.078 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-26T23:39:07.078 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-26T23:39:07.078 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-26T23:39:07.078 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-26T23:39:07.079 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-26T23:39:07.079 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-26T23:39:07.079 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-26T23:39:07.079 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-26T23:39:07.079 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-26T23:39:07.079 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-26T23:39:07.079 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-26T23:39:07.079 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-26T23:39:07.079 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-26T23:39:07.079 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-26T23:39:07.079 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-26T23:39:07.079 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-26T23:39:07.080 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-26T23:39:07.081 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-26T23:39:07.081 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-26T23:39:07.081 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-26T23:39:07.081 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-26T23:39:07.081 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-26T23:39:07.081 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-26T23:39:07.081 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-26T23:39:07.081 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-26T23:39:07.081 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-26T23:39:07.081 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-26T23:39:07.081 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-26T23:39:07.081 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-26T23:39:07.082 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-26T23:39:07.082 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-26T23:39:07.082 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-26T23:39:07.082 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-26T23:39:07.082 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-26T23:39:07.082 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-26T23:39:07.082 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-26T23:39:07.082 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-26T23:39:07.082 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-26T23:39:07.082 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-26T23:39:07.082 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-26T23:39:07.083 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-26T23:39:07.083 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-26T23:39:07.083 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-26T23:39:07.083 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-26T23:39:07.083 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-26T23:39:07.083 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-26T23:39:07.083 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-26T23:39:07.083 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-26T23:39:07.083 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-26T23:39:07.083 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-26T23:39:07.083 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-26T23:39:07.083 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-26T23:39:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-26T23:39:07.085 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-26T23:39:07.086 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-26T23:39:07.087 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-26T23:39:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-26T23:39:07.089 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-26T23:39:07.090 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-26T23:39:07.091 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-26T23:39:07.091 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-26T23:39:07.091 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-26T23:39:07.091 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-26T23:39:07.091 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-26T23:39:07.091 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-26T23:39:07.091 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-26T23:39:07.091 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-26T23:39:07.091 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-26T23:39:07.091 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-26T23:39:07.091 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-26T23:39:07.091 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-26T23:39:07.092 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-26T23:39:07.093 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-26T23:39:07.093 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-26T23:39:07.093 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-26T23:39:07.093 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-26T23:39:07.093 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-26T23:39:07.093 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-26T23:39:07.093 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-26T23:39:07.093 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-26T23:39:07.093 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-26T23:39:07.093 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-26T23:39:07.093 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-26T23:39:07.093 INFO:teuthology.orchestra.run.smithi040.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T23:39:07.097 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-26T23:39:07.097 DEBUG:teuthology.orchestra.run.smithi040:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-26T23:39:07.121 INFO:teuthology.orchestra.run.smithi040.stdout:postinstall scriptlet (using /bin/sh): 2024-04-26T23:39:07.122 INFO:teuthology.orchestra.run.smithi040.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T23:39:07.122 INFO:teuthology.orchestra.run.smithi040.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T23:39:07.122 INFO:teuthology.orchestra.run.smithi040.stdout:fi 2024-04-26T23:39:07.122 INFO:teuthology.orchestra.run.smithi040.stdout:for file in vmlinuz System.map config; do 2024-04-26T23:39:07.122 INFO:teuthology.orchestra.run.smithi040.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-26T23:39:07.122 INFO:teuthology.orchestra.run.smithi040.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-26T23:39:07.122 INFO:teuthology.orchestra.run.smithi040.stdout: fi 2024-04-26T23:39:07.122 INFO:teuthology.orchestra.run.smithi040.stdout:done 2024-04-26T23:39:07.122 INFO:teuthology.orchestra.run.smithi040.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-26T23:39:07.122 INFO:teuthology.orchestra.run.smithi040.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-26T23:39:07.122 INFO:teuthology.orchestra.run.smithi040.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T23:39:07.123 INFO:teuthology.orchestra.run.smithi040.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T23:39:07.123 INFO:teuthology.orchestra.run.smithi040.stdout:fi 2024-04-26T23:39:07.123 INFO:teuthology.orchestra.run.smithi040.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-26T23:39:07.123 INFO:teuthology.orchestra.run.smithi040.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-26T23:39:07.123 INFO:teuthology.orchestra.run.smithi040.stdout:fi 2024-04-26T23:39:07.123 INFO:teuthology.orchestra.run.smithi040.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-26T23:39:07.123 INFO:teuthology.orchestra.run.smithi040.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T23:39:07.123 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-26T23:39:07.123 INFO:teuthology.orchestra.run.smithi040.stdout:fi 2024-04-26T23:39:07.124 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -qi grub2-tools 2024-04-26T23:39:07.198 INFO:teuthology.orchestra.run.smithi040.stdout:Name : grub2-tools 2024-04-26T23:39:07.198 INFO:teuthology.orchestra.run.smithi040.stdout:Epoch : 1 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:Version : 2.06 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:Release : 68.el9 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: x86_64 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:Group : Unspecified 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:Size : 8264254 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:License : GPLv3+ 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:Packager : builder@centos.org 2024-04-26T23:39:07.199 INFO:teuthology.orchestra.run.smithi040.stdout:Vendor : CentOS 2024-04-26T23:39:07.200 INFO:teuthology.orchestra.run.smithi040.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-26T23:39:07.200 INFO:teuthology.orchestra.run.smithi040.stdout:Summary : Support tools for GRUB. 2024-04-26T23:39:07.200 INFO:teuthology.orchestra.run.smithi040.stdout:Description : 2024-04-26T23:39:07.200 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:39:07.200 INFO:teuthology.orchestra.run.smithi040.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-26T23:39:07.200 INFO:teuthology.orchestra.run.smithi040.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-26T23:39:07.200 INFO:teuthology.orchestra.run.smithi040.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-26T23:39:07.200 INFO:teuthology.orchestra.run.smithi040.stdout:hardware devices. 2024-04-26T23:39:07.200 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:39:07.200 INFO:teuthology.orchestra.run.smithi040.stdout:This subpackage provides tools for support of all platforms. 2024-04-26T23:39:07.201 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-26T23:39:07.201 INFO:teuthology.task.kernel:Updating grub on smithi040 to boot 6.9.0-rc3-g806ec36b2112 2024-04-26T23:39:07.201 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-26T23:39:07.773 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2024-04-26T23:39:09.215 INFO:teuthology.orchestra.run.smithi040.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-26T23:39:09.260 INFO:teuthology.orchestra.run.smithi040.stderr:done 2024-04-26T23:39:09.262 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-04-26T23:39:09.285 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.lycIm5EslC 2024-04-26T23:39:09.285 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lycIm5EslC 2024-04-26T23:39:09.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.lycIm5EslC 2024-04-26T23:39:09.491 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.lycIm5EslC is 6KB 2024-04-26T23:39:09.511 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.lycIm5EslC 2024-04-26T23:39:09.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo /bin/ls /boot/loader/entries || true 2024-04-26T23:39:09.588 INFO:teuthology.orchestra.run.smithi040.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-26T23:39:09.588 INFO:teuthology.orchestra.run.smithi040.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-26T23:39:09.588 INFO:teuthology.orchestra.run.smithi040.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-26T23:39:09.588 INFO:teuthology.orchestra.run.smithi040.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-26T23:39:09.590 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-26T23:39:09.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2024-04-26T23:39:39.746 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-26T23:39:39.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-26T23:39:39.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:40:39.750 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T23:40:48.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-26T23:40:48.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:40:49.392 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-04-26T23:40:49.967 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-26T23:40:49.967 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-26T23:40:49.967 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-04-26T23:40:50.010 INFO:teuthology.orchestra.run.smithi040.stdout:6.9.0-rc3-g806ec36b2112 2024-04-26T23:40:50.010 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-26T23:40:50.010 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T23:40:50.010 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-26T23:40:51.013 DEBUG:teuthology.parallel:result is None 2024-04-26T23:40:51.014 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T23:40:51.021 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T23:40:51.022 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T23:40:51.030 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T23:40:51.033 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T23:40:51.077 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T23:40:51.084 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T23:40:51.093 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T23:40:51.093 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T23:40:51.097 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T23:40:51.100 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T23:40:51.218 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T23:40:51.225 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T23:40:51.225 DEBUG:teuthology.orchestra.run.smithi080:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T23:40:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:40:51.241 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T23:40:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:40:51.259 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T23:40:51.264 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T23:40:51.284 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T23:40:51.324 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T23:40:51.343 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T23:40:51.359 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T23:40:51.370 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T23:40:51.379 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T23:40:51.411 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T23:40:51.416 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T23:40:51.426 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T23:40:51.426 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T23:40:51.433 DEBUG:teuthology.orchestra.run.smithi080:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T23:40:51.435 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T23:40:51.499 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T23:40:51.507 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T23:40:51.507 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T23:40:51.512 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T23:40:51.514 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T23:40:51.547 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T23:40:51.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T23:40:51.765 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T23:40:51.836 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T23:40:51.929 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:40:51.929 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T23:40:52.040 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T23:40:52.085 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T23:40:52.143 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T23:40:52.226 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T23:40:52.268 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-26T23:40:52.268 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T23:40:52.333 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T23:40:52.375 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T23:40:52.429 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T23:40:52.514 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T23:40:52.554 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:40:52.554 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T23:40:52.620 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-04-26T23:40:52.627 DEBUG:teuthology.orchestra.run.smithi080:> sudo service rsyslog restart 2024-04-26T23:40:52.629 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-04-26T23:40:52.662 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T23:40:52.710 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T23:40:52.746 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T23:40:53.134 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T23:40:53.141 INFO:teuthology.task.internal:Starting timer... 2024-04-26T23:40:53.142 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T23:40:53.161 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T23:40:53.169 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-26T23:40:53.169 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-04-26T23:40:53.269 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-04-26T23:40:53.273 DEBUG:teuthology.orchestra.run.smithi080:> sudo service auditd rotate 2024-04-26T23:40:53.330 INFO:teuthology.orchestra.run.smithi080.stdout:Rotating logs: 2024-04-26T23:40:53.332 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-04-26T23:40:53.399 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-04-26T23:40:53.400 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T23:40:53.401 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-04-26T23:40:53.467 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-04-26T23:40:53.467 DEBUG:teuthology.orchestra.run.smithi080:> /usr/sbin/getenforce 2024-04-26T23:40:53.490 INFO:teuthology.orchestra.run.smithi080.stdout:Permissive 2024-04-26T23:40:53.490 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-04-26T23:40:53.516 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-04-26T23:40:53.516 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi080.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive'} 2024-04-26T23:40:53.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T23:40:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:40:53.597 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T23:40:53.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:40:53.624 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T23:40:53.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:40:53.652 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T23:40:53.653 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-04-26T23:40:53.749 DEBUG:teuthology.orchestra.run.smithi080:> sudo /usr/sbin/setenforce permissive 2024-04-26T23:40:53.774 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-04-26T23:40:53.799 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T23:40:53.809 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-26T23:40:53.814 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-26T23:40:53.942 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T23:40:53.957 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T23:40:53.958 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com,smithi080.front.sepia.ceph.com,smithi103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T23:46:06.511 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi080.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com')] 2024-04-26T23:46:06.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-26T23:46:06.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:46:06.624 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-04-26T23:46:06.778 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-04-26T23:46:06.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-04-26T23:46:06.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:46:06.854 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-04-26T23:46:06.946 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-04-26T23:46:06.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-26T23:46:06.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:46:07.017 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-26T23:46:07.059 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-26T23:46:07.060 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T23:46:07.070 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T23:46:07.070 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T23:46:07.071 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T23:46:07.078 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T23:46:07.079 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T23:46:07.081 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T23:46:07.081 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T23:46:07.109 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T23:46:07.123 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T23:46:07.125 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T23:46:07.139 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T23:46:07.147 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: ntpd: command not found 2024-04-26T23:46:07.158 INFO:teuthology.orchestra.run.smithi080.stdout:506 Cannot talk to daemon 2024-04-26T23:46:07.161 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-04-26T23:46:07.171 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-04-26T23:46:07.172 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T23:46:07.174 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T23:46:07.185 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T23:46:07.185 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T23:46:07.198 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T23:46:07.219 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T23:46:07.239 INFO:teuthology.orchestra.run.smithi080.stderr:bash: line 1: ntpq: command not found 2024-04-26T23:46:07.242 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T23:46:07.243 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2024-04-26T23:46:07.243 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:46:07.243 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:46:07.243 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:46:07.243 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:46:07.257 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-26T23:46:07.261 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T23:46:07.261 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-26T23:46:07.261 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:46:07.261 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:46:07.261 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:46:07.261 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:46:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2024-04-26T23:46:07.330 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2024-04-26T23:46:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T23:46:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T23:46:07.561 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-04-26T23:46:07.571 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T23:46:07.572 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-26T23:46:07.572 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:46:07.572 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:46:07.572 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:46:07.572 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:46:07.573 INFO:teuthology.run_tasks:Running task install... 2024-04-26T23:46:07.583 DEBUG:teuthology.task.install:project ceph 2024-04-26T23:46:07.583 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}} 2024-04-26T23:46:07.583 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'} 2024-04-26T23:46:07.583 INFO:teuthology.task.install:Using flavor: default 2024-04-26T23:46:07.591 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T23:46:07.591 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T23:46:07.591 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T23:46:07.592 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:46:07.594 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T23:46:07.594 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:46:07.595 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T23:46:07.595 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:46:07.741 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T23:46:07.742 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T23:46:07.748 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T23:46:07.748 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T23:46:07.753 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T23:46:07.753 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T23:46:07.860 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T23:46:07.860 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:46:07.860 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T23:46:07.868 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T23:46:07.868 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-26T23:46:07.868 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T23:46:07.874 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T23:46:07.875 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:46:07.875 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T23:46:07.894 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T23:46:07.894 DEBUG:teuthology.orchestra.run.smithi080:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T23:46:07.901 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T23:46:07.901 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T23:46:07.955 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T23:46:07.955 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T23:46:07.959 DEBUG:teuthology.orchestra.run.smithi080:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T23:46:07.966 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T23:46:08.033 DEBUG:teuthology.orchestra.run.smithi080:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T23:46:08.039 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T23:46:08.088 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T23:46:08.096 INFO:teuthology.orchestra.run.smithi080.stdout:check_obsoletes = 1 2024-04-26T23:46:08.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum clean all 2024-04-26T23:46:08.102 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-04-26T23:46:08.103 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-04-26T23:46:08.209 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T23:46:08.293 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2024-04-26T23:46:08.298 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2024-04-26T23:46:08.321 INFO:teuthology.orchestra.run.smithi080.stdout:48 files removed 2024-04-26T23:46:08.329 INFO:teuthology.orchestra.run.smithi103.stdout:48 files removed 2024-04-26T23:46:08.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T23:46:08.355 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T23:46:08.638 INFO:teuthology.orchestra.run.smithi040.stdout:48 files removed 2024-04-26T23:46:08.678 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T23:46:09.002 INFO:teuthology.orchestra.run.smithi080.stdout:ceph packages for x86_64 211 kB/s | 84 kB 00:00 2024-04-26T23:46:09.086 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 179 kB/s | 84 kB 00:00 2024-04-26T23:46:09.378 INFO:teuthology.orchestra.run.smithi080.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-26T23:46:09.520 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-04-26T23:46:09.602 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 176 kB/s | 84 kB 00:00 2024-04-26T23:46:09.753 INFO:teuthology.orchestra.run.smithi080.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-04-26T23:46:09.878 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-26T23:46:10.187 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 24 kB/s | 12 kB 00:00 2024-04-26T23:46:10.195 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.0 MB 00:00 2024-04-26T23:46:10.431 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-26T23:46:10.761 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-04-26T23:46:11.468 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-26T23:46:12.391 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-26T23:46:13.831 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - AppStream 7.3 MB/s | 19 MB 00:02 2024-04-26T23:46:16.383 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-04-26T23:46:17.934 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-04-26T23:46:17.965 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-26T23:46:19.338 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-26T23:46:19.616 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-26T23:46:21.062 INFO:teuthology.orchestra.run.smithi080.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-26T23:46:24.628 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 5.3 kB/s | 1.7 kB 00:00 2024-04-26T23:46:26.325 INFO:teuthology.orchestra.run.smithi080.stdout:lab-extras 5.8 kB/s | 1.7 kB 00:00 2024-04-26T23:46:26.793 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T23:46:26.794 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T23:46:26.805 INFO:teuthology.orchestra.run.smithi103.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T23:46:26.806 INFO:teuthology.orchestra.run.smithi103.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T23:46:26.807 INFO:teuthology.orchestra.run.smithi103.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T23:46:26.808 INFO:teuthology.orchestra.run.smithi103.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T23:46:26.809 INFO:teuthology.orchestra.run.smithi103.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T23:46:26.812 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T23:46:26.813 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T23:46:26.873 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-04-26T23:46:26.890 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-26T23:46:26.890 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-04-26T23:46:26.890 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T23:46:26.891 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T23:46:26.892 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-04-26T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T23:46:26.894 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T23:46:26.895 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T23:46:26.896 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T23:46:26.896 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T23:46:26.896 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T23:46:26.896 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T23:46:26.896 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T23:46:26.896 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T23:46:26.896 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T23:46:26.897 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T23:46:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T23:46:26.899 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T23:46:26.900 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T23:46:26.901 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:46:26.902 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-04-26T23:46:26.903 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-04-26T23:46:26.903 INFO:teuthology.orchestra.run.smithi103.stdout:Install 148 Packages 2024-04-26T23:46:26.903 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 4 Packages 2024-04-26T23:46:26.903 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:46:26.903 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 222 M 2024-04-26T23:46:26.904 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-04-26T23:46:27.496 INFO:teuthology.orchestra.run.smithi103.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-26T23:46:27.879 INFO:teuthology.orchestra.run.smithi103.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 2.3 MB/s | 889 kB 00:00 2024-04-26T23:46:27.946 INFO:teuthology.orchestra.run.smithi103.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 2.1 MB/s | 145 kB 00:00 2024-04-26T23:46:28.346 INFO:teuthology.orchestra.run.smithi103.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 5.9 MB/s | 2.4 MB 00:00 2024-04-26T23:46:28.468 INFO:teuthology.orchestra.run.smithi080.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T23:46:28.468 INFO:teuthology.orchestra.run.smithi080.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T23:46:28.479 INFO:teuthology.orchestra.run.smithi080.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T23:46:28.480 INFO:teuthology.orchestra.run.smithi080.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T23:46:28.481 INFO:teuthology.orchestra.run.smithi080.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T23:46:28.482 INFO:teuthology.orchestra.run.smithi080.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T23:46:28.483 INFO:teuthology.orchestra.run.smithi080.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T23:46:28.486 INFO:teuthology.orchestra.run.smithi080.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T23:46:28.486 INFO:teuthology.orchestra.run.smithi080.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T23:46:28.530 INFO:teuthology.orchestra.run.smithi103.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 8.3 MB/s | 1.5 MB 00:00 2024-04-26T23:46:28.547 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-26T23:46:28.556 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================= 2024-04-26T23:46:28.556 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-04-26T23:46:28.557 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================= 2024-04-26T23:46:28.557 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-26T23:46:28.557 INFO:teuthology.orchestra.run.smithi080.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T23:46:28.557 INFO:teuthology.orchestra.run.smithi080.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T23:46:28.557 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T23:46:28.557 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T23:46:28.557 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T23:46:28.557 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T23:46:28.557 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T23:46:28.557 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T23:46:28.557 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T23:46:28.557 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T23:46:28.558 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T23:46:28.558 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T23:46:28.558 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T23:46:28.558 INFO:teuthology.orchestra.run.smithi080.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T23:46:28.558 INFO:teuthology.orchestra.run.smithi080.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T23:46:28.558 INFO:teuthology.orchestra.run.smithi080.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T23:46:28.558 INFO:teuthology.orchestra.run.smithi080.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T23:46:28.558 INFO:teuthology.orchestra.run.smithi080.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T23:46:28.558 INFO:teuthology.orchestra.run.smithi080.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T23:46:28.558 INFO:teuthology.orchestra.run.smithi080.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T23:46:28.558 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T23:46:28.558 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T23:46:28.559 INFO:teuthology.orchestra.run.smithi080.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T23:46:28.559 INFO:teuthology.orchestra.run.smithi080.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T23:46:28.559 INFO:teuthology.orchestra.run.smithi080.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T23:46:28.559 INFO:teuthology.orchestra.run.smithi080.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T23:46:28.559 INFO:teuthology.orchestra.run.smithi080.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T23:46:28.559 INFO:teuthology.orchestra.run.smithi080.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T23:46:28.559 INFO:teuthology.orchestra.run.smithi080.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T23:46:28.559 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T23:46:28.559 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T23:46:28.559 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout:Upgrading: 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T23:46:28.560 INFO:teuthology.orchestra.run.smithi080.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T23:46:28.561 INFO:teuthology.orchestra.run.smithi080.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T23:46:28.562 INFO:teuthology.orchestra.run.smithi080.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T23:46:28.562 INFO:teuthology.orchestra.run.smithi080.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T23:46:28.562 INFO:teuthology.orchestra.run.smithi080.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T23:46:28.562 INFO:teuthology.orchestra.run.smithi080.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T23:46:28.562 INFO:teuthology.orchestra.run.smithi080.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T23:46:28.562 INFO:teuthology.orchestra.run.smithi080.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T23:46:28.562 INFO:teuthology.orchestra.run.smithi080.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T23:46:28.562 INFO:teuthology.orchestra.run.smithi080.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T23:46:28.562 INFO:teuthology.orchestra.run.smithi080.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T23:46:28.562 INFO:teuthology.orchestra.run.smithi080.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T23:46:28.562 INFO:teuthology.orchestra.run.smithi080.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T23:46:28.562 INFO:teuthology.orchestra.run.smithi080.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T23:46:28.563 INFO:teuthology.orchestra.run.smithi080.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T23:46:28.564 INFO:teuthology.orchestra.run.smithi080.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T23:46:28.565 INFO:teuthology.orchestra.run.smithi080.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T23:46:28.566 INFO:teuthology.orchestra.run.smithi080.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T23:46:28.566 INFO:teuthology.orchestra.run.smithi080.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T23:46:28.566 INFO:teuthology.orchestra.run.smithi080.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T23:46:28.566 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T23:46:28.566 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T23:46:28.566 INFO:teuthology.orchestra.run.smithi080.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T23:46:28.566 INFO:teuthology.orchestra.run.smithi080.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T23:46:28.566 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T23:46:28.566 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T23:46:28.566 INFO:teuthology.orchestra.run.smithi080.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T23:46:28.566 INFO:teuthology.orchestra.run.smithi080.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T23:46:28.566 INFO:teuthology.orchestra.run.smithi080.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T23:46:28.567 INFO:teuthology.orchestra.run.smithi080.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T23:46:28.568 INFO:teuthology.orchestra.run.smithi080.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T23:46:28.568 INFO:teuthology.orchestra.run.smithi080.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T23:46:28.568 INFO:teuthology.orchestra.run.smithi080.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T23:46:28.568 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T23:46:28.568 INFO:teuthology.orchestra.run.smithi080.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T23:46:28.568 INFO:teuthology.orchestra.run.smithi080.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T23:46:28.568 INFO:teuthology.orchestra.run.smithi080.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T23:46:28.568 INFO:teuthology.orchestra.run.smithi080.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T23:46:28.568 INFO:teuthology.orchestra.run.smithi080.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T23:46:28.568 INFO:teuthology.orchestra.run.smithi080.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T23:46:28.568 INFO:teuthology.orchestra.run.smithi080.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T23:46:28.568 INFO:teuthology.orchestra.run.smithi080.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T23:46:28.569 INFO:teuthology.orchestra.run.smithi080.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T23:46:28.569 INFO:teuthology.orchestra.run.smithi080.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T23:46:28.569 INFO:teuthology.orchestra.run.smithi080.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T23:46:28.569 INFO:teuthology.orchestra.run.smithi080.stdout:Installing weak dependencies: 2024-04-26T23:46:28.569 INFO:teuthology.orchestra.run.smithi080.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T23:46:28.569 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:46:28.569 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-26T23:46:28.569 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================= 2024-04-26T23:46:28.569 INFO:teuthology.orchestra.run.smithi080.stdout:Install 148 Packages 2024-04-26T23:46:28.569 INFO:teuthology.orchestra.run.smithi080.stdout:Upgrade 4 Packages 2024-04-26T23:46:28.569 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:46:28.570 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 222 M 2024-04-26T23:46:28.570 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-26T23:46:28.897 INFO:teuthology.orchestra.run.smithi103.stdout:(6/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 13 MB/s | 4.7 MB 00:00 2024-04-26T23:46:29.197 INFO:teuthology.orchestra.run.smithi103.stdout:(7/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 2.9 MB/s | 5.4 MB 00:01 2024-04-26T23:46:29.534 INFO:teuthology.orchestra.run.smithi080.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-04-26T23:46:30.031 INFO:teuthology.orchestra.run.smithi103.stdout:(8/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x86 15 MB/s | 17 MB 00:01 2024-04-26T23:46:30.106 INFO:teuthology.orchestra.run.smithi103.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 336 kB/s | 25 kB 00:00 2024-04-26T23:46:30.143 INFO:teuthology.orchestra.run.smithi080.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.4 MB/s | 889 kB 00:00 2024-04-26T23:46:30.318 INFO:teuthology.orchestra.run.smithi080.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 829 kB/s | 145 kB 00:00 2024-04-26T23:46:31.040 INFO:teuthology.orchestra.run.smithi103.stdout:(10/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 5.7 MB/s | 11 MB 00:01 2024-04-26T23:46:31.148 INFO:teuthology.orchestra.run.smithi103.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.0 MB/s | 225 kB 00:00 2024-04-26T23:46:31.257 INFO:teuthology.orchestra.run.smithi103.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 249 kB/s | 27 kB 00:00 2024-04-26T23:46:31.374 INFO:teuthology.orchestra.run.smithi103.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 272 kB/s | 32 kB 00:00 2024-04-26T23:46:31.418 INFO:teuthology.orchestra.run.smithi080.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 2.1 MB/s | 2.4 MB 00:01 2024-04-26T23:46:31.566 INFO:teuthology.orchestra.run.smithi103.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 3.8 MB/s | 747 kB 00:00 2024-04-26T23:46:31.841 INFO:teuthology.orchestra.run.smithi103.stdout:(15/152): ceph-common-19.0.0-3306.gbc612ffb.el9 4.7 MB/s | 21 MB 00:04 2024-04-26T23:46:31.866 INFO:teuthology.orchestra.run.smithi103.stdout:(16/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 544 kB/s | 163 kB 00:00 2024-04-26T23:46:31.894 INFO:teuthology.orchestra.run.smithi080.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 3.2 MB/s | 1.5 MB 00:00 2024-04-26T23:46:31.917 INFO:teuthology.orchestra.run.smithi103.stdout:(17/152): librados-devel-19.0.0-3306.gbc612ffb. 1.7 MB/s | 127 kB 00:00 2024-04-26T23:46:32.000 INFO:teuthology.orchestra.run.smithi103.stdout:(18/152): libradosstriper1-19.0.0-3306.gbc612ff 3.7 MB/s | 501 kB 00:00 2024-04-26T23:46:32.101 INFO:teuthology.orchestra.run.smithi103.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 451 kB/s | 45 kB 00:00 2024-04-26T23:46:32.209 INFO:teuthology.orchestra.run.smithi103.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 1.2 MB/s | 135 kB 00:00 2024-04-26T23:46:32.318 INFO:teuthology.orchestra.run.smithi103.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.5 MB/s | 161 kB 00:00 2024-04-26T23:46:32.443 INFO:teuthology.orchestra.run.smithi103.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 2.5 MB/s | 321 kB 00:00 2024-04-26T23:46:32.552 INFO:teuthology.orchestra.run.smithi103.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 2.7 MB/s | 298 kB 00:00 2024-04-26T23:46:32.569 INFO:teuthology.orchestra.run.smithi080.stdout:(6/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 1.6 MB/s | 5.4 MB 00:03 2024-04-26T23:46:32.617 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 978 kB/s | 19 MB 00:19 2024-04-26T23:46:32.636 INFO:teuthology.orchestra.run.smithi103.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.2 MB/s | 99 kB 00:00 2024-04-26T23:46:32.728 INFO:teuthology.orchestra.run.smithi103.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 932 kB/s | 85 kB 00:00 2024-04-26T23:46:32.820 INFO:teuthology.orchestra.run.smithi103.stdout:(26/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 5.9 MB/s | 5.4 MB 00:00 2024-04-26T23:46:32.920 INFO:teuthology.orchestra.run.smithi103.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 1.7 MB/s | 171 kB 00:00 2024-04-26T23:46:33.020 INFO:teuthology.orchestra.run.smithi103.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 342 kB/s | 34 kB 00:00 2024-04-26T23:46:33.121 INFO:teuthology.orchestra.run.smithi103.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.4 MB/s | 142 kB 00:00 2024-04-26T23:46:33.313 INFO:teuthology.orchestra.run.smithi103.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 5.3 MB/s | 3.1 MB 00:00 2024-04-26T23:46:33.595 INFO:teuthology.orchestra.run.smithi080.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 2.8 MB/s | 4.7 MB 00:01 2024-04-26T23:46:33.788 INFO:teuthology.orchestra.run.smithi103.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 5.3 MB/s | 3.5 MB 00:00 2024-04-26T23:46:33.896 INFO:teuthology.orchestra.run.smithi103.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.3 MB/s | 249 kB 00:00 2024-04-26T23:46:33.988 INFO:teuthology.orchestra.run.smithi103.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 538 kB/s | 49 kB 00:00 2024-04-26T23:46:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 179 kB/s | 16 kB 00:00 2024-04-26T23:46:34.189 INFO:teuthology.orchestra.run.smithi103.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 2.5 MB/s | 273 kB 00:00 2024-04-26T23:46:34.381 INFO:teuthology.orchestra.run.smithi103.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 3.9 MB/s | 762 kB 00:00 2024-04-26T23:46:34.481 INFO:teuthology.orchestra.run.smithi103.stdout:(37/152): ceph-mgr-diskprediction-local-19.0.0- 6.3 MB/s | 7.4 MB 00:01 2024-04-26T23:46:34.506 INFO:teuthology.orchestra.run.smithi103.stdout:(38/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 146 kB/s | 18 kB 00:00 2024-04-26T23:46:34.957 INFO:teuthology.orchestra.run.smithi103.stdout:(39/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 10 MB/s | 49 MB 00:04 2024-04-26T23:46:35.082 INFO:teuthology.orchestra.run.smithi103.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 134 kB/s | 80 kB 00:00 2024-04-26T23:46:35.141 INFO:teuthology.orchestra.run.smithi103.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-04-26T23:46:35.200 INFO:teuthology.orchestra.run.smithi103.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-04-26T23:46:35.233 INFO:teuthology.orchestra.run.smithi103.stdout:(43/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-04-26T23:46:35.258 INFO:teuthology.orchestra.run.smithi103.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-26T23:46:35.292 INFO:teuthology.orchestra.run.smithi103.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-26T23:46:35.334 INFO:teuthology.orchestra.run.smithi103.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 6.4 MB/s | 1.2 MB 00:00 2024-04-26T23:46:35.344 INFO:teuthology.orchestra.run.smithi080.stdout:(8/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x86 6.1 MB/s | 17 MB 00:02 2024-04-26T23:46:35.359 INFO:teuthology.orchestra.run.smithi103.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-04-26T23:46:35.393 INFO:teuthology.orchestra.run.smithi103.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.1 MB/s | 215 kB 00:00 2024-04-26T23:46:35.435 INFO:teuthology.orchestra.run.smithi103.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 6.7 MB/s | 689 kB 00:00 2024-04-26T23:46:35.460 INFO:teuthology.orchestra.run.smithi103.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-26T23:46:35.502 INFO:teuthology.orchestra.run.smithi103.stdout:(51/152): bison-3.7.4-5.el9.x86_64.rpm 6.5 MB/s | 943 kB 00:00 2024-04-26T23:46:35.519 INFO:teuthology.orchestra.run.smithi103.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 255 kB/s | 21 kB 00:00 2024-04-26T23:46:35.544 INFO:teuthology.orchestra.run.smithi103.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 313 kB 00:00 2024-04-26T23:46:35.561 INFO:teuthology.orchestra.run.smithi103.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 554 kB/s | 33 kB 00:00 2024-04-26T23:46:35.595 INFO:teuthology.orchestra.run.smithi103.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 363 kB/s | 18 kB 00:00 2024-04-26T23:46:35.612 INFO:teuthology.orchestra.run.smithi103.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 221 kB/s | 11 kB 00:00 2024-04-26T23:46:35.636 INFO:teuthology.orchestra.run.smithi080.stdout:(9/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 3.3 MB/s | 21 MB 00:06 2024-04-26T23:46:35.654 INFO:teuthology.orchestra.run.smithi103.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-26T23:46:35.661 INFO:teuthology.orchestra.run.smithi080.stdout:(10/152): ceph-selinux-19.0.0-3306.gbc612ffb.el 79 kB/s | 25 kB 00:00 2024-04-26T23:46:35.679 INFO:teuthology.orchestra.run.smithi103.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 677 kB/s | 45 kB 00:00 2024-04-26T23:46:35.770 INFO:teuthology.orchestra.run.smithi080.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.0 MB/s | 225 kB 00:00 2024-04-26T23:46:35.771 INFO:teuthology.orchestra.run.smithi103.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-04-26T23:46:35.796 INFO:teuthology.orchestra.run.smithi103.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 662 kB 00:00 2024-04-26T23:46:35.830 INFO:teuthology.orchestra.run.smithi103.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 3.8 MB/s | 582 kB 00:00 2024-04-26T23:46:35.855 INFO:teuthology.orchestra.run.smithi103.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 304 kB/s | 25 kB 00:00 2024-04-26T23:46:35.881 INFO:teuthology.orchestra.run.smithi103.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-04-26T23:46:35.904 INFO:teuthology.orchestra.run.smithi080.stdout:(12/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 4.6 MB/s | 11 MB 00:02 2024-04-26T23:46:35.906 INFO:teuthology.orchestra.run.smithi103.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-26T23:46:35.929 INFO:teuthology.orchestra.run.smithi080.stdout:(13/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 171 kB/s | 27 kB 00:00 2024-04-26T23:46:35.931 INFO:teuthology.orchestra.run.smithi103.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-26T23:46:35.954 INFO:teuthology.orchestra.run.smithi080.stdout:(14/152): libcephfs-devel-19.0.0-3306.gbc612ffb 637 kB/s | 32 kB 00:00 2024-04-26T23:46:35.956 INFO:teuthology.orchestra.run.smithi103.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 467 kB/s | 35 kB 00:00 2024-04-26T23:46:36.005 INFO:teuthology.orchestra.run.smithi080.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 3.2 MB/s | 163 kB 00:00 2024-04-26T23:46:36.055 INFO:teuthology.orchestra.run.smithi080.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 2.5 MB/s | 127 kB 00:00 2024-04-26T23:46:36.065 INFO:teuthology.orchestra.run.smithi103.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-26T23:46:36.080 INFO:teuthology.orchestra.run.smithi080.stdout:(17/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 4.8 MB/s | 747 kB 00:00 2024-04-26T23:46:36.149 INFO:teuthology.orchestra.run.smithi103.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-26T23:46:36.164 INFO:teuthology.orchestra.run.smithi080.stdout:(18/152): libradosstriper1-19.0.0-3306.gbc612ff 4.5 MB/s | 501 kB 00:00 2024-04-26T23:46:36.207 INFO:teuthology.orchestra.run.smithi103.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-26T23:46:36.222 INFO:teuthology.orchestra.run.smithi080.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 774 kB/s | 45 kB 00:00 2024-04-26T23:46:36.274 INFO:teuthology.orchestra.run.smithi103.stdout:(70/152): openblas-openmp-0.3.21-2.el9.x86_64.r 13 MB/s | 4.7 MB 00:00 2024-04-26T23:46:36.281 INFO:teuthology.orchestra.run.smithi080.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 2.3 MB/s | 135 kB 00:00 2024-04-26T23:46:36.341 INFO:teuthology.orchestra.run.smithi103.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-26T23:46:36.348 INFO:teuthology.orchestra.run.smithi080.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 2.4 MB/s | 161 kB 00:00 2024-04-26T23:46:36.408 INFO:teuthology.orchestra.run.smithi103.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 7.6 MB/s | 1.0 MB 00:00 2024-04-26T23:46:36.425 INFO:teuthology.orchestra.run.smithi103.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-04-26T23:46:36.440 INFO:teuthology.orchestra.run.smithi080.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 3.4 MB/s | 321 kB 00:00 2024-04-26T23:46:36.484 INFO:teuthology.orchestra.run.smithi103.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-26T23:46:36.532 INFO:teuthology.orchestra.run.smithi080.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 3.2 MB/s | 298 kB 00:00 2024-04-26T23:46:36.584 INFO:teuthology.orchestra.run.smithi103.stdout:(75/152): postgresql-server-13.14-1.el9.x86_64. 15 MB/s | 5.8 MB 00:00 2024-04-26T23:46:36.591 INFO:teuthology.orchestra.run.smithi080.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.7 MB/s | 99 kB 00:00 2024-04-26T23:46:36.609 INFO:teuthology.orchestra.run.smithi103.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-04-26T23:46:36.641 INFO:teuthology.orchestra.run.smithi080.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-26T23:46:36.643 INFO:teuthology.orchestra.run.smithi103.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 438 kB/s | 26 kB 00:00 2024-04-26T23:46:36.668 INFO:teuthology.orchestra.run.smithi103.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 326 kB/s | 19 kB 00:00 2024-04-26T23:46:36.702 INFO:teuthology.orchestra.run.smithi103.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-26T23:46:36.727 INFO:teuthology.orchestra.run.smithi103.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 593 kB/s | 35 kB 00:00 2024-04-26T23:46:36.791 INFO:teuthology.orchestra.run.smithi080.stdout:(26/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 7.6 MB/s | 5.4 MB 00:00 2024-04-26T23:46:36.803 INFO:teuthology.orchestra.run.smithi103.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-04-26T23:46:36.828 INFO:teuthology.orchestra.run.smithi103.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-04-26T23:46:36.850 INFO:teuthology.orchestra.run.smithi080.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.9 MB/s | 171 kB 00:00 2024-04-26T23:46:36.862 INFO:teuthology.orchestra.run.smithi103.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-26T23:46:36.887 INFO:teuthology.orchestra.run.smithi103.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-26T23:46:36.912 INFO:teuthology.orchestra.run.smithi103.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-04-26T23:46:36.917 INFO:teuthology.orchestra.run.smithi080.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 511 kB/s | 34 kB 00:00 2024-04-26T23:46:36.945 INFO:teuthology.orchestra.run.smithi103.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-26T23:46:36.971 INFO:teuthology.orchestra.run.smithi103.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-26T23:46:36.992 INFO:teuthology.orchestra.run.smithi080.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.9 MB/s | 142 kB 00:00 2024-04-26T23:46:37.038 INFO:teuthology.orchestra.run.smithi103.stdout:(88/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 5.1 MB 00:00 2024-04-26T23:46:37.055 INFO:teuthology.orchestra.run.smithi103.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 609 kB/s | 51 kB 00:00 2024-04-26T23:46:37.080 INFO:teuthology.orchestra.run.smithi103.stdout:(90/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.0 MB/s | 279 kB 00:00 2024-04-26T23:46:37.097 INFO:teuthology.orchestra.run.smithi103.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 912 kB/s | 54 kB 00:00 2024-04-26T23:46:37.130 INFO:teuthology.orchestra.run.smithi103.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-04-26T23:46:37.147 INFO:teuthology.orchestra.run.smithi103.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 879 kB/s | 44 kB 00:00 2024-04-26T23:46:37.168 INFO:teuthology.orchestra.run.smithi080.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 5.9 MB/s | 3.1 MB 00:00 2024-04-26T23:46:37.184 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-26T23:46:37.189 INFO:teuthology.orchestra.run.smithi103.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-26T23:46:37.215 INFO:teuthology.orchestra.run.smithi103.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 534 kB/s | 36 kB 00:00 2024-04-26T23:46:37.240 INFO:teuthology.orchestra.run.smithi103.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-04-26T23:46:37.274 INFO:teuthology.orchestra.run.smithi103.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-26T23:46:37.299 INFO:teuthology.orchestra.run.smithi103.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 984 kB/s | 58 kB 00:00 2024-04-26T23:46:37.333 INFO:teuthology.orchestra.run.smithi103.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-26T23:46:37.358 INFO:teuthology.orchestra.run.smithi103.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-26T23:46:37.391 INFO:teuthology.orchestra.run.smithi103.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 764 kB/s | 45 kB 00:00 2024-04-26T23:46:37.401 INFO:teuthology.orchestra.run.smithi080.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 8.7 MB/s | 3.5 MB 00:00 2024-04-26T23:46:37.467 INFO:teuthology.orchestra.run.smithi103.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-04-26T23:46:37.477 INFO:teuthology.orchestra.run.smithi080.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 3.2 MB/s | 249 kB 00:00 2024-04-26T23:46:37.544 INFO:teuthology.orchestra.run.smithi080.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 742 kB/s | 49 kB 00:00 2024-04-26T23:46:37.602 INFO:teuthology.orchestra.run.smithi080.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 281 kB/s | 16 kB 00:00 2024-04-26T23:46:37.694 INFO:teuthology.orchestra.run.smithi080.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 2.9 MB/s | 273 kB 00:00 2024-04-26T23:46:37.717 INFO:teuthology.orchestra.run.smithi103.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-04-26T23:46:37.793 INFO:teuthology.orchestra.run.smithi103.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-04-26T23:46:37.828 INFO:teuthology.orchestra.run.smithi080.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 5.6 MB/s | 762 kB 00:00 2024-04-26T23:46:37.895 INFO:teuthology.orchestra.run.smithi080.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 273 kB/s | 18 kB 00:00 2024-04-26T23:46:38.043 INFO:teuthology.orchestra.run.smithi103.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 952 kB/s | 548 kB 00:00 2024-04-26T23:46:38.076 INFO:teuthology.orchestra.run.smithi103.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 76 kB/s | 19 kB 00:00 2024-04-26T23:46:38.087 INFO:teuthology.orchestra.run.smithi080.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 420 kB/s | 80 kB 00:00 2024-04-26T23:46:38.168 INFO:teuthology.orchestra.run.smithi103.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-04-26T23:46:38.193 INFO:teuthology.orchestra.run.smithi103.stdout:(108/152): gperftools-libs-2.9.1-2.el9.x86_64.r 651 kB/s | 309 kB 00:00 2024-04-26T23:46:38.245 INFO:teuthology.orchestra.run.smithi080.stdout:(39/152): ceph-mgr-diskprediction-local-19.0.0- 6.9 MB/s | 7.4 MB 00:01 2024-04-26T23:46:38.279 INFO:teuthology.orchestra.run.smithi080.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 803 kB 00:00 2024-04-26T23:46:38.321 INFO:teuthology.orchestra.run.smithi080.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-04-26T23:46:38.336 INFO:teuthology.orchestra.run.smithi103.stdout:(109/152): libarrow-9.0.0-11.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-04-26T23:46:38.361 INFO:teuthology.orchestra.run.smithi103.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 402 kB/s | 67 kB 00:00 2024-04-26T23:46:38.386 INFO:teuthology.orchestra.run.smithi103.stdout:(111/152): liboath-2.6.7-2.el9.x86_64.rpm 223 kB/s | 49 kB 00:00 2024-04-26T23:46:38.412 INFO:teuthology.orchestra.run.smithi103.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-26T23:46:38.446 INFO:teuthology.orchestra.run.smithi080.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-04-26T23:46:38.479 INFO:teuthology.orchestra.run.smithi103.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 651 kB/s | 43 kB 00:00 2024-04-26T23:46:38.513 INFO:teuthology.orchestra.run.smithi080.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-26T23:46:38.529 INFO:teuthology.orchestra.run.smithi103.stdout:(114/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.9 MB/s | 839 kB 00:00 2024-04-26T23:46:38.547 INFO:teuthology.orchestra.run.smithi080.stdout:(44/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 946 kB/s | 253 kB 00:00 2024-04-26T23:46:38.554 INFO:teuthology.orchestra.run.smithi103.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-04-26T23:46:38.589 INFO:teuthology.orchestra.run.smithi080.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-04-26T23:46:38.596 INFO:teuthology.orchestra.run.smithi103.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 225 kB/s | 15 kB 00:00 2024-04-26T23:46:38.621 INFO:teuthology.orchestra.run.smithi103.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-26T23:46:38.622 INFO:teuthology.orchestra.run.smithi080.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-26T23:46:38.647 INFO:teuthology.orchestra.run.smithi103.stdout:(118/152): python3-asyncssh-2.13.2-5.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-04-26T23:46:38.680 INFO:teuthology.orchestra.run.smithi103.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-26T23:46:38.681 INFO:teuthology.orchestra.run.smithi080.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-04-26T23:46:38.705 INFO:teuthology.orchestra.run.smithi103.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-04-26T23:46:38.731 INFO:teuthology.orchestra.run.smithi103.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 421 kB/s | 35 kB 00:00 2024-04-26T23:46:38.756 INFO:teuthology.orchestra.run.smithi103.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-26T23:46:38.798 INFO:teuthology.orchestra.run.smithi080.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 3.8 MB/s | 689 kB 00:00 2024-04-26T23:46:38.806 INFO:teuthology.orchestra.run.smithi103.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-04-26T23:46:38.832 INFO:teuthology.orchestra.run.smithi103.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-04-26T23:46:38.832 INFO:teuthology.orchestra.run.smithi080.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 6.1 MB/s | 943 kB 00:00 2024-04-26T23:46:38.857 INFO:teuthology.orchestra.run.smithi103.stdout:(125/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-26T23:46:38.874 INFO:teuthology.orchestra.run.smithi103.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 263 kB/s | 18 kB 00:00 2024-04-26T23:46:38.891 INFO:teuthology.orchestra.run.smithi080.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 367 kB/s | 21 kB 00:00 2024-04-26T23:46:38.899 INFO:teuthology.orchestra.run.smithi103.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 344 kB/s | 23 kB 00:00 2024-04-26T23:46:38.916 INFO:teuthology.orchestra.run.smithi080.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 912 kB/s | 107 kB 00:00 2024-04-26T23:46:38.924 INFO:teuthology.orchestra.run.smithi103.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 289 kB/s | 19 kB 00:00 2024-04-26T23:46:38.950 INFO:teuthology.orchestra.run.smithi103.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-04-26T23:46:38.983 INFO:teuthology.orchestra.run.smithi080.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 493 kB/s | 33 kB 00:00 2024-04-26T23:46:38.992 INFO:teuthology.orchestra.run.smithi103.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 691 kB/s | 46 kB 00:00 2024-04-26T23:46:39.017 INFO:teuthology.orchestra.run.smithi103.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-04-26T23:46:39.025 INFO:teuthology.orchestra.run.smithi080.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-04-26T23:46:39.051 INFO:teuthology.orchestra.run.smithi103.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-04-26T23:46:39.068 INFO:teuthology.orchestra.run.smithi103.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 764 kB/s | 58 kB 00:00 2024-04-26T23:46:39.084 INFO:teuthology.orchestra.run.smithi080.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 311 kB/s | 18 kB 00:00 2024-04-26T23:46:39.093 INFO:teuthology.orchestra.run.smithi103.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-26T23:46:39.102 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-04-26T23:46:39.118 INFO:teuthology.orchestra.run.smithi103.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-04-26T23:46:39.142 INFO:teuthology.orchestra.run.smithi080.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-04-26T23:46:39.143 INFO:teuthology.orchestra.run.smithi103.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-26T23:46:39.169 INFO:teuthology.orchestra.run.smithi103.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-04-26T23:46:39.194 INFO:teuthology.orchestra.run.smithi103.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-04-26T23:46:39.209 INFO:teuthology.orchestra.run.smithi080.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 160 kB 00:00 2024-04-26T23:46:39.228 INFO:teuthology.orchestra.run.smithi103.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-26T23:46:39.253 INFO:teuthology.orchestra.run.smithi103.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 701 kB/s | 59 kB 00:00 2024-04-26T23:46:39.278 INFO:teuthology.orchestra.run.smithi103.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 425 kB/s | 36 kB 00:00 2024-04-26T23:46:39.285 INFO:teuthology.orchestra.run.smithi080.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 603 kB/s | 45 kB 00:00 2024-04-26T23:46:39.303 INFO:teuthology.orchestra.run.smithi103.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-04-26T23:46:39.329 INFO:teuthology.orchestra.run.smithi103.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-26T23:46:39.343 INFO:teuthology.orchestra.run.smithi080.stdout:(58/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.3 MB/s | 3.0 MB 00:00 2024-04-26T23:46:39.354 INFO:teuthology.orchestra.run.smithi103.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-26T23:46:39.396 INFO:teuthology.orchestra.run.smithi103.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 298 kB/s | 20 kB 00:00 2024-04-26T23:46:39.421 INFO:teuthology.orchestra.run.smithi103.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-04-26T23:46:39.444 INFO:teuthology.orchestra.run.smithi080.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-04-26T23:46:39.447 INFO:teuthology.orchestra.run.smithi103.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-26T23:46:39.469 INFO:teuthology.orchestra.run.smithi080.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 4.5 MB/s | 582 kB 00:00 2024-04-26T23:46:39.489 INFO:teuthology.orchestra.run.smithi103.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-26T23:46:39.511 INFO:teuthology.orchestra.run.smithi080.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 380 kB/s | 25 kB 00:00 2024-04-26T23:46:39.536 INFO:teuthology.orchestra.run.smithi080.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 709 kB/s | 47 kB 00:00 2024-04-26T23:46:39.603 INFO:teuthology.orchestra.run.smithi080.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-04-26T23:46:39.629 INFO:teuthology.orchestra.run.smithi080.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-04-26T23:46:39.670 INFO:teuthology.orchestra.run.smithi080.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 528 kB/s | 35 kB 00:00 2024-04-26T23:46:39.747 INFO:teuthology.orchestra.run.smithi103.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 5.2 MB/s | 1.3 MB 00:00 2024-04-26T23:46:39.798 INFO:teuthology.orchestra.run.smithi103.stdout:(150/152): librados2-19.0.0-3306.gbc612ffb.el9. 9.0 MB/s | 3.4 MB 00:00 2024-04-26T23:46:39.940 INFO:teuthology.orchestra.run.smithi103.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-04-26T23:46:40.006 INFO:teuthology.orchestra.run.smithi103.stdout:(152/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 5.6 MB/s | 3.1 MB 00:00 2024-04-26T23:46:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-04-26T23:46:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:Total 17 MB/s | 222 MB 00:13 2024-04-26T23:46:40.037 INFO:teuthology.orchestra.run.smithi080.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-04-26T23:46:40.096 INFO:teuthology.orchestra.run.smithi080.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-26T23:46:40.246 INFO:teuthology.orchestra.run.smithi080.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-04-26T23:46:40.272 INFO:teuthology.orchestra.run.smithi080.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 5.1 MB/s | 911 kB 00:00 2024-04-26T23:46:40.330 INFO:teuthology.orchestra.run.smithi080.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-26T23:46:40.522 INFO:teuthology.orchestra.run.smithi080.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-04-26T23:46:40.537 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-26T23:46:40.597 INFO:teuthology.orchestra.run.smithi080.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-04-26T23:46:40.698 INFO:teuthology.orchestra.run.smithi080.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 14 MB/s | 5.8 MB 00:00 2024-04-26T23:46:40.781 INFO:teuthology.orchestra.run.smithi080.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-04-26T23:46:40.873 INFO:teuthology.orchestra.run.smithi080.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-04-26T23:46:40.932 INFO:teuthology.orchestra.run.smithi080.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 439 kB/s | 26 kB 00:00 2024-04-26T23:46:40.982 INFO:teuthology.orchestra.run.smithi080.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-04-26T23:46:41.111 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-04-26T23:46:41.116 INFO:teuthology.orchestra.run.smithi080.stdout:(78/152): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-04-26T23:46:41.141 INFO:teuthology.orchestra.run.smithi080.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-26T23:46:41.183 INFO:teuthology.orchestra.run.smithi080.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 520 kB/s | 35 kB 00:00 2024-04-26T23:46:41.256 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-04-26T23:46:41.256 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-04-26T23:46:41.267 INFO:teuthology.orchestra.run.smithi080.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-04-26T23:46:41.351 INFO:teuthology.orchestra.run.smithi080.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-04-26T23:46:41.418 INFO:teuthology.orchestra.run.smithi080.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-04-26T23:46:41.501 INFO:teuthology.orchestra.run.smithi080.stdout:(84/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-04-26T23:46:41.526 INFO:teuthology.orchestra.run.smithi080.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 389 kB/s | 42 kB 00:00 2024-04-26T23:46:41.593 INFO:teuthology.orchestra.run.smithi080.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 2.9 MB/s | 269 kB 00:00 2024-04-26T23:46:41.619 INFO:teuthology.orchestra.run.smithi080.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-26T23:46:41.677 INFO:teuthology.orchestra.run.smithi080.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 875 kB/s | 51 kB 00:00 2024-04-26T23:46:41.702 INFO:teuthology.orchestra.run.smithi080.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-04-26T23:46:41.753 INFO:teuthology.orchestra.run.smithi080.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 714 kB/s | 54 kB 00:00 2024-04-26T23:46:41.812 INFO:teuthology.orchestra.run.smithi080.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 712 kB/s | 42 kB 00:00 2024-04-26T23:46:41.870 INFO:teuthology.orchestra.run.smithi080.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 760 kB/s | 44 kB 00:00 2024-04-26T23:46:41.945 INFO:teuthology.orchestra.run.smithi080.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-04-26T23:46:41.996 INFO:teuthology.orchestra.run.smithi080.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-04-26T23:46:42.054 INFO:teuthology.orchestra.run.smithi080.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 816 kB/s | 47 kB 00:00 2024-04-26T23:46:42.606 INFO:teuthology.orchestra.run.smithi080.stdout:(96/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 7.0 MB/s | 49 MB 00:06 2024-04-26T23:46:42.772 INFO:teuthology.orchestra.run.smithi080.stdout:(97/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-04-26T23:46:42.797 INFO:teuthology.orchestra.run.smithi080.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 410 kB/s | 305 kB 00:00 2024-04-26T23:46:42.822 INFO:teuthology.orchestra.run.smithi080.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 268 kB/s | 58 kB 00:00 2024-04-26T23:46:42.856 INFO:teuthology.orchestra.run.smithi080.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.1 MB/s | 94 kB 00:00 2024-04-26T23:46:42.881 INFO:teuthology.orchestra.run.smithi080.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 760 kB/s | 64 kB 00:00 2024-04-26T23:46:42.906 INFO:teuthology.orchestra.run.smithi080.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 535 kB/s | 45 kB 00:00 2024-04-26T23:46:43.031 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-04-26T23:46:43.031 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-04-26T23:46:43.432 INFO:teuthology.orchestra.run.smithi080.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-04-26T23:46:43.615 INFO:teuthology.orchestra.run.smithi080.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 773 kB/s | 548 kB 00:00 2024-04-26T23:46:43.740 INFO:teuthology.orchestra.run.smithi080.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 157 kB/s | 19 kB 00:00 2024-04-26T23:46:44.040 INFO:teuthology.orchestra.run.smithi080.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 508 kB/s | 309 kB 00:00 2024-04-26T23:46:44.157 INFO:teuthology.orchestra.run.smithi080.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-04-26T23:46:44.233 INFO:teuthology.orchestra.run.smithi080.stdout:(108/152): libarrow-9.0.0-11.el9.x86_64.rpm 9.0 MB/s | 4.4 MB 00:00 2024-04-26T23:46:44.267 INFO:teuthology.orchestra.run.smithi080.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 446 kB/s | 49 kB 00:00 2024-04-26T23:46:44.342 INFO:teuthology.orchestra.run.smithi080.stdout:(110/152): protobuf-compiler-3.14.0-13.el9.x86_ 591 kB/s | 863 kB 00:01 2024-04-26T23:46:44.367 INFO:teuthology.orchestra.run.smithi080.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 503 kB/s | 67 kB 00:00 2024-04-26T23:46:44.401 INFO:teuthology.orchestra.run.smithi080.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-04-26T23:46:44.561 INFO:teuthology.orchestra.run.smithi080.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-04-26T23:46:44.593 INFO:teuthology.orchestra.run.smithi080.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 226 kB/s | 43 kB 00:00 2024-04-26T23:46:44.660 INFO:teuthology.orchestra.run.smithi080.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 324 kB/s | 32 kB 00:00 2024-04-26T23:46:44.694 INFO:teuthology.orchestra.run.smithi080.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-26T23:46:44.769 INFO:teuthology.orchestra.run.smithi080.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.6 MB/s | 172 kB 00:00 2024-04-26T23:46:44.870 INFO:teuthology.orchestra.run.smithi080.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.0 MB/s | 358 kB 00:00 2024-04-26T23:46:44.894 INFO:teuthology.orchestra.run.smithi080.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-04-26T23:46:44.986 INFO:teuthology.orchestra.run.smithi080.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 302 kB/s | 35 kB 00:00 2024-04-26T23:46:45.012 INFO:teuthology.orchestra.run.smithi080.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.7 MB/s | 206 kB 00:00 2024-04-26T23:46:45.054 INFO:teuthology.orchestra.run.smithi080.stdout:(122/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.2 MB/s | 839 kB 00:00 2024-04-26T23:46:45.121 INFO:teuthology.orchestra.run.smithi080.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-04-26T23:46:45.154 INFO:teuthology.orchestra.run.smithi080.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-04-26T23:46:45.229 INFO:teuthology.orchestra.run.smithi080.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-04-26T23:46:45.263 INFO:teuthology.orchestra.run.smithi080.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-04-26T23:46:45.322 INFO:teuthology.orchestra.run.smithi080.stdout:(127/152): python3-grpcio-1.46.7-10.el9.x86_64. 6.1 MB/s | 2.0 MB 00:00 2024-04-26T23:46:45.347 INFO:teuthology.orchestra.run.smithi080.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-04-26T23:46:45.372 INFO:teuthology.orchestra.run.smithi080.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 180 kB/s | 20 kB 00:00 2024-04-26T23:46:45.464 INFO:teuthology.orchestra.run.smithi080.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-04-26T23:46:45.489 INFO:teuthology.orchestra.run.smithi080.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 326 kB/s | 46 kB 00:00 2024-04-26T23:46:45.515 INFO:teuthology.orchestra.run.smithi080.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 554 kB/s | 79 kB 00:00 2024-04-26T23:46:45.573 INFO:teuthology.orchestra.run.smithi080.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 531 kB/s | 58 kB 00:00 2024-04-26T23:46:45.599 INFO:teuthology.orchestra.run.smithi080.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-26T23:46:45.624 INFO:teuthology.orchestra.run.smithi080.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-26T23:46:45.682 INFO:teuthology.orchestra.run.smithi080.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 826 kB/s | 90 kB 00:00 2024-04-26T23:46:45.708 INFO:teuthology.orchestra.run.smithi080.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 259 kB/s | 28 kB 00:00 2024-04-26T23:46:45.733 INFO:teuthology.orchestra.run.smithi080.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 282 kB/s | 31 kB 00:00 2024-04-26T23:46:45.792 INFO:teuthology.orchestra.run.smithi080.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-26T23:46:45.817 INFO:teuthology.orchestra.run.smithi080.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 543 kB/s | 59 kB 00:00 2024-04-26T23:46:45.842 INFO:teuthology.orchestra.run.smithi080.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 328 kB/s | 36 kB 00:00 2024-04-26T23:46:45.901 INFO:teuthology.orchestra.run.smithi080.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 469 kB/s | 51 kB 00:00 2024-04-26T23:46:45.926 INFO:teuthology.orchestra.run.smithi080.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-26T23:46:45.952 INFO:teuthology.orchestra.run.smithi080.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 823 kB/s | 90 kB 00:00 2024-04-26T23:46:46.019 INFO:teuthology.orchestra.run.smithi080.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-26T23:46:46.044 INFO:teuthology.orchestra.run.smithi080.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 170 kB/s | 20 kB 00:00 2024-04-26T23:46:46.069 INFO:teuthology.orchestra.run.smithi080.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-04-26T23:46:46.194 INFO:teuthology.orchestra.run.smithi080.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-04-26T23:46:46.536 INFO:teuthology.orchestra.run.smithi080.stdout:(149/152): librados2-19.0.0-3306.gbc612ffb.el9. 6.9 MB/s | 3.4 MB 00:00 2024-04-26T23:46:46.612 INFO:teuthology.orchestra.run.smithi080.stdout:(150/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 5.8 MB/s | 3.1 MB 00:00 2024-04-26T23:46:46.629 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-26T23:46:46.654 INFO:teuthology.orchestra.run.smithi080.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 2.9 MB/s | 1.3 MB 00:00 2024-04-26T23:46:46.979 INFO:teuthology.orchestra.run.smithi080.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 5.4 MB/s | 2.4 MB 00:00 2024-04-26T23:46:46.983 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-26T23:46:46.983 INFO:teuthology.orchestra.run.smithi080.stdout:Total 12 MB/s | 222 MB 00:18 2024-04-26T23:46:48.048 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-26T23:46:48.189 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-26T23:46:48.189 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-26T23:46:48.988 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-04-26T23:46:49.088 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T23:46:49.089 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T23:46:49.110 INFO:teuthology.orchestra.run.smithi040.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T23:46:49.111 INFO:teuthology.orchestra.run.smithi040.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T23:46:49.113 INFO:teuthology.orchestra.run.smithi040.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T23:46:49.114 INFO:teuthology.orchestra.run.smithi040.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T23:46:49.116 INFO:teuthology.orchestra.run.smithi040.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T23:46:49.120 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T23:46:49.121 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T23:46:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T23:46:49.196 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-04-26T23:46:49.217 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-26T23:46:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-04-26T23:46:49.217 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-26T23:46:49.217 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-04-26T23:46:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T23:46:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T23:46:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T23:46:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T23:46:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T23:46:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T23:46:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T23:46:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T23:46:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T23:46:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T23:46:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T23:46:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T23:46:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T23:46:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T23:46:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T23:46:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T23:46:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T23:46:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T23:46:49.220 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-04-26T23:46:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T23:46:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T23:46:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T23:46:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T23:46:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T23:46:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T23:46:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T23:46:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T23:46:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T23:46:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T23:46:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T23:46:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T23:46:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T23:46:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T23:46:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T23:46:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T23:46:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T23:46:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T23:46:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T23:46:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T23:46:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T23:46:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T23:46:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T23:46:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T23:46:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T23:46:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T23:46:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T23:46:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T23:46:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T23:46:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T23:46:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T23:46:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T23:46:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T23:46:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T23:46:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T23:46:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T23:46:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T23:46:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T23:46:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T23:46:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T23:46:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T23:46:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T23:46:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T23:46:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T23:46:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T23:46:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T23:46:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T23:46:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T23:46:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T23:46:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T23:46:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T23:46:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T23:46:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T23:46:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T23:46:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T23:46:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T23:46:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T23:46:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T23:46:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T23:46:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T23:46:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T23:46:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T23:46:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T23:46:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T23:46:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T23:46:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T23:46:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T23:46:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T23:46:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T23:46:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T23:46:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T23:46:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T23:46:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T23:46:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T23:46:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T23:46:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T23:46:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T23:46:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T23:46:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T23:46:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T23:46:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T23:46:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T23:46:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T23:46:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T23:46:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T23:46:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T23:46:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T23:46:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T23:46:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T23:46:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T23:46:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T23:46:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T23:46:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T23:46:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T23:46:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T23:46:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T23:46:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T23:46:49.233 INFO:teuthology.orchestra.run.smithi040.stdout:Installing weak dependencies: 2024-04-26T23:46:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T23:46:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:46:49.234 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-04-26T23:46:49.234 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-04-26T23:46:49.234 INFO:teuthology.orchestra.run.smithi040.stdout:Install 148 Packages 2024-04-26T23:46:49.234 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 4 Packages 2024-04-26T23:46:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:46:49.235 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 222 M 2024-04-26T23:46:49.235 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-04-26T23:46:49.360 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T23:46:49.413 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T23:46:49.454 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T23:46:49.479 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T23:46:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T23:46:49.851 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T23:46:49.955 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-26T23:46:49.956 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-26T23:46:49.980 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:46:50.101 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:46:50.131 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:46:50.222 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:46:50.252 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T23:46:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T23:46:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T23:46:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:46:50.580 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:46:50.603 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T23:46:50.666 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T23:46:50.687 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T23:46:50.708 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T23:46:50.781 INFO:teuthology.orchestra.run.smithi040.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-04-26T23:46:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T23:46:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T23:46:51.037 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T23:46:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T23:46:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T23:46:51.199 INFO:teuthology.orchestra.run.smithi040.stdout:(2/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 7.0 MB/s | 5.4 MB 00:00 2024-04-26T23:46:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T23:46:51.257 INFO:teuthology.orchestra.run.smithi040.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 2.5 MB/s | 145 kB 00:00 2024-04-26T23:46:51.277 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T23:46:51.321 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T23:46:51.340 INFO:teuthology.orchestra.run.smithi040.stdout:(4/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.6 MB/s | 889 kB 00:00 2024-04-26T23:46:51.365 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T23:46:51.397 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T23:46:51.429 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T23:46:51.474 INFO:teuthology.orchestra.run.smithi040.stdout:(5/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 11 MB/s | 2.4 MB 00:00 2024-04-26T23:46:51.515 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T23:46:51.572 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T23:46:51.603 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T23:46:51.639 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T23:46:51.716 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T23:46:51.749 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T23:46:51.793 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T23:46:51.847 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T23:46:51.889 INFO:teuthology.orchestra.run.smithi040.stdout:(6/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 12 MB/s | 4.7 MB 00:00 2024-04-26T23:46:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T23:46:51.941 INFO:teuthology.orchestra.run.smithi040.stdout:(7/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 2.5 MB/s | 1.5 MB 00:00 2024-04-26T23:46:51.970 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T23:46:52.044 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T23:46:52.185 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T23:46:52.238 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T23:46:52.770 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T23:46:52.858 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T23:46:52.888 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T23:46:52.931 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T23:46:52.958 INFO:teuthology.orchestra.run.smithi040.stdout:(8/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 8.4 MB/s | 21 MB 00:02 2024-04-26T23:46:52.969 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T23:46:53.017 INFO:teuthology.orchestra.run.smithi040.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 441 kB/s | 25 kB 00:00 2024-04-26T23:46:53.019 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T23:46:53.184 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T23:46:53.230 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T23:46:53.285 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T23:46:53.309 INFO:teuthology.orchestra.run.smithi040.stdout:(10/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 12 MB/s | 17 MB 00:01 2024-04-26T23:46:53.358 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T23:46:53.384 INFO:teuthology.orchestra.run.smithi040.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 3.0 MB/s | 225 kB 00:00 2024-04-26T23:46:53.389 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T23:46:53.426 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T23:46:53.434 INFO:teuthology.orchestra.run.smithi040.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 552 kB/s | 27 kB 00:00 2024-04-26T23:46:53.472 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T23:46:53.509 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T23:46:53.509 INFO:teuthology.orchestra.run.smithi040.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 428 kB/s | 32 kB 00:00 2024-04-26T23:46:53.551 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T23:46:53.594 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T23:46:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T23:46:53.775 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T23:46:53.813 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T23:46:53.854 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T23:46:53.918 INFO:teuthology.orchestra.run.smithi040.stdout:(14/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 5.4 MB/s | 11 MB 00:01 2024-04-26T23:46:53.948 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T23:46:53.951 INFO:teuthology.orchestra.run.smithi040.stdout:(15/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 1.7 MB/s | 747 kB 00:00 2024-04-26T23:46:53.985 INFO:teuthology.orchestra.run.smithi040.stdout:(16/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 2.4 MB/s | 163 kB 00:00 2024-04-26T23:46:53.988 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T23:46:54.028 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T23:46:54.054 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T23:46:54.060 INFO:teuthology.orchestra.run.smithi040.stdout:(17/152): librados-devel-19.0.0-3306.gbc612ffb. 1.2 MB/s | 127 kB 00:00 2024-04-26T23:46:54.078 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T23:46:54.110 INFO:teuthology.orchestra.run.smithi040.stdout:(18/152): libradosstriper1-19.0.0-3306.gbc612ff 3.9 MB/s | 501 kB 00:00 2024-04-26T23:46:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T23:46:54.194 INFO:teuthology.orchestra.run.smithi040.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 543 kB/s | 45 kB 00:00 2024-04-26T23:46:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T23:46:54.294 INFO:teuthology.orchestra.run.smithi040.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 1.3 MB/s | 135 kB 00:00 2024-04-26T23:46:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T23:46:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T23:46:54.386 INFO:teuthology.orchestra.run.smithi040.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.7 MB/s | 161 kB 00:00 2024-04-26T23:46:54.390 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T23:46:54.429 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T23:46:54.461 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T23:46:54.491 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T23:46:54.503 INFO:teuthology.orchestra.run.smithi040.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 2.7 MB/s | 321 kB 00:00 2024-04-26T23:46:54.618 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T23:46:54.654 INFO:teuthology.orchestra.run.smithi040.stdout:(23/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 9.1 MB/s | 5.4 MB 00:00 2024-04-26T23:46:54.687 INFO:teuthology.orchestra.run.smithi040.stdout:(24/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 1.6 MB/s | 298 kB 00:00 2024-04-26T23:46:54.721 INFO:teuthology.orchestra.run.smithi040.stdout:(25/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.5 MB/s | 99 kB 00:00 2024-04-26T23:46:54.754 INFO:teuthology.orchestra.run.smithi040.stdout:(26/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.3 MB/s | 85 kB 00:00 2024-04-26T23:46:54.827 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T23:46:54.838 INFO:teuthology.orchestra.run.smithi040.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.0 MB/s | 171 kB 00:00 2024-04-26T23:46:54.869 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T23:46:54.902 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T23:46:54.938 INFO:teuthology.orchestra.run.smithi040.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 344 kB/s | 34 kB 00:00 2024-04-26T23:46:54.985 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T23:46:55.033 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T23:46:55.064 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T23:46:55.072 INFO:teuthology.orchestra.run.smithi040.stdout:(29/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 8.8 MB/s | 3.1 MB 00:00 2024-04-26T23:46:55.106 INFO:teuthology.orchestra.run.smithi040.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 855 kB/s | 142 kB 00:00 2024-04-26T23:46:55.111 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T23:46:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T23:46:55.258 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T23:46:55.656 INFO:teuthology.orchestra.run.smithi040.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 6.1 MB/s | 3.5 MB 00:00 2024-04-26T23:46:55.740 INFO:teuthology.orchestra.run.smithi040.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.9 MB/s | 249 kB 00:00 2024-04-26T23:46:55.915 INFO:teuthology.orchestra.run.smithi040.stdout:(33/152): ceph-mgr-diskprediction-local-19.0.0- 9.2 MB/s | 7.4 MB 00:00 2024-04-26T23:46:55.934 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-26T23:46:55.940 INFO:teuthology.orchestra.run.smithi040.stdout:(34/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 247 kB/s | 49 kB 00:00 2024-04-26T23:46:55.975 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T23:46:55.982 INFO:teuthology.orchestra.run.smithi040.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3306.gb 249 kB/s | 16 kB 00:00 2024-04-26T23:46:56.016 INFO:teuthology.orchestra.run.smithi040.stdout:(36/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 3.6 MB/s | 273 kB 00:00 2024-04-26T23:46:56.021 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T23:46:56.023 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T23:46:56.063 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T23:46:56.108 INFO:teuthology.orchestra.run.smithi040.stdout:(37/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 6.0 MB/s | 762 kB 00:00 2024-04-26T23:46:56.133 INFO:teuthology.orchestra.run.smithi040.stdout:(38/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 157 kB/s | 18 kB 00:00 2024-04-26T23:46:56.161 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T23:46:56.164 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T23:46:56.212 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T23:46:56.300 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T23:46:56.340 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T23:46:56.368 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T23:46:56.409 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T23:46:56.434 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T23:46:56.477 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T23:46:56.519 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T23:46:56.554 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T23:46:56.649 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T23:46:56.759 INFO:teuthology.orchestra.run.smithi040.stdout:(39/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 13 MB/s | 49 MB 00:03 2024-04-26T23:46:56.792 INFO:teuthology.orchestra.run.smithi040.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 118 kB/s | 80 kB 00:00 2024-04-26T23:46:56.807 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T23:46:56.918 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:46:56.934 INFO:teuthology.orchestra.run.smithi040.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-04-26T23:46:56.976 INFO:teuthology.orchestra.run.smithi040.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 954 kB/s | 803 kB 00:00 2024-04-26T23:46:57.040 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:46:57.043 INFO:teuthology.orchestra.run.smithi040.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-26T23:46:57.076 INFO:teuthology.orchestra.run.smithi040.stdout:(44/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 895 kB/s | 253 kB 00:00 2024-04-26T23:46:57.077 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:46:57.127 INFO:teuthology.orchestra.run.smithi040.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-26T23:46:57.152 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:46:57.160 INFO:teuthology.orchestra.run.smithi040.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-26T23:46:57.181 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T23:46:57.211 INFO:teuthology.orchestra.run.smithi040.stdout:(47/152): python3-cryptography-36.0.1-4.el9.x86 4.5 MB/s | 1.2 MB 00:00 2024-04-26T23:46:57.244 INFO:teuthology.orchestra.run.smithi040.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-04-26T23:46:57.287 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T23:46:57.331 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T23:46:57.379 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T23:46:57.386 INFO:teuthology.orchestra.run.smithi040.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 757 kB/s | 107 kB 00:00 2024-04-26T23:46:57.394 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T23:46:57.407 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:46:57.445 INFO:teuthology.orchestra.run.smithi040.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 373 kB/s | 21 kB 00:00 2024-04-26T23:46:57.478 INFO:teuthology.orchestra.run.smithi040.stdout:(51/152): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-04-26T23:46:57.510 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:46:57.520 INFO:teuthology.orchestra.run.smithi040.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 3.0 MB/s | 943 kB 00:00 2024-04-26T23:46:57.533 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T23:46:57.546 INFO:teuthology.orchestra.run.smithi040.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 495 kB/s | 33 kB 00:00 2024-04-26T23:46:57.579 INFO:teuthology.orchestra.run.smithi040.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-04-26T23:46:57.595 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T23:46:57.596 INFO:teuthology.orchestra.run.smithi040.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 368 kB/s | 18 kB 00:00 2024-04-26T23:46:57.617 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T23:46:57.630 INFO:teuthology.orchestra.run.smithi040.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 225 kB/s | 11 kB 00:00 2024-04-26T23:46:57.636 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T23:46:57.664 INFO:teuthology.orchestra.run.smithi040.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 160 kB 00:00 2024-04-26T23:46:57.689 INFO:teuthology.orchestra.run.smithi040.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 780 kB/s | 45 kB 00:00 2024-04-26T23:46:57.806 INFO:teuthology.orchestra.run.smithi040.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-04-26T23:46:57.808 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T23:46:57.840 INFO:teuthology.orchestra.run.smithi040.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-04-26T23:46:57.841 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T23:46:57.865 INFO:teuthology.orchestra.run.smithi040.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 441 kB/s | 25 kB 00:00 2024-04-26T23:46:57.898 INFO:teuthology.orchestra.run.smithi040.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 822 kB/s | 47 kB 00:00 2024-04-26T23:46:57.949 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T23:46:57.957 INFO:teuthology.orchestra.run.smithi040.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-04-26T23:46:58.033 INFO:teuthology.orchestra.run.smithi040.stdout:(64/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.9 MB/s | 3.0 MB 00:00 2024-04-26T23:46:58.044 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T23:46:58.058 INFO:teuthology.orchestra.run.smithi040.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-04-26T23:46:58.083 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T23:46:58.083 INFO:teuthology.orchestra.run.smithi040.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 282 kB/s | 35 kB 00:00 2024-04-26T23:46:58.118 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T23:46:58.192 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T23:46:58.225 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T23:46:58.269 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T23:46:58.302 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T23:46:58.330 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T23:46:58.333 INFO:teuthology.orchestra.run.smithi040.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 6.4 MB/s | 1.6 MB 00:00 2024-04-26T23:46:58.400 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T23:46:58.453 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T23:46:58.491 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T23:46:58.521 INFO:teuthology.orchestra.run.smithi040.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 4.8 MB/s | 911 kB 00:00 2024-04-26T23:46:58.535 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T23:46:58.592 INFO:teuthology.orchestra.run.smithi040.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-04-26T23:46:58.611 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T23:46:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T23:46:58.646 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T23:46:58.681 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T23:46:58.685 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T23:46:58.700 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T23:46:58.712 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T23:46:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T23:46:58.770 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T23:46:58.816 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T23:46:58.890 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T23:46:59.001 INFO:teuthology.orchestra.run.smithi040.stdout:(70/152): openblas-openmp-0.3.21-2.el9.x86_64.r 4.9 MB/s | 4.7 MB 00:00 2024-04-26T23:46:59.025 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T23:46:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T23:46:59.076 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T23:46:59.093 INFO:teuthology.orchestra.run.smithi040.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 4.3 MB/s | 4.4 MB 00:01 2024-04-26T23:46:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T23:46:59.152 INFO:teuthology.orchestra.run.smithi040.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-26T23:46:59.190 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T23:46:59.222 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T23:46:59.252 INFO:teuthology.orchestra.run.smithi040.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-04-26T23:46:59.258 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T23:46:59.345 INFO:teuthology.orchestra.run.smithi040.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.6 MB/s | 244 kB 00:00 2024-04-26T23:46:59.444 INFO:teuthology.orchestra.run.smithi040.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-26T23:46:59.528 INFO:teuthology.orchestra.run.smithi040.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 311 kB/s | 26 kB 00:00 2024-04-26T23:46:59.604 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T23:46:59.636 INFO:teuthology.orchestra.run.smithi040.stdout:(77/152): postgresql-server-13.14-1.el9.x86_64. 5.5 MB/s | 5.8 MB 00:01 2024-04-26T23:46:59.686 INFO:teuthology.orchestra.run.smithi040.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 121 kB/s | 19 kB 00:00 2024-04-26T23:46:59.688 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T23:46:59.718 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T23:46:59.736 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T23:46:59.740 INFO:teuthology.orchestra.run.smithi040.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-04-26T23:46:59.761 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T23:46:59.770 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T23:46:59.795 INFO:teuthology.orchestra.run.smithi040.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 322 kB/s | 35 kB 00:00 2024-04-26T23:46:59.798 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T23:46:59.839 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T23:46:59.841 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T23:46:59.864 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T23:46:59.896 INFO:teuthology.orchestra.run.smithi040.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.5 MB/s | 250 kB 00:00 2024-04-26T23:46:59.979 INFO:teuthology.orchestra.run.smithi040.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-04-26T23:47:00.000 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T23:47:00.038 INFO:teuthology.orchestra.run.smithi040.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-26T23:47:00.060 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T23:47:00.114 INFO:teuthology.orchestra.run.smithi040.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 569 kB/s | 42 kB 00:00 2024-04-26T23:47:00.115 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T23:47:00.188 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T23:47:00.206 INFO:teuthology.orchestra.run.smithi040.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 2.9 MB/s | 269 kB 00:00 2024-04-26T23:47:00.219 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T23:47:00.255 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T23:47:00.302 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T23:47:00.323 INFO:teuthology.orchestra.run.smithi040.stdout:(86/152): python3-babel-2.9.1-2.el9.noarch.rpm 5.1 MB/s | 6.0 MB 00:01 2024-04-26T23:47:00.338 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T23:47:00.356 INFO:teuthology.orchestra.run.smithi040.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-04-26T23:47:00.381 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T23:47:00.415 INFO:teuthology.orchestra.run.smithi040.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.0 MB/s | 279 kB 00:00 2024-04-26T23:47:00.423 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T23:47:00.457 INFO:teuthology.orchestra.run.smithi040.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 513 kB/s | 51 kB 00:00 2024-04-26T23:47:00.499 INFO:teuthology.orchestra.run.smithi040.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 647 kB/s | 54 kB 00:00 2024-04-26T23:47:00.559 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T23:47:00.613 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T23:47:00.641 INFO:teuthology.orchestra.run.smithi040.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.7 MB/s | 5.1 MB 00:00 2024-04-26T23:47:00.642 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T23:47:00.683 INFO:teuthology.orchestra.run.smithi040.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 228 kB/s | 42 kB 00:00 2024-04-26T23:47:00.684 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T23:47:00.758 INFO:teuthology.orchestra.run.smithi040.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 598 kB/s | 44 kB 00:00 2024-04-26T23:47:00.778 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T23:47:00.792 INFO:teuthology.orchestra.run.smithi040.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-04-26T23:47:00.817 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T23:47:00.834 INFO:teuthology.orchestra.run.smithi040.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 482 kB/s | 36 kB 00:00 2024-04-26T23:47:00.858 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T23:47:00.876 INFO:teuthology.orchestra.run.smithi040.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 573 kB/s | 47 kB 00:00 2024-04-26T23:47:00.883 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T23:47:00.909 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T23:47:00.951 INFO:teuthology.orchestra.run.smithi040.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.6 MB/s | 305 kB 00:00 2024-04-26T23:47:00.984 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T23:47:01.001 INFO:teuthology.orchestra.run.smithi040.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 464 kB/s | 58 kB 00:00 2024-04-26T23:47:01.026 INFO:teuthology.orchestra.run.smithi040.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.2 MB/s | 94 kB 00:00 2024-04-26T23:47:01.077 INFO:teuthology.orchestra.run.smithi040.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 856 kB/s | 64 kB 00:00 2024-04-26T23:47:01.110 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T23:47:01.119 INFO:teuthology.orchestra.run.smithi040.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 491 kB/s | 45 kB 00:00 2024-04-26T23:47:01.149 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T23:47:01.161 INFO:teuthology.orchestra.run.smithi040.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 269 kB/s | 22 kB 00:00 2024-04-26T23:47:01.177 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T23:47:01.211 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T23:47:01.251 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T23:47:01.282 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T23:47:01.294 INFO:teuthology.orchestra.run.smithi040.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 4.8 MB/s | 863 kB 00:00 2024-04-26T23:47:01.312 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T23:47:01.347 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T23:47:01.388 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T23:47:01.430 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T23:47:01.464 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T23:47:01.550 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T23:47:01.603 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T23:47:01.634 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T23:47:01.682 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T23:47:01.753 INFO:teuthology.orchestra.run.smithi040.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 928 kB/s | 548 kB 00:00 2024-04-26T23:47:01.780 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T23:47:01.803 INFO:teuthology.orchestra.run.smithi040.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 608 kB/s | 309 kB 00:00 2024-04-26T23:47:01.827 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T23:47:02.037 INFO:teuthology.orchestra.run.smithi040.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 69 kB/s | 19 kB 00:00 2024-04-26T23:47:02.124 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:47:02.160 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:47:02.396 INFO:teuthology.orchestra.run.smithi040.stdout:(107/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.4 MB/s | 16 MB 00:01 2024-04-26T23:47:02.421 INFO:teuthology.orchestra.run.smithi040.stdout:(108/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 68 kB/s | 26 kB 00:00 2024-04-26T23:47:02.551 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T23:47:02.600 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T23:47:02.634 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T23:47:02.704 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:47:02.730 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T23:47:02.739 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T23:47:02.746 INFO:teuthology.orchestra.run.smithi040.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 139 kB/s | 49 kB 00:00 2024-04-26T23:47:02.771 INFO:teuthology.orchestra.run.smithi040.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 193 kB/s | 67 kB 00:00 2024-04-26T23:47:02.783 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T23:47:02.831 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T23:47:02.832 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T23:47:02.832 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T23:47:02.832 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:02.865 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T23:47:02.870 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T23:47:02.923 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T23:47:03.096 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T23:47:03.134 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T23:47:03.172 INFO:teuthology.orchestra.run.smithi040.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 356 kB/s | 151 kB 00:00 2024-04-26T23:47:03.221 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T23:47:03.280 INFO:teuthology.orchestra.run.smithi040.stdout:(112/152): libarrow-9.0.0-11.el9.x86_64.rpm 3.0 MB/s | 4.4 MB 00:01 2024-04-26T23:47:03.481 INFO:teuthology.orchestra.run.smithi040.stdout:(113/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.2 MB/s | 839 kB 00:00 2024-04-26T23:47:03.639 INFO:teuthology.orchestra.run.smithi040.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 121 kB/s | 43 kB 00:00 2024-04-26T23:47:03.748 INFO:teuthology.orchestra.run.smithi040.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 121 kB/s | 32 kB 00:00 2024-04-26T23:47:03.798 INFO:teuthology.orchestra.run.smithi040.stdout:(116/152): python3-asyncssh-2.13.2-5.el9.noarch 876 kB/s | 548 kB 00:00 2024-04-26T23:47:03.871 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T23:47:03.899 INFO:teuthology.orchestra.run.smithi040.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 58 kB/s | 15 kB 00:00 2024-04-26T23:47:03.990 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T23:47:04.190 INFO:teuthology.orchestra.run.smithi040.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 389 kB/s | 172 kB 00:00 2024-04-26T23:47:04.324 INFO:teuthology.orchestra.run.smithi040.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 683 kB/s | 358 kB 00:00 2024-04-26T23:47:04.416 INFO:teuthology.orchestra.run.smithi040.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 587 kB/s | 303 kB 00:00 2024-04-26T23:47:04.491 INFO:teuthology.orchestra.run.smithi040.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 118 kB/s | 35 kB 00:00 2024-04-26T23:47:04.842 INFO:teuthology.orchestra.run.smithi040.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 398 kB/s | 206 kB 00:00 2024-04-26T23:47:04.942 INFO:teuthology.orchestra.run.smithi040.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 321 kB/s | 144 kB 00:00 2024-04-26T23:47:05.050 INFO:teuthology.orchestra.run.smithi040.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 51 kB/s | 11 kB 00:00 2024-04-26T23:47:05.209 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T23:47:05.234 INFO:teuthology.orchestra.run.smithi040.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 61 kB/s | 18 kB 00:00 2024-04-26T23:47:05.297 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T23:47:05.317 INFO:teuthology.orchestra.run.smithi040.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 87 kB/s | 23 kB 00:00 2024-04-26T23:47:05.325 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T23:47:05.358 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T23:47:05.376 INFO:teuthology.orchestra.run.smithi040.stdout:(127/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.1 MB/s | 2.0 MB 00:00 2024-04-26T23:47:05.501 INFO:teuthology.orchestra.run.smithi040.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 73 kB/s | 19 kB 00:00 2024-04-26T23:47:05.585 INFO:teuthology.orchestra.run.smithi040.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 74 kB/s | 20 kB 00:00 2024-04-26T23:47:05.670 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T23:47:05.705 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T23:47:05.794 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T23:47:05.826 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T23:47:05.852 INFO:teuthology.orchestra.run.smithi040.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 132 kB/s | 46 kB 00:00 2024-04-26T23:47:05.862 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T23:47:06.010 INFO:teuthology.orchestra.run.smithi040.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 186 kB/s | 79 kB 00:00 2024-04-26T23:47:06.069 INFO:teuthology.orchestra.run.smithi040.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 1.5 MB/s | 1.0 MB 00:00 2024-04-26T23:47:06.220 INFO:teuthology.orchestra.run.smithi040.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 157 kB/s | 58 kB 00:00 2024-04-26T23:47:06.337 INFO:teuthology.orchestra.run.smithi040.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 61 kB/s | 16 kB 00:00 2024-04-26T23:47:06.345 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T23:47:06.374 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T23:47:06.437 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T23:47:06.459 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T23:47:06.662 INFO:teuthology.orchestra.run.smithi040.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 418 kB/s | 272 kB 00:00 2024-04-26T23:47:06.687 INFO:teuthology.orchestra.run.smithi040.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 81 kB/s | 28 kB 00:00 2024-04-26T23:47:06.713 INFO:teuthology.orchestra.run.smithi040.stdout:(137/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 183 kB/s | 90 kB 00:00 2024-04-26T23:47:06.930 INFO:teuthology.orchestra.run.smithi040.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 115 kB/s | 31 kB 00:00 2024-04-26T23:47:07.072 INFO:teuthology.orchestra.run.smithi040.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 165 kB/s | 59 kB 00:00 2024-04-26T23:47:07.122 INFO:teuthology.orchestra.run.smithi040.stdout:(140/152): python3-routes-2.5.1-5.el9.noarch.rp 435 kB/s | 188 kB 00:00 2024-04-26T23:47:07.197 INFO:teuthology.orchestra.run.smithi040.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 134 kB/s | 36 kB 00:00 2024-04-26T23:47:07.414 INFO:teuthology.orchestra.run.smithi040.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 149 kB/s | 51 kB 00:00 2024-04-26T23:47:07.548 INFO:teuthology.orchestra.run.smithi040.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 257 kB/s | 90 kB 00:00 2024-04-26T23:47:07.623 INFO:teuthology.orchestra.run.smithi040.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 459 kB/s | 230 kB 00:00 2024-04-26T23:47:07.823 INFO:teuthology.orchestra.run.smithi040.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 73 kB/s | 20 kB 00:00 2024-04-26T23:47:08.007 INFO:teuthology.orchestra.run.smithi040.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 721 kB/s | 427 kB 00:00 2024-04-26T23:47:08.049 INFO:teuthology.orchestra.run.smithi040.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 451 kB/s | 192 kB 00:00 2024-04-26T23:47:08.424 INFO:teuthology.orchestra.run.smithi040.stdout:(148/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 8.4 MB/s | 3.1 MB 00:00 2024-04-26T23:47:08.516 INFO:teuthology.orchestra.run.smithi040.stdout:(149/152): librados2-19.0.0-3306.gbc612ffb.el9. 6.7 MB/s | 3.4 MB 00:00 2024-04-26T23:47:08.650 INFO:teuthology.orchestra.run.smithi040.stdout:(150/152): thrift-0.15.0-2.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-04-26T23:47:08.712 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:47:08.749 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:47:08.875 INFO:teuthology.orchestra.run.smithi040.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 3.0 MB/s | 1.3 MB 00:00 2024-04-26T23:47:09.009 INFO:teuthology.orchestra.run.smithi040.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 4.9 MB/s | 2.4 MB 00:00 2024-04-26T23:47:09.017 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-26T23:47:09.018 INFO:teuthology.orchestra.run.smithi040.stdout:Total 11 MB/s | 222 MB 00:19 2024-04-26T23:47:09.294 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:47:09.336 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T23:47:09.430 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T23:47:09.430 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T23:47:09.431 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T23:47:09.431 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:09.462 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T23:47:11.965 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-04-26T23:47:12.229 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-04-26T23:47:12.229 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-04-26T23:47:14.746 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T23:47:14.746 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-04-26T23:47:14.746 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-04-26T23:47:14.747 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-04-26T23:47:14.747 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-04-26T23:47:14.747 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-04-26T23:47:14.747 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-04-26T23:47:14.747 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-04-26T23:47:14.747 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:14.930 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T23:47:14.946 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T23:47:14.946 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:14.946 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T23:47:14.946 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:15.178 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-04-26T23:47:15.178 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-04-26T23:47:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T23:47:16.425 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T23:47:16.426 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:16.426 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T23:47:16.426 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T23:47:16.426 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T23:47:16.426 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:16.642 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T23:47:16.665 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T23:47:16.665 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:16.665 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T23:47:16.665 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T23:47:16.665 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T23:47:16.665 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:17.102 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T23:47:17.124 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T23:47:17.124 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:17.124 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T23:47:17.124 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T23:47:17.124 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T23:47:17.124 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:17.232 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T23:47:17.281 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T23:47:17.336 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T23:47:17.521 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T23:47:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T23:47:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T23:47:17.735 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T23:47:17.883 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T23:47:17.919 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T23:47:17.960 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T23:47:17.988 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T23:47:18.348 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T23:47:18.378 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T23:47:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T23:47:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T23:47:18.510 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T23:47:19.554 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T23:47:19.586 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T23:47:20.552 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T23:47:20.578 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T23:47:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T23:47:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T23:47:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T23:47:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T23:47:20.912 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:20.912 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T23:47:20.912 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T23:47:20.913 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T23:47:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:20.957 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T23:47:20.957 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /sys 2024-04-26T23:47:20.957 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /proc 2024-04-26T23:47:20.957 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /mnt 2024-04-26T23:47:20.957 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /var/tmp 2024-04-26T23:47:20.957 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /home 2024-04-26T23:47:20.957 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /root 2024-04-26T23:47:20.957 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /tmp 2024-04-26T23:47:20.957 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:20.957 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T23:47:20.968 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T23:47:21.066 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T23:47:21.122 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T23:47:21.134 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T23:47:21.150 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T23:47:21.150 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:21.150 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T23:47:21.150 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:21.583 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-04-26T23:47:21.802 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T23:47:21.917 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T23:47:22.043 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T23:47:22.045 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T23:47:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T23:47:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T23:47:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T23:47:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T23:47:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T23:47:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:22.109 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T23:47:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T23:47:22.145 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T23:47:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T23:47:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T23:47:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:22.211 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T23:47:22.233 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T23:47:22.233 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:22.233 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T23:47:22.233 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T23:47:22.233 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T23:47:22.233 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:22.287 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T23:47:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T23:47:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T23:47:22.559 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:22.559 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T23:47:22.560 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T23:47:22.560 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T23:47:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:22.606 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T23:47:22.629 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T23:47:22.630 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:22.630 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T23:47:22.630 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T23:47:22.630 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T23:47:22.630 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:22.720 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T23:47:22.845 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:47:22.847 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T23:47:22.869 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T23:47:22.869 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:22.869 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T23:47:22.869 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T23:47:22.869 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T23:47:22.869 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:23.031 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:47:23.059 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:47:23.202 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:47:23.269 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T23:47:23.314 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T23:47:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T23:47:23.336 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:23.336 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T23:47:23.336 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T23:47:23.336 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T23:47:23.336 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:23.452 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T23:47:23.500 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T23:47:23.556 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T23:47:23.641 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T23:47:23.707 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T23:47:23.730 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:47:23.741 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T23:47:23.790 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T23:47:23.867 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T23:47:23.919 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:47:23.948 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T23:47:23.955 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T23:47:24.036 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T23:47:24.074 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T23:47:24.088 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T23:47:24.123 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T23:47:24.142 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T23:47:24.155 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T23:47:24.183 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T23:47:24.410 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T23:47:24.468 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T23:47:24.561 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T23:47:24.590 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T23:47:24.622 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T23:47:24.671 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T23:47:24.692 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T23:47:24.722 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T23:47:24.805 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T23:47:24.841 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T23:47:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T23:47:24.995 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T23:47:25.046 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T23:47:25.105 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T23:47:25.149 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T23:47:25.181 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T23:47:25.298 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T23:47:25.372 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T23:47:25.400 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T23:47:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T23:47:25.599 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T23:47:25.645 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T23:47:25.701 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T23:47:25.736 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T23:47:25.749 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T23:47:25.782 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T23:47:25.862 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T23:47:25.926 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T23:47:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T23:47:26.229 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T23:47:26.293 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T23:47:26.743 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T23:47:26.765 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T23:47:26.866 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T23:47:27.001 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T23:47:27.027 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T23:47:27.068 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T23:47:27.096 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T23:47:27.096 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:27.096 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T23:47:27.097 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T23:47:27.097 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T23:47:27.097 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T23:47:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T23:47:27.143 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T23:47:27.149 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T23:47:27.154 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T23:47:27.187 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T23:47:27.190 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T23:47:27.228 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T23:47:27.231 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T23:47:27.252 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T23:47:27.263 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T23:47:27.290 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T23:47:27.300 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T23:47:27.301 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T23:47:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T23:47:27.521 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T23:47:27.550 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T23:47:27.611 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T23:47:27.916 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T23:47:28.008 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T23:47:28.108 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T23:47:28.146 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T23:47:28.174 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:47:28.185 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T23:47:28.215 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:47:28.222 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T23:47:28.235 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:47:28.254 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T23:47:28.254 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T23:47:28.277 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T23:47:28.277 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:28.277 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T23:47:28.277 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T23:47:28.277 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T23:47:28.277 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:28.301 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T23:47:28.322 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T23:47:28.344 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T23:47:28.344 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:28.344 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T23:47:28.344 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:28.364 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T23:47:28.397 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T23:47:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T23:47:28.419 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:28.419 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T23:47:28.419 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T23:47:28.419 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T23:47:28.419 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:28.432 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T23:47:28.642 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T23:47:28.698 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T23:47:28.726 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T23:47:28.736 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T23:47:28.745 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T23:47:28.745 INFO:teuthology.orchestra.run.smithi080.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:28.745 INFO:teuthology.orchestra.run.smithi080.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T23:47:28.745 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T23:47:28.746 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T23:47:28.746 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:28.795 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T23:47:28.938 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T23:47:28.976 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T23:47:29.025 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T23:47:29.059 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T23:47:29.069 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T23:47:29.121 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T23:47:29.153 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T23:47:29.194 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T23:47:29.277 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T23:47:29.318 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T23:47:29.319 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T23:47:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T23:47:29.403 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T23:47:29.444 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T23:47:29.444 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T23:47:29.513 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T23:47:29.535 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T23:47:29.573 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T23:47:29.600 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:47:29.614 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T23:47:29.663 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T23:47:29.729 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T23:47:29.797 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T23:47:29.845 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T23:47:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T23:47:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T23:47:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T23:47:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T23:47:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T23:47:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T23:47:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T23:47:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T23:47:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T23:47:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T23:47:31.650 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T23:47:31.716 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T23:47:31.750 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T23:47:31.886 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T23:47:31.954 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T23:47:32.066 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T23:47:32.109 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T23:47:32.357 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T23:47:32.389 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T23:47:32.495 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T23:47:33.318 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T23:47:33.359 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T23:47:33.369 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T23:47:33.400 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T23:47:33.433 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T23:47:33.460 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T23:47:33.515 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T23:47:33.638 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T23:47:33.681 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T23:47:33.711 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T23:47:34.332 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:47:34.376 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:47:34.396 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:47:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T23:47:35.220 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T23:47:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T23:47:35.315 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T23:47:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T23:47:35.355 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T23:47:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T23:47:35.435 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:47:35.435 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T23:47:35.435 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T23:47:35.435 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T23:47:35.435 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T23:47:35.435 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T23:47:35.435 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T23:47:35.435 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T23:47:35.435 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:47:35.435 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:47:35.435 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T23:47:35.436 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T23:47:35.437 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T23:47:35.437 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T23:47:35.437 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T23:47:35.448 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T23:47:35.448 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T23:47:35.448 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T23:47:35.449 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T23:47:35.450 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T23:47:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T23:47:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T23:47:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T23:47:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T23:47:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T23:47:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T23:47:35.457 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T23:47:35.458 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T23:47:35.486 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T23:47:35.486 INFO:teuthology.orchestra.run.smithi080.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T23:47:35.611 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T23:47:35.644 INFO:teuthology.orchestra.run.smithi080.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T23:47:35.645 INFO:teuthology.orchestra.run.smithi080.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T23:47:35.736 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T23:47:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T23:47:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T23:47:35.801 INFO:teuthology.orchestra.run.smithi080.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:47:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T23:47:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T23:47:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T23:47:36.567 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T23:47:36.594 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T23:47:36.694 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T23:47:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T23:47:37.563 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:47:37.563 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:37.563 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-04-26T23:47:37.563 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.564 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.564 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T23:47:37.564 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T23:47:37.564 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-04-26T23:47:37.564 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T23:47:37.564 INFO:teuthology.orchestra.run.smithi103.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T23:47:37.564 INFO:teuthology.orchestra.run.smithi103.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T23:47:37.564 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T23:47:37.564 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.564 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.564 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.564 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:37.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:37.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:37.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:37.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:37.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:37.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.565 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T23:47:37.566 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T23:47:37.567 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T23:47:37.568 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T23:47:37.569 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T23:47:37.570 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T23:47:37.571 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T23:47:37.572 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T23:47:37.573 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:37.575 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T23:47:37.575 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T23:47:37.575 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T23:47:37.575 INFO:teuthology.orchestra.run.smithi103.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T23:47:37.575 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T23:47:37.575 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T23:47:37.575 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T23:47:37.575 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:47:37.575 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-04-26T23:47:37.969 DEBUG:teuthology.parallel:result is None 2024-04-26T23:47:39.517 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:47:39.568 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:47:40.259 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:47:40.314 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T23:47:40.478 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T23:47:40.478 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T23:47:40.479 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T23:47:40.479 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:47:40.523 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T23:47:41.910 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:47:41.910 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T23:47:41.910 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T23:47:41.910 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T23:47:41.911 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T23:47:41.912 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T23:47:41.912 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T23:47:41.912 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T23:47:41.912 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T23:47:41.912 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T23:47:41.912 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T23:47:41.912 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T23:47:41.912 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T23:47:41.912 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T23:47:41.912 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T23:47:41.912 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T23:47:41.912 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T23:47:41.924 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T23:47:41.924 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T23:47:41.924 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T23:47:41.924 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T23:47:41.924 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T23:47:41.925 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T23:47:41.926 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T23:47:41.927 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T23:47:41.928 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T23:47:41.929 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T23:47:41.930 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T23:47:41.931 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T23:47:41.932 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T23:47:41.933 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T23:47:41.933 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T23:47:41.933 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T23:47:41.933 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T23:47:41.933 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T23:47:41.933 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T23:47:41.933 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T23:47:41.933 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T23:47:41.933 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T23:47:41.933 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T23:47:44.013 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:47:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:44.014 INFO:teuthology.orchestra.run.smithi080.stdout:Upgraded: 2024-04-26T23:47:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T23:47:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T23:47:44.014 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-26T23:47:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T23:47:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T23:47:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T23:47:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T23:47:44.014 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.015 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T23:47:44.016 INFO:teuthology.orchestra.run.smithi080.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T23:47:44.017 INFO:teuthology.orchestra.run.smithi080.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T23:47:44.018 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T23:47:44.019 INFO:teuthology.orchestra.run.smithi080.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T23:47:44.020 INFO:teuthology.orchestra.run.smithi080.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T23:47:44.021 INFO:teuthology.orchestra.run.smithi080.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T23:47:44.022 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T23:47:44.023 INFO:teuthology.orchestra.run.smithi080.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T23:47:44.024 INFO:teuthology.orchestra.run.smithi080.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T23:47:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T23:47:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T23:47:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T23:47:44.025 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-26T23:47:44.025 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-26T23:47:44.432 DEBUG:teuthology.parallel:result is None 2024-04-26T23:47:53.779 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T23:47:53.780 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2024-04-26T23:47:53.780 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2024-04-26T23:47:53.780 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2024-04-26T23:47:53.780 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2024-04-26T23:47:53.780 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2024-04-26T23:47:53.780 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /root 2024-04-26T23:47:53.780 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2024-04-26T23:47:53.780 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:47:54.036 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T23:47:54.085 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T23:47:54.085 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:54.085 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T23:47:54.085 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:47:55.796 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T23:47:55.886 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T23:47:55.887 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:55.887 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T23:47:55.887 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T23:47:55.887 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T23:47:55.887 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:47:56.147 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T23:47:56.225 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T23:47:56.225 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:56.225 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T23:47:56.225 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T23:47:56.225 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T23:47:56.226 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:47:56.758 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T23:47:56.845 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T23:47:56.845 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:47:56.845 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T23:47:56.845 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T23:47:56.845 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T23:47:56.845 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:47:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T23:47:57.047 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T23:47:57.123 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T23:47:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T23:47:57.462 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T23:47:57.554 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T23:47:57.680 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T23:47:57.864 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T23:47:57.927 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T23:47:57.998 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T23:47:58.034 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T23:47:58.645 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T23:47:58.689 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T23:47:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T23:47:58.834 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T23:47:58.871 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T23:48:00.246 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T23:48:00.298 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T23:48:01.476 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T23:48:01.507 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T23:48:01.661 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T23:48:01.846 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T23:48:01.875 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T23:48:01.951 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T23:48:01.952 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:48:01.952 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T23:48:01.952 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T23:48:01.952 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T23:48:01.952 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:48:02.010 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T23:48:02.050 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T23:48:02.156 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T23:48:02.227 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T23:48:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T23:48:03.320 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T23:48:03.410 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T23:48:03.411 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:48:03.411 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T23:48:03.411 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T23:48:03.411 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T23:48:03.411 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:48:03.472 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T23:48:03.545 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T23:48:03.545 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:48:03.546 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T23:48:03.546 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:48:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T23:48:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T23:48:03.701 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:48:03.701 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T23:48:03.701 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T23:48:03.701 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T23:48:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:48:04.073 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T23:48:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T23:48:04.144 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:48:04.144 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T23:48:04.145 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T23:48:04.145 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T23:48:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:48:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T23:48:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T23:48:09.587 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T23:48:09.633 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T23:48:09.685 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T23:48:10.039 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T23:48:10.092 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T23:48:10.148 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T23:48:11.038 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:48:11.084 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:48:11.144 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:48:12.532 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T23:48:12.641 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T23:48:12.725 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T23:48:12.920 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T23:48:12.964 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T23:48:12.964 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T23:48:13.088 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T23:48:13.130 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T23:48:13.130 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T23:48:13.320 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T23:48:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:48:13.802 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:48:13.803 INFO:teuthology.orchestra.run.smithi040.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-26T23:48:13.803 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:48:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T23:48:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T23:48:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T23:48:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T23:48:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T23:48:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T23:48:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T23:48:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:48:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:48:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T23:48:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T23:48:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T23:48:19.695 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T23:48:19.695 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T23:48:19.695 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T23:48:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T23:48:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T23:48:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T23:48:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T23:48:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T23:48:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T23:48:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T23:48:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T23:48:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T23:48:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T23:48:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T23:48:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T23:48:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T23:48:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T23:48:19.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T23:48:19.711 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T23:48:19.712 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T23:48:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T23:48:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T23:48:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T23:48:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T23:48:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T23:48:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T23:48:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T23:48:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T23:48:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T23:48:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T23:48:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T23:48:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T23:48:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T23:48:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T23:48:19.716 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T23:48:19.716 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T23:48:19.716 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T23:48:19.716 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T23:48:19.716 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T23:48:19.716 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T23:48:19.716 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T23:48:19.716 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T23:48:22.198 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:48:22.198 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:48:22.198 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2024-04-26T23:48:22.198 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.198 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:48:22.199 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.200 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T23:48:22.201 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.202 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T23:48:22.203 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T23:48:22.204 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T23:48:22.205 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T23:48:22.205 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T23:48:22.205 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T23:48:22.205 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T23:48:22.205 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T23:48:22.205 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T23:48:22.205 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T23:48:22.205 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T23:48:22.205 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T23:48:22.205 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T23:48:22.205 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T23:48:22.205 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T23:48:22.206 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T23:48:22.207 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T23:48:22.208 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T23:48:22.209 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T23:48:22.210 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T23:48:22.210 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:48:22.210 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-04-26T23:48:22.903 DEBUG:teuthology.parallel:result is None 2024-04-26T23:48:22.904 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:48:23.120 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T23:48:23.177 INFO:teuthology.orchestra.run.smithi040.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T23:48:23.177 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T23:48:23.178 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T23:48:23.179 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:48:23.318 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T23:48:23.343 INFO:teuthology.orchestra.run.smithi080.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T23:48:23.343 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T23:48:23.343 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T23:48:23.345 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:48:23.480 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T23:48:23.506 INFO:teuthology.orchestra.run.smithi103.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T23:48:23.507 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T23:48:23.507 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T23:48:23.508 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T23:48:23.509 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:48:23.509 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T23:48:23.588 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-26T23:48:23.588 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T23:48:23.618 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:48:23.618 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T23:48:23.649 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T23:48:23.649 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:48:23.649 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T23:48:23.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T23:48:23.863 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-26T23:48:23.863 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T23:48:23.892 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T23:48:23.960 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:48:23.960 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T23:48:23.990 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T23:48:24.055 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T23:48:24.056 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:48:24.056 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T23:48:24.128 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T23:48:24.244 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-26T23:48:24.244 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T23:48:24.268 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T23:48:24.345 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:48:24.345 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T23:48:24.370 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T23:48:24.434 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T23:48:24.434 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:48:24.434 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T23:48:24.503 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T23:48:24.614 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-26T23:48:24.614 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T23:48:24.638 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T23:48:24.701 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:48:24.701 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T23:48:24.724 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T23:48:24.788 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T23:48:24.889 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-26T23:48:24.890 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:48:24.890 INFO:tasks.cephadm:Cluster fsid is 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:48:24.890 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T23:48:24.890 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.40', 'mon.b': '172.21.15.80', 'mon.c': '172.21.15.103'} 2024-04-26T23:48:24.891 INFO:tasks.cephadm:First mon is mon.a on smithi040 2024-04-26T23:48:24.891 INFO:tasks.cephadm:First mgr is x 2024-04-26T23:48:24.891 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T23:48:24.891 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname $(hostname -s) 2024-04-26T23:48:24.972 DEBUG:teuthology.orchestra.run.smithi080:> sudo hostname $(hostname -s) 2024-04-26T23:48:24.995 DEBUG:teuthology.orchestra.run.smithi103:> sudo hostname $(hostname -s) 2024-04-26T23:48:25.018 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T23:48:25.019 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:48:25.237 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78918/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3306-gbc612ffb', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3306.gbc612ffb'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 13:02:42.256491', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240426.111959', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-26T23:48:25.453 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-testing-20240426.111959, sha1 bc612ffb9c1d5bc27536bc917a705262a36d4387 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:48:25.454 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm 2024-04-26T23:48:25.454 DEBUG:teuthology.orchestra.run.smithi040:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T23:48:26.133 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 23:48 /home/ubuntu/cephtest/cephadm 2024-04-26T23:48:26.134 DEBUG:teuthology.orchestra.run.smithi080:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T23:48:26.958 INFO:teuthology.orchestra.run.smithi080.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 23:48 /home/ubuntu/cephtest/cephadm 2024-04-26T23:48:26.958 DEBUG:teuthology.orchestra.run.smithi103:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T23:48:27.781 INFO:teuthology.orchestra.run.smithi103.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 23:48 /home/ubuntu/cephtest/cephadm 2024-04-26T23:48:27.781 DEBUG:teuthology.orchestra.run.smithi040:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T23:48:27.828 DEBUG:teuthology.orchestra.run.smithi080:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T23:48:27.846 DEBUG:teuthology.orchestra.run.smithi103:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T23:48:27.881 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 on all hosts... 2024-04-26T23:48:27.881 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T23:48:27.885 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T23:48:27.889 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T23:48:29.407 INFO:teuthology.orchestra.run.smithi080.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T23:48:29.466 INFO:teuthology.orchestra.run.smithi103.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T23:48:29.995 INFO:teuthology.orchestra.run.smithi040.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T23:48:59.443 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-04-26T23:48:59.444 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T23:48:59.444 INFO:teuthology.orchestra.run.smithi080.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T23:48:59.444 INFO:teuthology.orchestra.run.smithi080.stdout: "repo_digests": [ 2024-04-26T23:48:59.444 INFO:teuthology.orchestra.run.smithi080.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T23:48:59.444 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-04-26T23:48:59.444 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-04-26T23:49:07.765 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-26T23:49:07.766 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T23:49:07.766 INFO:teuthology.orchestra.run.smithi103.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T23:49:07.766 INFO:teuthology.orchestra.run.smithi103.stdout: "repo_digests": [ 2024-04-26T23:49:07.766 INFO:teuthology.orchestra.run.smithi103.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T23:49:07.766 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-26T23:49:07.766 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-26T23:49:14.161 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:49:14.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T23:49:14.161 INFO:teuthology.orchestra.run.smithi040.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T23:49:14.161 INFO:teuthology.orchestra.run.smithi040.stdout: "repo_digests": [ 2024-04-26T23:49:14.161 INFO:teuthology.orchestra.run.smithi040.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T23:49:14.161 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-26T23:49:14.162 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:49:14.215 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph 2024-04-26T23:49:14.289 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /etc/ceph 2024-04-26T23:49:14.319 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph 2024-04-26T23:49:14.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /etc/ceph 2024-04-26T23:49:14.430 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 777 /etc/ceph 2024-04-26T23:49:14.458 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /etc/ceph 2024-04-26T23:49:14.486 INFO:tasks.cephadm:Writing seed config... 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-26T23:49:14.488 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T23:49:14.489 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T23:49:14.490 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T23:49:14.491 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-26T23:49:14.491 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T23:49:14.491 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-26T23:49:14.491 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:49:14.491 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T23:49:14.544 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 78cff6ea-0427-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-26T23:49:14.544 DEBUG:teuthology.orchestra.run.smithi040:mon.a> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T23:49:14.591 DEBUG:teuthology.orchestra.run.smithi040:mgr.x> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T23:49:14.636 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T23:49:14.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 -v bootstrap --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.40 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T23:49:14.898 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-04-26T23:49:14.899 INFO:teuthology.orchestra.run.smithi040.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387', '-v', 'bootstrap', '--fsid', '78cff6ea-0427-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.40', '--skip-admin-label'] 2024-04-26T23:49:14.899 INFO:teuthology.orchestra.run.smithi040.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T23:49:14.899 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying podman|docker is present... 2024-04-26T23:49:14.955 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T23:49:14.956 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying lvm2 is present... 2024-04-26T23:49:14.956 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying time synchronization is in place... 2024-04-26T23:49:14.971 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T23:49:14.971 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T23:49:14.984 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T23:49:14.984 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-26T23:49:15.002 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout enabled 2024-04-26T23:49:15.016 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout active 2024-04-26T23:49:15.017 INFO:teuthology.orchestra.run.smithi040.stdout:Unit chronyd.service is enabled and running 2024-04-26T23:49:15.017 INFO:teuthology.orchestra.run.smithi040.stdout:Repeating the final host check... 2024-04-26T23:49:15.071 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T23:49:15.072 INFO:teuthology.orchestra.run.smithi040.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T23:49:15.072 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl is present 2024-04-26T23:49:15.072 INFO:teuthology.orchestra.run.smithi040.stdout:lvcreate is present 2024-04-26T23:49:15.090 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T23:49:15.090 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T23:49:15.112 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T23:49:15.112 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-26T23:49:15.137 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout enabled 2024-04-26T23:49:15.158 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout active 2024-04-26T23:49:15.158 INFO:teuthology.orchestra.run.smithi040.stdout:Unit chronyd.service is enabled and running 2024-04-26T23:49:15.158 INFO:teuthology.orchestra.run.smithi040.stdout:Host looks OK 2024-04-26T23:49:15.158 INFO:teuthology.orchestra.run.smithi040.stdout:Cluster fsid: 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:49:15.159 INFO:teuthology.orchestra.run.smithi040.stdout:Acquiring lock 140473456372896 on /run/cephadm/78cff6ea-0427-11ef-bc93-c7b262605968.lock 2024-04-26T23:49:15.159 INFO:teuthology.orchestra.run.smithi040.stdout:Lock 140473456372896 acquired on /run/cephadm/78cff6ea-0427-11ef-bc93-c7b262605968.lock 2024-04-26T23:49:15.161 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying IP 172.21.15.40 port 3300 ... 2024-04-26T23:49:15.162 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying IP 172.21.15.40 port 6789 ... 2024-04-26T23:49:15.162 INFO:teuthology.orchestra.run.smithi040.stdout:Base mon IP(s) is [172.21.15.40:3300, 172.21.15.40:6789], mon addrv is [v2:172.21.15.40:3300,v1:172.21.15.40:6789] 2024-04-26T23:49:15.177 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.40 metric 100 2024-04-26T23:49:15.178 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.40 metric 100 2024-04-26T23:49:15.190 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T23:49:15.191 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-26T23:49:15.191 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-26T23:49:15.202 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T23:49:15.203 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T23:49:15.203 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T23:49:15.203 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-26T23:49:15.203 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-26T23:49:15.203 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T23:49:15.206 INFO:teuthology.orchestra.run.smithi040.stdout:Mon IP `172.21.15.40` is in CIDR network `172.21.0.0/20` 2024-04-26T23:49:15.206 INFO:teuthology.orchestra.run.smithi040.stdout:Mon IP `172.21.15.40` is in CIDR network `172.21.0.0/20` 2024-04-26T23:49:15.206 INFO:teuthology.orchestra.run.smithi040.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T23:49:15.209 INFO:teuthology.orchestra.run.smithi040.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T23:49:15.212 INFO:teuthology.orchestra.run.smithi040.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T23:49:16.161 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T23:49:16.162 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T23:49:16.162 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T23:49:16.162 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying blob sha256:f340d36a00bce3feed0ad509aeafc9092c1c625bf553b8a613c380a79149b87e 2024-04-26T23:49:16.162 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-26T23:49:16.162 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying config sha256:55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T23:49:16.162 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T23:49:17.365 INFO:teuthology.orchestra.run.smithi040.stdout:ceph: stdout ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T23:49:17.366 INFO:teuthology.orchestra.run.smithi040.stdout:Ceph version: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T23:49:17.366 INFO:teuthology.orchestra.run.smithi040.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T23:49:18.543 INFO:teuthology.orchestra.run.smithi040.stdout:stat: stdout 167 167 2024-04-26T23:49:18.543 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial keys... 2024-04-26T23:49:19.708 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQB/PSxmDKN0ARAAJau7d5G0JsGJfh+70Q2y4g== 2024-04-26T23:49:20.806 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQCAPSxmRDLLCBAAmOMNQKJIJdBuEwsEFdCI2g== 2024-04-26T23:49:21.900 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQCBPSxm0r1BEhAA2aDg4hXl8Q+40jFkMgeL8Q== 2024-04-26T23:49:21.901 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial monmap... 2024-04-26T23:49:22.982 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T23:49:22.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T23:49:22.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:49:22.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T23:49:22.983 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool for a [v2:172.21.15.40:3300,v1:172.21.15.40:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T23:49:22.983 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = quincy 2024-04-26T23:49:22.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: set fsid to 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:49:22.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T23:49:22.984 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:49:22.984 INFO:teuthology.orchestra.run.smithi040.stdout:Creating mon... 2024-04-26T23:49:24.119 INFO:teuthology.orchestra.run.smithi040.stdout:create mon.a on 2024-04-26T23:49:24.521 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T23:49:24.775 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T23:49:25.067 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-78cff6ea-0427-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-78cff6ea-0427-11ef-bc93-c7b262605968.target. 2024-04-26T23:49:25.067 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-78cff6ea-0427-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-78cff6ea-0427-11ef-bc93-c7b262605968.target. 2024-04-26T23:49:25.426 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.a 2024-04-26T23:49:25.426 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to reset failed state of unit ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T23:49:25.713 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-78cff6ea-0427-11ef-bc93-c7b262605968.target.wants/ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-78cff6ea-0427-11ef-bc93-c7b262605968@.service. 2024-04-26T23:49:26.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:26 smithi040 podman[27709]: 2024-04-26 23:49:26.010475843 +0000 UTC m=+0.032922911 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:49:26.363 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:26 smithi040 podman[27709]: 2024-04-26 23:49:26.357629964 +0000 UTC m=+0.380077100 container init 4dd87454e949bc8aedbde7462507bbad8cc573529f397171346993e2d3a798e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T23:49:26.534 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T23:49:26.534 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-04-26T23:49:26.555 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T23:49:26.555 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-26T23:49:26.555 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-04-26T23:49:26.556 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T23:49:26.556 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mon to start... 2024-04-26T23:49:26.556 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mon... 2024-04-26T23:49:26.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:26 smithi040 podman[27709]: 2024-04-26 23:49:26.368654768 +0000 UTC m=+0.391101865 container start 4dd87454e949bc8aedbde7462507bbad8cc573529f397171346993e2d3a798e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-26T23:49:26.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:26 smithi040 ceph-mon[27728]: mkfs 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:49:26.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:26 smithi040 ceph-mon[27728]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T23:49:26.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:26 smithi040 bash[27709]: 4dd87454e949bc8aedbde7462507bbad8cc573529f397171346993e2d3a798e0 2024-04-26T23:49:26.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:26 smithi040 systemd[1]: Started Ceph mon.a for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:49:27.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:27 smithi040 ceph-mon[27728]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T23:49:27.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:27 smithi040 ceph-mon[27728]: monmap epoch 1 2024-04-26T23:49:27.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:27 smithi040 ceph-mon[27728]: fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:49:27.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:27 smithi040 ceph-mon[27728]: last_changed 2024-04-26T23:49:22.376995+0000 2024-04-26T23:49:27.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:27 smithi040 ceph-mon[27728]: created 2024-04-26T23:49:22.376995+0000 2024-04-26T23:49:27.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:27 smithi040 ceph-mon[27728]: min_mon_release 19 (squid) 2024-04-26T23:49:27.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:27 smithi040 ceph-mon[27728]: election_strategy: 1 2024-04-26T23:49:27.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:27 smithi040 ceph-mon[27728]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-26T23:49:27.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:27 smithi040 ceph-mon[27728]: fsmap 2024-04-26T23:49:27.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:27 smithi040 ceph-mon[27728]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T23:49:27.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:27 smithi040 ceph-mon[27728]: mgrmap e1: no daemons active 2024-04-26T23:49:27.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:27 smithi040 ceph-mon[27728]: from='client.? 172.21.15.40:0/662532493' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T23:49:27.904 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T23:49:27.905 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout id: 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:49:27.905 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T23:49:27.905 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:27.905 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout services: 2024-04-26T23:49:27.905 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.81209s) 2024-04-26T23:49:27.905 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T23:49:27.905 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T23:49:27.905 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:27.905 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout data: 2024-04-26T23:49:27.905 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T23:49:27.905 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.165+0000 7f84c3400700 1 Processor -- start 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.165+0000 7f84c3400700 1 -- start start 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.165+0000 7f84c3400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 0x7f84bc10bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.165+0000 7f84c3400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc07ceb0 con 0x7f84bc109760 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.167+0000 7f84c0a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 0x7f84bc10bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.167+0000 7f84c0a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 0x7f84bc10bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47386/0 (socket says 172.21.15.40:47386) 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.167+0000 7f84c0a00700 1 -- 172.21.15.40:0/1985109772 learned_addr learned my addr 172.21.15.40:0/1985109772 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.167+0000 7f84c0a00700 1 -- 172.21.15.40:0/1985109772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84bc10c090 con 0x7f84bc109760 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.168+0000 7f84c0a00700 1 --2- 172.21.15.40:0/1985109772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 0x7f84bc10bb50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f84b00098a0 tx=0x7f84b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac12baa75303210 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:27.906 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.168+0000 7f84bb400700 1 -- 172.21.15.40:0/1985109772 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84b0004160 con 0x7f84bc109760 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.168+0000 7f84bb400700 1 -- 172.21.15.40:0/1985109772 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84b0004300 con 0x7f84bc109760 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.168+0000 7f84bb400700 1 -- 172.21.15.40:0/1985109772 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84b00036a0 con 0x7f84bc109760 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.169+0000 7f84c3400700 1 -- 172.21.15.40:0/1985109772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 msgr2=0x7f84bc10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.169+0000 7f84c3400700 1 --2- 172.21.15.40:0/1985109772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 0x7f84bc10bb50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f84b00098a0 tx=0x7f84b0009bb0 comp rx=0 tx=0).stop 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.169+0000 7f84c3400700 1 -- 172.21.15.40:0/1985109772 shutdown_connections 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.169+0000 7f84c3400700 1 --2- 172.21.15.40:0/1985109772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 0x7f84bc10bb50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.169+0000 7f84c3400700 1 -- 172.21.15.40:0/1985109772 >> 172.21.15.40:0/1985109772 conn(0x7f84bc1031d0 msgr2=0x7f84bc1055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.169+0000 7f84c3400700 1 -- 172.21.15.40:0/1985109772 shutdown_connections 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.169+0000 7f84c3400700 1 -- 172.21.15.40:0/1985109772 wait complete. 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.170+0000 7f84c3400700 1 Processor -- start 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.170+0000 7f84c3400700 1 -- start start 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.171+0000 7f84c3400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 0x7f84bc108f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.171+0000 7f84c3400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc10c500 con 0x7f84bc109760 2024-04-26T23:49:27.907 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.171+0000 7f84c0a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 0x7f84bc108f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.171+0000 7f84c0a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 0x7f84bc108f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47398/0 (socket says 172.21.15.40:47398) 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.171+0000 7f84c0a00700 1 -- 172.21.15.40:0/662532493 learned_addr learned my addr 172.21.15.40:0/662532493 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.172+0000 7f84c0a00700 1 -- 172.21.15.40:0/662532493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84bc1094d0 con 0x7f84bc109760 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.172+0000 7f84c0a00700 1 --2- 172.21.15.40:0/662532493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 0x7f84bc108f90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f84b0000c00 tx=0x7f84b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.172+0000 7f84b9600700 1 -- 172.21.15.40:0/662532493 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84b0003db0 con 0x7f84bc109760 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.172+0000 7f84b9600700 1 -- 172.21.15.40:0/662532493 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84b0011630 con 0x7f84bc109760 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.172+0000 7f84c3400700 1 -- 172.21.15.40:0/662532493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84bc107710 con 0x7f84bc109760 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.172+0000 7f84b9600700 1 -- 172.21.15.40:0/662532493 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84b0011950 con 0x7f84bc109760 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.172+0000 7f84c3400700 1 -- 172.21.15.40:0/662532493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84bc107c50 con 0x7f84bc109760 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.173+0000 7f84b9600700 1 -- 172.21.15.40:0/662532493 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f84b001a460 con 0x7f84bc109760 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.173+0000 7f84b9600700 1 -- 172.21.15.40:0/662532493 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f84b001b780 con 0x7f84bc109760 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.174+0000 7f84c3400700 1 -- 172.21.15.40:0/662532493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84bc064a80 con 0x7f84bc109760 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.177+0000 7f84b9600700 1 -- 172.21.15.40:0/662532493 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f84b001b980 con 0x7f84bc109760 2024-04-26T23:49:27.908 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.259+0000 7f84c3400700 1 -- 172.21.15.40:0/662532493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f84bc195df0 con 0x7f84bc109760 2024-04-26T23:49:27.909 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.259+0000 7f84b9600700 1 -- 172.21.15.40:0/662532493 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f84b001bb60 con 0x7f84bc109760 2024-04-26T23:49:27.909 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.260+0000 7f84c3400700 1 -- 172.21.15.40:0/662532493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 msgr2=0x7f84bc108f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:27.909 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.260+0000 7f84c3400700 1 --2- 172.21.15.40:0/662532493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 0x7f84bc108f90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f84b0000c00 tx=0x7f84b0004750 comp rx=0 tx=0).stop 2024-04-26T23:49:27.909 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.261+0000 7f84c3400700 1 -- 172.21.15.40:0/662532493 shutdown_connections 2024-04-26T23:49:27.909 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.261+0000 7f84c3400700 1 --2- 172.21.15.40:0/662532493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84bc109760 0x7f84bc108f90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:27.909 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.261+0000 7f84c3400700 1 -- 172.21.15.40:0/662532493 >> 172.21.15.40:0/662532493 conn(0x7f84bc1031d0 msgr2=0x7f84bc1040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:27.909 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.261+0000 7f84c3400700 1 -- 172.21.15.40:0/662532493 shutdown_connections 2024-04-26T23:49:27.909 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:27.261+0000 7f84c3400700 1 -- 172.21.15.40:0/662532493 wait complete. 2024-04-26T23:49:27.909 INFO:teuthology.orchestra.run.smithi040.stdout:mon is available 2024-04-26T23:49:27.909 INFO:teuthology.orchestra.run.smithi040.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T23:49:28.940 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:28 smithi040 ceph-mon[27728]: from='client.? 172.21.15.40:0/2922464449' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T23:49:28.940 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:28 smithi040 ceph-mon[27728]: from='client.? 172.21.15.40:0/2922464449' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T23:49:29.222 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:29.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T23:49:29.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout fsid = 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:49:29.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T23:49:29.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.40:3300,v1:172.21.15.40:6789] 2024-04-26T23:49:29.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T23:49:29.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T23:49:29.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.506+0000 7fd820800700 1 Processor -- start 2024-04-26T23:49:29.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.506+0000 7fd820800700 1 -- start start 2024-04-26T23:49:29.231 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.507+0000 7fd820800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 0x7fd81810bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:29.231 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.507+0000 7fd820800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd81807ce50 con 0x7fd8181097b0 2024-04-26T23:49:29.231 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.508+0000 7fd81de00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 0x7fd81810bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:29.231 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.508+0000 7fd81de00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 0x7fd81810bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47412/0 (socket says 172.21.15.40:47412) 2024-04-26T23:49:29.231 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.508+0000 7fd81de00700 1 -- 172.21.15.40:0/3525417088 learned_addr learned my addr 172.21.15.40:0/3525417088 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:29.231 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.509+0000 7fd81de00700 1 -- 172.21.15.40:0/3525417088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81810c0e0 con 0x7fd8181097b0 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.509+0000 7fd81de00700 1 --2- 172.21.15.40:0/3525417088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 0x7fd81810bba0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd80c0098a0 tx=0x7fd80c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889202b33198b85f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.509+0000 7fd81ca00700 1 -- 172.21.15.40:0/3525417088 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd80c004160 con 0x7fd8181097b0 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.510+0000 7fd81ca00700 1 -- 172.21.15.40:0/3525417088 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd80c004300 con 0x7fd8181097b0 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.510+0000 7fd81ca00700 1 -- 172.21.15.40:0/3525417088 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd80c0036a0 con 0x7fd8181097b0 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.510+0000 7fd820800700 1 -- 172.21.15.40:0/3525417088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 msgr2=0x7fd81810bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.510+0000 7fd820800700 1 --2- 172.21.15.40:0/3525417088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 0x7fd81810bba0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd80c0098a0 tx=0x7fd80c009bb0 comp rx=0 tx=0).stop 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.510+0000 7fd820800700 1 -- 172.21.15.40:0/3525417088 shutdown_connections 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.510+0000 7fd820800700 1 --2- 172.21.15.40:0/3525417088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 0x7fd81810bba0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.510+0000 7fd820800700 1 -- 172.21.15.40:0/3525417088 >> 172.21.15.40:0/3525417088 conn(0x7fd8181031a0 msgr2=0x7fd8181055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.511+0000 7fd820800700 1 -- 172.21.15.40:0/3525417088 shutdown_connections 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.511+0000 7fd820800700 1 -- 172.21.15.40:0/3525417088 wait complete. 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.512+0000 7fd820800700 1 Processor -- start 2024-04-26T23:49:29.232 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.512+0000 7fd820800700 1 -- start start 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.512+0000 7fd820800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 0x7fd81819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.512+0000 7fd820800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd81810c550 con 0x7fd8181097b0 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.512+0000 7fd81de00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 0x7fd81819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.512+0000 7fd81de00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 0x7fd81819e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47424/0 (socket says 172.21.15.40:47424) 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.512+0000 7fd81de00700 1 -- 172.21.15.40:0/2922464449 learned_addr learned my addr 172.21.15.40:0/2922464449 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.513+0000 7fd81de00700 1 -- 172.21.15.40:0/2922464449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81819eae0 con 0x7fd8181097b0 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.513+0000 7fd81de00700 1 --2- 172.21.15.40:0/2922464449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 0x7fd81819e5a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd80c000c00 tx=0x7fd80c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.513+0000 7fd812a00700 1 -- 172.21.15.40:0/2922464449 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd80c003db0 con 0x7fd8181097b0 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.513+0000 7fd812a00700 1 -- 172.21.15.40:0/2922464449 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd80c011630 con 0x7fd8181097b0 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.513+0000 7fd812a00700 1 -- 172.21.15.40:0/2922464449 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd80c011950 con 0x7fd8181097b0 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.513+0000 7fd820800700 1 -- 172.21.15.40:0/2922464449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd81819ed70 con 0x7fd8181097b0 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.514+0000 7fd820800700 1 -- 172.21.15.40:0/2922464449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8181a1a60 con 0x7fd8181097b0 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.514+0000 7fd812a00700 1 -- 172.21.15.40:0/2922464449 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd80c011af0 con 0x7fd8181097b0 2024-04-26T23:49:29.233 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.514+0000 7fd812a00700 1 -- 172.21.15.40:0/2922464449 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd80c01c450 con 0x7fd8181097b0 2024-04-26T23:49:29.234 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.515+0000 7fd820800700 1 -- 172.21.15.40:0/2922464449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd818064a80 con 0x7fd8181097b0 2024-04-26T23:49:29.234 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.518+0000 7fd812a00700 1 -- 172.21.15.40:0/2922464449 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fd80c032020 con 0x7fd8181097b0 2024-04-26T23:49:29.234 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.597+0000 7fd820800700 1 -- 172.21.15.40:0/2922464449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fd81802e6a0 con 0x7fd8181097b0 2024-04-26T23:49:29.234 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.606+0000 7fd812a00700 1 -- 172.21.15.40:0/2922464449 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7fd80c004260 con 0x7fd8181097b0 2024-04-26T23:49:29.234 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.607+0000 7fd812a00700 1 -- 172.21.15.40:0/2922464449 <== mon.0 v2:172.21.15.40:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd80c01b450 con 0x7fd8181097b0 2024-04-26T23:49:29.234 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.609+0000 7fd820800700 1 -- 172.21.15.40:0/2922464449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 msgr2=0x7fd81819e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:29.234 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.609+0000 7fd820800700 1 --2- 172.21.15.40:0/2922464449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 0x7fd81819e5a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd80c000c00 tx=0x7fd80c004750 comp rx=0 tx=0).stop 2024-04-26T23:49:29.234 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.609+0000 7fd820800700 1 -- 172.21.15.40:0/2922464449 shutdown_connections 2024-04-26T23:49:29.235 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.609+0000 7fd820800700 1 --2- 172.21.15.40:0/2922464449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8181097b0 0x7fd81819e5a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:29.235 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.609+0000 7fd820800700 1 -- 172.21.15.40:0/2922464449 >> 172.21.15.40:0/2922464449 conn(0x7fd8181031a0 msgr2=0x7fd818104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:29.235 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.610+0000 7fd820800700 1 -- 172.21.15.40:0/2922464449 shutdown_connections 2024-04-26T23:49:29.235 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:28.610+0000 7fd820800700 1 -- 172.21.15.40:0/2922464449 wait complete. 2024-04-26T23:49:29.235 INFO:teuthology.orchestra.run.smithi040.stdout:Generating new minimal ceph.conf... 2024-04-26T23:49:30.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:29 smithi040 ceph-mon[27728]: from='client.? 172.21.15.40:0/2254870976' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:49:30.630 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.838+0000 7f31f5c00700 1 Processor -- start 2024-04-26T23:49:30.630 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.838+0000 7f31f5c00700 1 -- start start 2024-04-26T23:49:30.630 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.839+0000 7f31f5c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 0x7f31f01079d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:30.630 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.839+0000 7f31f5c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f31f007ce50 con 0x7f31f01075b0 2024-04-26T23:49:30.630 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.839+0000 7f31eea00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 0x7f31f01079d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:30.630 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.840+0000 7f31eea00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 0x7f31f01079d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47430/0 (socket says 172.21.15.40:47430) 2024-04-26T23:49:30.630 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.840+0000 7f31eea00700 1 -- 172.21.15.40:0/94053305 learned_addr learned my addr 172.21.15.40:0/94053305 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.840+0000 7f31eea00700 1 -- 172.21.15.40:0/94053305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f0107f10 con 0x7f31f01075b0 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.841+0000 7f31eea00700 1 --2- 172.21.15.40:0/94053305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 0x7f31f01079d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f31e00098a0 tx=0x7f31e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d81af04640d4b50 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.841+0000 7f31ed600700 1 -- 172.21.15.40:0/94053305 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31e0004160 con 0x7f31f01075b0 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.841+0000 7f31ed600700 1 -- 172.21.15.40:0/94053305 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f31e0004300 con 0x7f31f01075b0 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.841+0000 7f31ed600700 1 -- 172.21.15.40:0/94053305 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31e0011af0 con 0x7f31f01075b0 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.842+0000 7f31f5c00700 1 -- 172.21.15.40:0/94053305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 msgr2=0x7f31f01079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.842+0000 7f31f5c00700 1 --2- 172.21.15.40:0/94053305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 0x7f31f01079d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f31e00098a0 tx=0x7f31e0009bb0 comp rx=0 tx=0).stop 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.842+0000 7f31f5c00700 1 -- 172.21.15.40:0/94053305 shutdown_connections 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.842+0000 7f31f5c00700 1 --2- 172.21.15.40:0/94053305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 0x7f31f01079d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.842+0000 7f31f5c00700 1 -- 172.21.15.40:0/94053305 >> 172.21.15.40:0/94053305 conn(0x7f31f01031d0 msgr2=0x7f31f01055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.842+0000 7f31f5c00700 1 -- 172.21.15.40:0/94053305 shutdown_connections 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.842+0000 7f31f5c00700 1 -- 172.21.15.40:0/94053305 wait complete. 2024-04-26T23:49:30.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.843+0000 7f31f5c00700 1 Processor -- start 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.843+0000 7f31f5c00700 1 -- start start 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.844+0000 7f31f5c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 0x7f31f019ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.844+0000 7f31f5c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f31f0108f40 con 0x7f31f01075b0 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.844+0000 7f31eea00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 0x7f31f019ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.844+0000 7f31eea00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 0x7f31f019ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47442/0 (socket says 172.21.15.40:47442) 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.844+0000 7f31eea00700 1 -- 172.21.15.40:0/2254870976 learned_addr learned my addr 172.21.15.40:0/2254870976 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.845+0000 7f31eea00700 1 -- 172.21.15.40:0/2254870976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f019f370 con 0x7f31f01075b0 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.845+0000 7f31eea00700 1 --2- 172.21.15.40:0/2254870976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 0x7f31f019ee30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f31e0011ac0 tx=0x7f31e0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.845+0000 7f31e7400700 1 -- 172.21.15.40:0/2254870976 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31e0003970 con 0x7f31f01075b0 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.845+0000 7f31e7400700 1 -- 172.21.15.40:0/2254870976 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f31e0003b10 con 0x7f31f01075b0 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.845+0000 7f31e7400700 1 -- 172.21.15.40:0/2254870976 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31e0024610 con 0x7f31f01075b0 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.845+0000 7f31f5c00700 1 -- 172.21.15.40:0/2254870976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f019f600 con 0x7f31f01075b0 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.845+0000 7f31f5c00700 1 -- 172.21.15.40:0/2254870976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f01a22f0 con 0x7f31f01075b0 2024-04-26T23:49:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.846+0000 7f31e7400700 1 -- 172.21.15.40:0/2254870976 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f31e0024bd0 con 0x7f31f01075b0 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.846+0000 7f31e7400700 1 -- 172.21.15.40:0/2254870976 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f31e0019700 con 0x7f31f01075b0 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.847+0000 7f31f5c00700 1 -- 172.21.15.40:0/2254870976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31f0064a80 con 0x7f31f01075b0 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.850+0000 7f31e7400700 1 -- 172.21.15.40:0/2254870976 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f31e001a9e0 con 0x7f31f01075b0 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.928+0000 7f31f5c00700 1 -- 172.21.15.40:0/2254870976 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f31f002e690 con 0x7f31f01075b0 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.929+0000 7f31e7400700 1 -- 172.21.15.40:0/2254870976 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f31e001a430 con 0x7f31f01075b0 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.930+0000 7f31f5c00700 1 -- 172.21.15.40:0/2254870976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 msgr2=0x7f31f019ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.930+0000 7f31f5c00700 1 --2- 172.21.15.40:0/2254870976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 0x7f31f019ee30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f31e0011ac0 tx=0x7f31e0011ba0 comp rx=0 tx=0).stop 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.930+0000 7f31f5c00700 1 -- 172.21.15.40:0/2254870976 shutdown_connections 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.930+0000 7f31f5c00700 1 --2- 172.21.15.40:0/2254870976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31f01075b0 0x7f31f019ee30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.930+0000 7f31f5c00700 1 -- 172.21.15.40:0/2254870976 >> 172.21.15.40:0/2254870976 conn(0x7f31f01031d0 msgr2=0x7f31f0195a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.931+0000 7f31f5c00700 1 -- 172.21.15.40:0/2254870976 shutdown_connections 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:29.931+0000 7f31f5c00700 1 -- 172.21.15.40:0/2254870976 wait complete. 2024-04-26T23:49:30.633 INFO:teuthology.orchestra.run.smithi040.stdout:Restarting the monitor... 2024-04-26T23:49:30.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:30 smithi040 systemd[1]: Stopping Ceph mon.a for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:49:31.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:31 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a[27724]: 2024-04-26T23:49:31.137+0000 7ff829400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:49:31.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:31 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a[27724]: 2024-04-26T23:49:31.137+0000 7ff829400700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T23:49:31.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:31 smithi040 podman[27972]: 2024-04-26 23:49:31.167748333 +0000 UTC m=+0.160204248 container died 4dd87454e949bc8aedbde7462507bbad8cc573529f397171346993e2d3a798e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:49:38.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:38 smithi040 podman[27972]: 2024-04-26 23:49:38.134610703 +0000 UTC m=+7.127066655 container cleanup 4dd87454e949bc8aedbde7462507bbad8cc573529f397171346993e2d3a798e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-26T23:49:38.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:38 smithi040 bash[27972]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a 2024-04-26T23:49:40.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:40 smithi040 podman[27986]: 2024-04-26 23:49:40.205548092 +0000 UTC m=+9.016058053 container remove 4dd87454e949bc8aedbde7462507bbad8cc573529f397171346993e2d3a798e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-26T23:49:41.142 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:40 smithi040 systemd[1]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T23:49:41.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:40 smithi040 systemd[1]: Stopped Ceph mon.a for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:49:41.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:40 smithi040 systemd[1]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.a.service: Consumed 1.991s CPU time. 2024-04-26T23:49:41.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:40 smithi040 systemd[1]: Starting Ceph mon.a for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:49:41.423 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 podman[28077]: 2024-04-26 23:49:41.04335866 +0000 UTC m=+0.036414699 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:49:41.423 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 podman[28077]: 2024-04-26 23:49:41.139500338 +0000 UTC m=+0.132556412 container create 9cf2eb2d66cdc8fe654d31f941f9251881ec8123740de6205165f0d72680cfd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:49:41.423 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 podman[28077]: 2024-04-26 23:49:41.418391124 +0000 UTC m=+0.411447207 container init 9cf2eb2d66cdc8fe654d31f941f9251881ec8123740de6205165f0d72680cfd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:49:41.576 INFO:teuthology.orchestra.run.smithi040.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T23:49:41.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 podman[28077]: 2024-04-26 23:49:41.430932296 +0000 UTC m=+0.423988379 container start 9cf2eb2d66cdc8fe654d31f941f9251881ec8123740de6205165f0d72680cfd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-26T23:49:41.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T23:49:41.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-26T23:49:41.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: pidfile_write: ignore empty --pid-file 2024-04-26T23:49:41.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: load: jerasure load: lrc 2024-04-26T23:49:41.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: RocksDB version: 7.9.2 2024-04-26T23:49:41.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Git sha 0 2024-04-26T23:49:41.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-26T23:49:41.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: DB SUMMARY 2024-04-26T23:49:41.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: DB Session ID: 6QK5NXYN70QEV1DRNSVG 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: CURRENT file: CURRENT 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104082 ; 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.error_if_exists: 0 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.create_if_missing: 0 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.paranoid_checks: 1 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.env: 0x5592b26a5cc0 2024-04-26T23:49:41.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T23:49:41.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.info_log: 0x5592d709bdc0 2024-04-26T23:49:41.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T23:49:41.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.statistics: (nil) 2024-04-26T23:49:41.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.use_fsync: 0 2024-04-26T23:49:41.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_log_file_size: 0 2024-04-26T23:49:41.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T23:49:41.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T23:49:41.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T23:49:41.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T23:49:41.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.allow_fallocate: 1 2024-04-26T23:49:41.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T23:49:41.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T23:49:41.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.use_direct_reads: 0 2024-04-26T23:49:41.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T23:49:41.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T23:49:41.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.db_log_dir: 2024-04-26T23:49:41.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.wal_dir: 2024-04-26T23:49:41.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T23:49:41.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.write_buffer_manager: 0x5592d706d0e0 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T23:49:41.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.unordered_write: 0 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.row_cache: None 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.wal_filter: None 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.two_write_queues: 0 2024-04-26T23:49:41.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.wal_compression: 0 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.atomic_flush: 0 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.log_readahead_size: 0 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_background_jobs: 2 2024-04-26T23:49:41.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_background_compactions: -1 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_subcompactions: 1 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_open_files: -1 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T23:49:41.728 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T23:49:41.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_background_flushes: -1 2024-04-26T23:49:41.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Compression algorithms supported: 2024-04-26T23:49:41.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T23:49:41.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: kZSTD supported: 0 2024-04-26T23:49:41.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: kXpressCompression supported: 0 2024-04-26T23:49:41.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T23:49:41.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: kZlibCompression supported: 1 2024-04-26T23:49:41.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: kSnappyCompression supported: 1 2024-04-26T23:49:41.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: kLZ4Compression supported: 1 2024-04-26T23:49:41.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: kBZip2Compression supported: 0 2024-04-26T23:49:41.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T23:49:41.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.merge_operator: 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_filter: None 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5592d709bf80) 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout: cache_index_and_filter_blocks: 1 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T23:49:41.730 INFO:journalctl@ceph.mon.a.smithi040.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: pin_top_level_index_and_filter: 1 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_type: 0 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: data_block_index_type: 0 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_shortening: 1 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: checksum: 4 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: no_block_cache: 0 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache: 0x5592d709f1f0 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_name: BinnedLRUCache 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_options: 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: capacity : 536870912 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: num_shard_bits : 4 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: strict_capacity_limit : 0 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: high_pri_pool_ratio: 0.000 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_compressed: (nil) 2024-04-26T23:49:41.731 INFO:journalctl@ceph.mon.a.smithi040.stdout: persistent_cache: (nil) 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_size: 4096 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_size_deviation: 10 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_restart_interval: 16 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_block_restart_interval: 1 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: metadata_block_size: 4096 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: partition_filters: 0 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: use_delta_encoding: 1 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: filter_policy: bloomfilter 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: whole_key_filtering: 1 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: verify_compression: 0 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: read_amp_bytes_per_bit: 0 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: format_version: 5 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: enable_index_compression: 1 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_align: 0 2024-04-26T23:49:41.732 INFO:journalctl@ceph.mon.a.smithi040.stdout: max_auto_readahead_size: 262144 2024-04-26T23:49:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout: prepopulate_block_cache: 0 2024-04-26T23:49:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout: initial_auto_readahead_size: 8192 2024-04-26T23:49:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T23:49:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T23:49:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T23:49:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compression: NoCompression 2024-04-26T23:49:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T23:49:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T23:49:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T23:49:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.num_levels: 7 2024-04-26T23:49:41.733 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T23:49:41.734 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T23:49:41.734 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T23:49:41.734 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T23:49:41.734 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T23:49:41.734 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T23:49:41.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T23:49:41.736 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T23:49:41.737 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.inplace_update_support: 0 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.bloom_locality: 0 2024-04-26T23:49:41.738 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.max_successive_merges: 0 2024-04-26T23:49:41.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T23:49:41.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T23:49:41.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T23:49:41.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T23:49:41.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.ttl: 2592000 2024-04-26T23:49:41.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T23:49:41.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T23:49:41.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T23:49:41.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.enable_blob_files: false 2024-04-26T23:49:41.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.min_blob_size: 0 2024-04-26T23:49:41.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b9d4c4bd-6335-4e91-b798-ee8ee8ebb403 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714175381491271, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T23:49:41.740 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714175381493209, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99309, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 322, "table_properties": {"data_size": 97255, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13919, "raw_average_key_size": 49, "raw_value_size": 89648, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714175381, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b9d4c4bd-6335-4e91-b798-ee8ee8ebb403", "db_session_id": "6QK5NXYN70QEV1DRNSVG", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714175381493414, "job": 1, "event": "recovery_finished"} 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5592d71bc000 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: DB pointer 0x5592d71a8000 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** DB Stats ** 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T23:49:41.741 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** Compaction Stats [default] ** 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: L0 2/0 98.84 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: Sum 2/0 98.84 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** Compaction Stats [default] ** 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T23:49:41.742 INFO:journalctl@ceph.mon.a.smithi040.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 76.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative compaction: 0.00 GB write, 13.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval compaction: 0.00 GB write, 13.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: Block cache BinnedLRUCache@0x5592d709f1f0#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-26T23:49:41.743 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: starting mon.a rank 0 at public addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] at bind addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: mon.a@-1(???) e1 preinit fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: mon.a@-1(???).mds e1 new map 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: mon.a@-1(???).mds e1 print_map 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout: e1 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout: legacy client fscid: -1 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout: No filesystems configured 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T23:49:41.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 bash[28077]: 9cf2eb2d66cdc8fe654d31f941f9251881ec8123740de6205165f0d72680cfd6 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: monmap epoch 1 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: last_changed 2024-04-26T23:49:22.376995+0000 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: created 2024-04-26T23:49:22.376995+0000 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: min_mon_release 19 (squid) 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: election_strategy: 1 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: fsmap 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T23:49:41.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 ceph-mon[28095]: mgrmap e1: no daemons active 2024-04-26T23:49:41.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:41 smithi040 systemd[1]: Started Ceph mon.a for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:49:43.166 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.209+0000 7f8c11400700 1 Processor -- start 2024-04-26T23:49:43.166 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.210+0000 7f8c11400700 1 -- start start 2024-04-26T23:49:43.167 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.210+0000 7f8c11400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 0x7f8c0c109100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:43.167 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.210+0000 7f8c11400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0c1096d0 con 0x7f8c0c108ce0 2024-04-26T23:49:43.167 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.211+0000 7f8c0a800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 0x7f8c0c109100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:43.168 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.211+0000 7f8c0a800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 0x7f8c0c109100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56690/0 (socket says 172.21.15.40:56690) 2024-04-26T23:49:43.168 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.211+0000 7f8c0a800700 1 -- 172.21.15.40:0/3506492327 learned_addr learned my addr 172.21.15.40:0/3506492327 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:43.168 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.212+0000 7f8c0a800700 1 -- 172.21.15.40:0/3506492327 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0c109ee0 con 0x7f8c0c108ce0 2024-04-26T23:49:43.168 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.212+0000 7f8c0a800700 1 --2- 172.21.15.40:0/3506492327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 0x7f8c0c109100 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0098a0 tx=0x7f8bfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26666c8183251ed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:43.168 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.212+0000 7f8c09e00700 1 -- 172.21.15.40:0/3506492327 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bfc004160 con 0x7f8c0c108ce0 2024-04-26T23:49:43.168 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.213+0000 7f8c09e00700 1 -- 172.21.15.40:0/3506492327 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8bfc004300 con 0x7f8c0c108ce0 2024-04-26T23:49:43.168 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.213+0000 7f8c09e00700 1 -- 172.21.15.40:0/3506492327 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bfc011af0 con 0x7f8c0c108ce0 2024-04-26T23:49:43.169 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.213+0000 7f8c11400700 1 -- 172.21.15.40:0/3506492327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 msgr2=0x7f8c0c109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:43.169 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.213+0000 7f8c11400700 1 --2- 172.21.15.40:0/3506492327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 0x7f8c0c109100 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0098a0 tx=0x7f8bfc009bb0 comp rx=0 tx=0).stop 2024-04-26T23:49:43.169 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.214+0000 7f8c11400700 1 -- 172.21.15.40:0/3506492327 shutdown_connections 2024-04-26T23:49:43.169 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.214+0000 7f8c11400700 1 --2- 172.21.15.40:0/3506492327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 0x7f8c0c109100 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:43.172 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.214+0000 7f8c11400700 1 -- 172.21.15.40:0/3506492327 >> 172.21.15.40:0/3506492327 conn(0x7f8c0c1042a0 msgr2=0x7f8c0c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:43.172 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.214+0000 7f8c11400700 1 -- 172.21.15.40:0/3506492327 shutdown_connections 2024-04-26T23:49:43.172 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.214+0000 7f8c11400700 1 -- 172.21.15.40:0/3506492327 wait complete. 2024-04-26T23:49:43.172 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.215+0000 7f8c11400700 1 Processor -- start 2024-04-26T23:49:43.172 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.215+0000 7f8c11400700 1 -- start start 2024-04-26T23:49:43.172 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.215+0000 7f8c11400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 0x7f8c0c07eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.215+0000 7f8c11400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0c10d290 con 0x7f8c0c108ce0 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.216+0000 7f8c0a800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 0x7f8c0c07eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.216+0000 7f8c0a800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 0x7f8c0c07eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56704/0 (socket says 172.21.15.40:56704) 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.216+0000 7f8c0a800700 1 -- 172.21.15.40:0/2863429035 learned_addr learned my addr 172.21.15.40:0/2863429035 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.216+0000 7f8c0a800700 1 -- 172.21.15.40:0/2863429035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0c07f0d0 con 0x7f8c0c108ce0 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.216+0000 7f8c0a800700 1 --2- 172.21.15.40:0/2863429035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 0x7f8c0c07eb90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc000c00 tx=0x7f8bfc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.217+0000 7f8c03400700 1 -- 172.21.15.40:0/2863429035 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bfc003b20 con 0x7f8c0c108ce0 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.217+0000 7f8c03400700 1 -- 172.21.15.40:0/2863429035 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8bfc01a8c0 con 0x7f8c0c108ce0 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.217+0000 7f8c03400700 1 -- 172.21.15.40:0/2863429035 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bfc024940 con 0x7f8c0c108ce0 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.217+0000 7f8c11400700 1 -- 172.21.15.40:0/2863429035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c0c07d2c0 con 0x7f8c0c108ce0 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.217+0000 7f8c11400700 1 -- 172.21.15.40:0/2863429035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c0c07d800 con 0x7f8c0c108ce0 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.217+0000 7f8c03400700 1 -- 172.21.15.40:0/2863429035 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8bfc022020 con 0x7f8c0c108ce0 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.218+0000 7f8c03400700 1 -- 172.21.15.40:0/2863429035 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8bfc02dde0 con 0x7f8c0c108ce0 2024-04-26T23:49:43.173 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.218+0000 7f8c11400700 1 -- 172.21.15.40:0/2863429035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c0c064a80 con 0x7f8c0c108ce0 2024-04-26T23:49:43.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.222+0000 7f8c03400700 1 -- 172.21.15.40:0/2863429035 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f8bfc0042c0 con 0x7f8c0c108ce0 2024-04-26T23:49:43.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.300+0000 7f8c11400700 1 -- 172.21.15.40:0/2863429035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f8c0c105890 con 0x7f8c0c108ce0 2024-04-26T23:49:43.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.303+0000 7f8c03400700 1 -- 172.21.15.40:0/2863429035 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f8bfc01aa70 con 0x7f8c0c108ce0 2024-04-26T23:49:43.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.305+0000 7f8c11400700 1 -- 172.21.15.40:0/2863429035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 msgr2=0x7f8c0c07eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:43.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.305+0000 7f8c11400700 1 --2- 172.21.15.40:0/2863429035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 0x7f8c0c07eb90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc000c00 tx=0x7f8bfc011b70 comp rx=0 tx=0).stop 2024-04-26T23:49:43.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.305+0000 7f8c11400700 1 -- 172.21.15.40:0/2863429035 shutdown_connections 2024-04-26T23:49:43.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.305+0000 7f8c11400700 1 --2- 172.21.15.40:0/2863429035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c0c108ce0 0x7f8c0c07eb90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:43.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.305+0000 7f8c11400700 1 -- 172.21.15.40:0/2863429035 >> 172.21.15.40:0/2863429035 conn(0x7f8c0c1042a0 msgr2=0x7f8c0c104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:43.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.305+0000 7f8c11400700 1 -- 172.21.15.40:0/2863429035 shutdown_connections 2024-04-26T23:49:43.174 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:42.306+0000 7f8c11400700 1 -- 172.21.15.40:0/2863429035 wait complete. 2024-04-26T23:49:43.175 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T23:49:43.202 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T23:49:43.202 INFO:teuthology.orchestra.run.smithi040.stdout:Creating mgr... 2024-04-26T23:49:43.204 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T23:49:43.205 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T23:49:43.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:43 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2863429035' entity='client.admin' 2024-04-26T23:49:43.621 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.x 2024-04-26T23:49:43.621 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to reset failed state of unit ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-26T23:49:43.881 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-78cff6ea-0427-11ef-bc93-c7b262605968.target.wants/ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-78cff6ea-0427-11ef-bc93-c7b262605968@.service. 2024-04-26T23:49:44.220 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:43 smithi040 systemd[1]: Starting Ceph mgr.x for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:49:44.560 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:44 smithi040 podman[28324]: 2024-04-26 23:49:44.179760458 +0000 UTC m=+0.033532464 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:49:44.560 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:44 smithi040 podman[28324]: 2024-04-26 23:49:44.292694856 +0000 UTC m=+0.146466869 container create 24a7832aa9b89649af7e873f241423459df6a1260301487a9d59ed83d836998a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True) 2024-04-26T23:49:44.560 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:44 smithi040 podman[28324]: 2024-04-26 23:49:44.554699125 +0000 UTC m=+0.408471190 container init 24a7832aa9b89649af7e873f241423459df6a1260301487a9d59ed83d836998a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-26T23:49:44.727 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T23:49:44.727 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-04-26T23:49:44.745 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T23:49:44.746 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-26T23:49:44.746 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-04-26T23:49:44.746 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T23:49:44.767 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T23:49:44.767 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-04-26T23:49:44.785 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T23:49:44.785 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-26T23:49:44.785 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-04-26T23:49:44.785 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T23:49:44.785 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr to start... 2024-04-26T23:49:44.785 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr... 2024-04-26T23:49:44.829 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:44 smithi040 podman[28324]: 2024-04-26 23:49:44.565348749 +0000 UTC m=+0.419120797 container start 24a7832aa9b89649af7e873f241423459df6a1260301487a9d59ed83d836998a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-26T23:49:44.829 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:44 smithi040 bash[28324]: 24a7832aa9b89649af7e873f241423459df6a1260301487a9d59ed83d836998a 2024-04-26T23:49:44.829 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:44 smithi040 systemd[1]: Started Ceph mgr.x for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:49:45.134 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:44 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:44.824+0000 7f45e0af8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T23:49:45.470 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:45 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:45.127+0000 7f45e0af8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:49:45.470 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:45 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:45.265+0000 7f45e0af8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T23:49:45.948 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:45 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:45.682+0000 7f45e0af8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T23:49:45.948 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:45 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:45.803+0000 7f45e0af8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T23:49:46.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:46 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1806952924' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T23:49:47.087 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:47.087 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:49:47.087 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "78cff6ea-0427-11ef-bc93-c7b262605968", 2024-04-26T23:49:47.087 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T23:49:47.088 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T23:49:47.088 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T23:49:47.088 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T23:49:47.088 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:47.088 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T23:49:47.088 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T23:49:47.088 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2024-04-26T23:49:47.088 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:49:47.088 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T23:49:47.089 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T23:49:47.089 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:49:47.089 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-04-26T23:49:47.089 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T23:49:47.089 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:47.089 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T23:49:47.089 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T23:49:47.089 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:47.089 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T23:49:47.089 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:47.089 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T23:49:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T23:49:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T23:49:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T23:49:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T23:49:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T23:49:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T23:49:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T23:49:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T23:49:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T23:49:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T23:49:47.090 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T23:49:47.091 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T23:49:47.091 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T23:49:47.091 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T23:49:47.091 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:47.091 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T23:49:47.091 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:47.091 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T23:49:47.091 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T23:49:47.091 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:47.091 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T23:49:47.091 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T23:49:47.091 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T23:49:26.449500+0000", 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:49:47.092 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.105+0000 7fc69b200700 1 Processor -- start 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.105+0000 7fc69b200700 1 -- start start 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.105+0000 7fc69b200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 0x7fc69c107e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.106+0000 7fc69b200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1083f0 con 0x7fc69c107a00 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.106+0000 7fc699e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 0x7fc69c107e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.106+0000 7fc699e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 0x7fc69c107e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56724/0 (socket says 172.21.15.40:56724) 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.106+0000 7fc699e00700 1 -- 172.21.15.40:0/341573258 learned_addr learned my addr 172.21.15.40:0/341573258 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.107+0000 7fc699e00700 1 -- 172.21.15.40:0/341573258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c108c00 con 0x7fc69c107a00 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.107+0000 7fc699e00700 1 --2- 172.21.15.40:0/341573258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 0x7fc69c107e20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0098a0 tx=0x7fc68c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10c4c06b878b44c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.107+0000 7fc699400700 1 -- 172.21.15.40:0/341573258 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc68c004160 con 0x7fc69c107a00 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.107+0000 7fc699400700 1 -- 172.21.15.40:0/341573258 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc68c004300 con 0x7fc69c107a00 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.107+0000 7fc699400700 1 -- 172.21.15.40:0/341573258 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc68c011af0 con 0x7fc69c107a00 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.108+0000 7fc69b200700 1 -- 172.21.15.40:0/341573258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 msgr2=0x7fc69c107e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:47.093 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.108+0000 7fc69b200700 1 --2- 172.21.15.40:0/341573258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 0x7fc69c107e20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0098a0 tx=0x7fc68c009bb0 comp rx=0 tx=0).stop 2024-04-26T23:49:47.094 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.108+0000 7fc69b200700 1 -- 172.21.15.40:0/341573258 shutdown_connections 2024-04-26T23:49:47.094 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.108+0000 7fc69b200700 1 --2- 172.21.15.40:0/341573258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 0x7fc69c107e20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:47.094 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.108+0000 7fc69b200700 1 -- 172.21.15.40:0/341573258 >> 172.21.15.40:0/341573258 conn(0x7fc69c103040 msgr2=0x7fc69c105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:47.094 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.109+0000 7fc69b200700 1 -- 172.21.15.40:0/341573258 shutdown_connections 2024-04-26T23:49:47.094 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.109+0000 7fc69b200700 1 -- 172.21.15.40:0/341573258 wait complete. 2024-04-26T23:49:47.094 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.110+0000 7fc69b200700 1 Processor -- start 2024-04-26T23:49:47.094 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.110+0000 7fc69b200700 1 -- start start 2024-04-26T23:49:47.094 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.111+0000 7fc69b200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 0x7fc69c192b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:47.094 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.111+0000 7fc69b200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c10bfb0 con 0x7fc69c107a00 2024-04-26T23:49:47.094 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.111+0000 7fc699e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 0x7fc69c192b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:47.094 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.111+0000 7fc699e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 0x7fc69c192b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56728/0 (socket says 172.21.15.40:56728) 2024-04-26T23:49:47.095 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.111+0000 7fc699e00700 1 -- 172.21.15.40:0/1806952924 learned_addr learned my addr 172.21.15.40:0/1806952924 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:47.095 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.111+0000 7fc699e00700 1 -- 172.21.15.40:0/1806952924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c1930b0 con 0x7fc69c107a00 2024-04-26T23:49:47.095 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.112+0000 7fc699e00700 1 --2- 172.21.15.40:0/1806952924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 0x7fc69c192b70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0117b0 tx=0x7fc68c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:47.095 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.112+0000 7fc692a00700 1 -- 172.21.15.40:0/1806952924 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc68c003af0 con 0x7fc69c107a00 2024-04-26T23:49:47.095 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.112+0000 7fc692a00700 1 -- 172.21.15.40:0/1806952924 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc68c020070 con 0x7fc69c107a00 2024-04-26T23:49:47.095 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.112+0000 7fc692a00700 1 -- 172.21.15.40:0/1806952924 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc68c019630 con 0x7fc69c107a00 2024-04-26T23:49:47.095 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.112+0000 7fc69b200700 1 -- 172.21.15.40:0/1806952924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc69c1912a0 con 0x7fc69c107a00 2024-04-26T23:49:47.095 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.112+0000 7fc69b200700 1 -- 172.21.15.40:0/1806952924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc69c1917e0 con 0x7fc69c107a00 2024-04-26T23:49:47.095 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.113+0000 7fc692a00700 1 -- 172.21.15.40:0/1806952924 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc68c019c30 con 0x7fc69c107a00 2024-04-26T23:49:47.095 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.113+0000 7fc692a00700 1 -- 172.21.15.40:0/1806952924 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc68c00b720 con 0x7fc69c107a00 2024-04-26T23:49:47.095 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.113+0000 7fc69b200700 1 -- 172.21.15.40:0/1806952924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc69c064a80 con 0x7fc69c107a00 2024-04-26T23:49:47.096 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.116+0000 7fc692a00700 1 -- 172.21.15.40:0/1806952924 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fc68c0042a0 con 0x7fc69c107a00 2024-04-26T23:49:47.096 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.196+0000 7fc69b200700 1 -- 172.21.15.40:0/1806952924 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc69c196800 con 0x7fc69c107a00 2024-04-26T23:49:47.096 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.197+0000 7fc692a00700 1 -- 172.21.15.40:0/1806952924 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc68c01a9c0 con 0x7fc69c107a00 2024-04-26T23:49:47.096 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.199+0000 7fc69b200700 1 -- 172.21.15.40:0/1806952924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 msgr2=0x7fc69c192b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:47.096 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.199+0000 7fc69b200700 1 --2- 172.21.15.40:0/1806952924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 0x7fc69c192b70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0117b0 tx=0x7fc68c011b70 comp rx=0 tx=0).stop 2024-04-26T23:49:47.096 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.199+0000 7fc69b200700 1 -- 172.21.15.40:0/1806952924 shutdown_connections 2024-04-26T23:49:47.096 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.199+0000 7fc69b200700 1 --2- 172.21.15.40:0/1806952924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc69c107a00 0x7fc69c192b70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:47.096 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.199+0000 7fc69b200700 1 -- 172.21.15.40:0/1806952924 >> 172.21.15.40:0/1806952924 conn(0x7fc69c103040 msgr2=0x7fc69c103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:47.096 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.199+0000 7fc69b200700 1 -- 172.21.15.40:0/1806952924 shutdown_connections 2024-04-26T23:49:47.096 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:46.199+0000 7fc69b200700 1 -- 172.21.15.40:0/1806952924 wait complete. 2024-04-26T23:49:47.096 INFO:teuthology.orchestra.run.smithi040.stdout:mgr not available, waiting (1/15)... 2024-04-26T23:49:47.720 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:47 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:47.456+0000 7f45e0af8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T23:49:47.720 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:47 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:47.689+0000 7f45e0af8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T23:49:48.219 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:47 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:47.908+0000 7f45e0af8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T23:49:48.689 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:48 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:48.299+0000 7f45e0af8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T23:49:48.690 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:48 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:48.425+0000 7f45e0af8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T23:49:48.951 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:48 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:48.684+0000 7f45e0af8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T23:49:49.219 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:48 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:48.946+0000 7f45e0af8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:49:49.807 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:49 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:49.774+0000 7f45e0af8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T23:49:50.158 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:49 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1405342118' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T23:49:50.469 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:50 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:50.352+0000 7f45e0af8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T23:49:50.544 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:50.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:49:50.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "78cff6ea-0427-11ef-bc93-c7b262605968", 2024-04-26T23:49:50.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T23:49:50.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T23:49:50.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T23:49:50.545 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T23:49:50.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:50.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T23:49:50.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T23:49:50.546 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2024-04-26T23:49:50.547 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:49:50.547 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T23:49:50.547 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T23:49:50.547 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:49:50.547 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-26T23:49:50.548 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T23:49:50.548 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:50.548 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T23:49:50.548 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T23:49:50.548 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:50.548 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T23:49:50.548 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:50.548 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T23:49:50.549 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T23:49:50.549 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T23:49:50.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T23:49:50.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T23:49:50.551 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:50.552 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T23:49:50.553 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T23:49:50.553 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:50.553 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T23:49:50.553 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T23:49:50.553 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T23:49:50.553 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T23:49:50.553 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T23:49:50.553 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T23:49:50.553 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T23:49:50.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:49:50.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:49:50.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:50.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T23:49:50.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:50.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T23:49:26.449500+0000", 2024-04-26T23:49:50.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:49:50.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.712+0000 7f01a2400700 1 Processor -- start 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.713+0000 7f01a2400700 1 -- start start 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.713+0000 7f01a2400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 0x7f019c07d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.713+0000 7f01a2400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f019c07d930 con 0x7f019c07eeb0 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.714+0000 7f019b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 0x7f019c07d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.714+0000 7f019b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 0x7f019c07d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56732/0 (socket says 172.21.15.40:56732) 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.714+0000 7f019b400700 1 -- 172.21.15.40:0/3862061255 learned_addr learned my addr 172.21.15.40:0/3862061255 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.714+0000 7f019b400700 1 -- 172.21.15.40:0/3862061255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019c07dab0 con 0x7f019c07eeb0 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.714+0000 7f019b400700 1 --2- 172.21.15.40:0/3862061255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 0x7f019c07d360 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f018c0098a0 tx=0x7f018c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b26fbfb4c37ed4c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.715+0000 7f019a000700 1 -- 172.21.15.40:0/3862061255 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f018c004160 con 0x7f019c07eeb0 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.715+0000 7f019a000700 1 -- 172.21.15.40:0/3862061255 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f018c004300 con 0x7f019c07eeb0 2024-04-26T23:49:50.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.715+0000 7f019a000700 1 -- 172.21.15.40:0/3862061255 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f018c011af0 con 0x7f019c07eeb0 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.716+0000 7f01a2400700 1 -- 172.21.15.40:0/3862061255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 msgr2=0x7f019c07d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.716+0000 7f01a2400700 1 --2- 172.21.15.40:0/3862061255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 0x7f019c07d360 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f018c0098a0 tx=0x7f018c009bb0 comp rx=0 tx=0).stop 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.716+0000 7f01a2400700 1 -- 172.21.15.40:0/3862061255 shutdown_connections 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.716+0000 7f01a2400700 1 --2- 172.21.15.40:0/3862061255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 0x7f019c07d360 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.716+0000 7f01a2400700 1 -- 172.21.15.40:0/3862061255 >> 172.21.15.40:0/3862061255 conn(0x7f019c1042a0 msgr2=0x7f019c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.716+0000 7f01a2400700 1 -- 172.21.15.40:0/3862061255 shutdown_connections 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.716+0000 7f01a2400700 1 -- 172.21.15.40:0/3862061255 wait complete. 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.717+0000 7f01a2400700 1 Processor -- start 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.717+0000 7f01a2400700 1 -- start start 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.717+0000 7f01a2400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 0x7f019c1a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.717+0000 7f01a2400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f019c10ae40 con 0x7f019c07eeb0 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.718+0000 7f019b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 0x7f019c1a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.718+0000 7f019b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 0x7f019c1a31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56744/0 (socket says 172.21.15.40:56744) 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.718+0000 7f019b400700 1 -- 172.21.15.40:0/1405342118 learned_addr learned my addr 172.21.15.40:0/1405342118 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.718+0000 7f019b400700 1 -- 172.21.15.40:0/1405342118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019c1a36f0 con 0x7f019c07eeb0 2024-04-26T23:49:50.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.718+0000 7f019b400700 1 --2- 172.21.15.40:0/1405342118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 0x7f019c1a31b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f018c011ac0 tx=0x7f018c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:50.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.719+0000 7f0193e00700 1 -- 172.21.15.40:0/1405342118 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f018c003b20 con 0x7f019c07eeb0 2024-04-26T23:49:50.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.719+0000 7f01a2400700 1 -- 172.21.15.40:0/1405342118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f019c1a3980 con 0x7f019c07eeb0 2024-04-26T23:49:50.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.719+0000 7f0193e00700 1 -- 172.21.15.40:0/1405342118 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f018c01a8c0 con 0x7f019c07eeb0 2024-04-26T23:49:50.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.719+0000 7f0193e00700 1 -- 172.21.15.40:0/1405342118 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f018c0248d0 con 0x7f019c07eeb0 2024-04-26T23:49:50.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.719+0000 7f01a2400700 1 -- 172.21.15.40:0/1405342118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f019c1a6670 con 0x7f019c07eeb0 2024-04-26T23:49:50.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.719+0000 7f0193e00700 1 -- 172.21.15.40:0/1405342118 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f018c022020 con 0x7f019c07eeb0 2024-04-26T23:49:50.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.719+0000 7f0193e00700 1 -- 172.21.15.40:0/1405342118 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f018c0193c0 con 0x7f019c07eeb0 2024-04-26T23:49:50.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.720+0000 7f01a2400700 1 -- 172.21.15.40:0/1405342118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f019c064a80 con 0x7f019c07eeb0 2024-04-26T23:49:50.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.723+0000 7f0193e00700 1 -- 172.21.15.40:0/1405342118 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f018c042ec0 con 0x7f019c07eeb0 2024-04-26T23:49:50.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.803+0000 7f01a2400700 1 -- 172.21.15.40:0/1405342118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f019c02e690 con 0x7f019c07eeb0 2024-04-26T23:49:50.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.804+0000 7f0193e00700 1 -- 172.21.15.40:0/1405342118 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f018c031020 con 0x7f019c07eeb0 2024-04-26T23:49:50.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.805+0000 7f01a2400700 1 -- 172.21.15.40:0/1405342118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 msgr2=0x7f019c1a31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:50.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.805+0000 7f01a2400700 1 --2- 172.21.15.40:0/1405342118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 0x7f019c1a31b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f018c011ac0 tx=0x7f018c011ba0 comp rx=0 tx=0).stop 2024-04-26T23:49:50.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.805+0000 7f01a2400700 1 -- 172.21.15.40:0/1405342118 shutdown_connections 2024-04-26T23:49:50.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.805+0000 7f01a2400700 1 --2- 172.21.15.40:0/1405342118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f019c07eeb0 0x7f019c1a31b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:50.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.805+0000 7f01a2400700 1 -- 172.21.15.40:0/1405342118 >> 172.21.15.40:0/1405342118 conn(0x7f019c1042a0 msgr2=0x7f019c106690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:50.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.806+0000 7f01a2400700 1 -- 172.21.15.40:0/1405342118 shutdown_connections 2024-04-26T23:49:50.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:49.806+0000 7f01a2400700 1 -- 172.21.15.40:0/1405342118 wait complete. 2024-04-26T23:49:50.563 INFO:teuthology.orchestra.run.smithi040.stdout:mgr not available, waiting (2/15)... 2024-04-26T23:49:50.764 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:50 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:50.486+0000 7f45e0af8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T23:49:51.219 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:50 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:50.759+0000 7f45e0af8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T23:49:51.719 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:51 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:51.426+0000 7f45e0af8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T23:49:51.720 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:51 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:51.553+0000 7f45e0af8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T23:49:52.198 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:51 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:51.881+0000 7f45e0af8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T23:49:52.198 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:52 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:52.063+0000 7f45e0af8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T23:49:52.198 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:52 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:52.192+0000 7f45e0af8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T23:49:52.470 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:52 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:52.347+0000 7f45e0af8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T23:49:53.035 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:52 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:52.727+0000 7f45e0af8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T23:49:53.664 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:53 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1199396039' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T23:49:53.665 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:53 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:53.533+0000 7f45e0af8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T23:49:53.965 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:49:53 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:49:53.824+0000 7f45e0af8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T23:49:53.972 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:53.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:49:53.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "78cff6ea-0427-11ef-bc93-c7b262605968", 2024-04-26T23:49:53.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T23:49:53.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T23:49:53.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T23:49:53.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T23:49:53.975 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:53.975 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T23:49:53.975 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T23:49:53.975 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2024-04-26T23:49:53.975 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:49:53.975 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T23:49:53.975 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T23:49:53.975 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:49:53.976 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-04-26T23:49:53.976 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T23:49:53.976 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:53.977 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T23:49:53.977 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T23:49:53.977 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:53.977 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T23:49:53.977 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:53.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T23:49:53.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T23:49:53.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T23:49:53.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T23:49:53.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T23:49:53.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T23:49:53.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:53.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T23:49:53.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T23:49:53.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T23:49:53.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T23:49:53.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T23:49:53.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T23:49:53.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T23:49:53.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T23:49:53.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T23:49:53.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:53.987 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T23:49:53.987 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:53.987 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T23:49:53.987 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T23:49:53.987 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:53.987 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T23:49:53.987 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T23:49:53.987 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T23:49:26.449500+0000", 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T23:49:53.988 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.179+0000 7f4e6ec00700 1 Processor -- start 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.179+0000 7f4e6ec00700 1 -- start start 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.179+0000 7f4e6ec00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 0x7f4e6807d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.179+0000 7f4e6ec00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6807d930 con 0x7f4e6807ef00 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.180+0000 7f4e6d800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 0x7f4e6807d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.180+0000 7f4e6d800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 0x7f4e6807d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51760/0 (socket says 172.21.15.40:51760) 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.180+0000 7f4e6d800700 1 -- 172.21.15.40:0/2658753022 learned_addr learned my addr 172.21.15.40:0/2658753022 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.181+0000 7f4e6d800700 1 -- 172.21.15.40:0/2658753022 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e6807dab0 con 0x7f4e6807ef00 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.181+0000 7f4e6d800700 1 --2- 172.21.15.40:0/2658753022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 0x7f4e6807d360 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0098a0 tx=0x7f4e5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db441b1c3b92dd24 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.181+0000 7f4e67e00700 1 -- 172.21.15.40:0/2658753022 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e5c004160 con 0x7f4e6807ef00 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.181+0000 7f4e67e00700 1 -- 172.21.15.40:0/2658753022 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4e5c004300 con 0x7f4e6807ef00 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.181+0000 7f4e67e00700 1 -- 172.21.15.40:0/2658753022 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e5c011af0 con 0x7f4e6807ef00 2024-04-26T23:49:53.989 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.182+0000 7f4e6ec00700 1 -- 172.21.15.40:0/2658753022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 msgr2=0x7f4e6807d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.182+0000 7f4e6ec00700 1 --2- 172.21.15.40:0/2658753022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 0x7f4e6807d360 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0098a0 tx=0x7f4e5c009bb0 comp rx=0 tx=0).stop 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.182+0000 7f4e6ec00700 1 -- 172.21.15.40:0/2658753022 shutdown_connections 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.182+0000 7f4e6ec00700 1 --2- 172.21.15.40:0/2658753022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 0x7f4e6807d360 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.182+0000 7f4e6ec00700 1 -- 172.21.15.40:0/2658753022 >> 172.21.15.40:0/2658753022 conn(0x7f4e681040c0 msgr2=0x7f4e681064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.182+0000 7f4e6ec00700 1 -- 172.21.15.40:0/2658753022 shutdown_connections 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.182+0000 7f4e6ec00700 1 -- 172.21.15.40:0/2658753022 wait complete. 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.183+0000 7f4e6ec00700 1 Processor -- start 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.184+0000 7f4e6ec00700 1 -- start start 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.184+0000 7f4e6ec00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 0x7f4e681a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.184+0000 7f4e6ec00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6810ac90 con 0x7f4e6807ef00 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.184+0000 7f4e6d800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 0x7f4e681a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.184+0000 7f4e6d800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 0x7f4e681a2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51776/0 (socket says 172.21.15.40:51776) 2024-04-26T23:49:53.990 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.184+0000 7f4e6d800700 1 -- 172.21.15.40:0/1199396039 learned_addr learned my addr 172.21.15.40:0/1199396039 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.185+0000 7f4e6d800700 1 -- 172.21.15.40:0/1199396039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e681a3530 con 0x7f4e6807ef00 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.185+0000 7f4e6d800700 1 --2- 172.21.15.40:0/1199396039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 0x7f4e681a2ff0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c003f40 tx=0x7f4e5c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.185+0000 7f4e66000700 1 -- 172.21.15.40:0/1199396039 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e5c0038b0 con 0x7f4e6807ef00 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.185+0000 7f4e66000700 1 -- 172.21.15.40:0/1199396039 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4e5c003a50 con 0x7f4e6807ef00 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.185+0000 7f4e66000700 1 -- 172.21.15.40:0/1199396039 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e5c01a5c0 con 0x7f4e6807ef00 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.185+0000 7f4e6ec00700 1 -- 172.21.15.40:0/1199396039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e681a37c0 con 0x7f4e6807ef00 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.185+0000 7f4e6ec00700 1 -- 172.21.15.40:0/1199396039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e681a64b0 con 0x7f4e6807ef00 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.186+0000 7f4e66000700 1 -- 172.21.15.40:0/1199396039 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4e5c01ac00 con 0x7f4e6807ef00 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.186+0000 7f4e66000700 1 -- 172.21.15.40:0/1199396039 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4e5c019700 con 0x7f4e6807ef00 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.186+0000 7f4e6ec00700 1 -- 172.21.15.40:0/1199396039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e6819c580 con 0x7f4e6807ef00 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.191+0000 7f4e66000700 1 -- 172.21.15.40:0/1199396039 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f4e5c0199a0 con 0x7f4e6807ef00 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.271+0000 7f4e6ec00700 1 -- 172.21.15.40:0/1199396039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4e68064a80 con 0x7f4e6807ef00 2024-04-26T23:49:53.991 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.271+0000 7f4e66000700 1 -- 172.21.15.40:0/1199396039 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f4e5c019b80 con 0x7f4e6807ef00 2024-04-26T23:49:53.992 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.272+0000 7f4e6ec00700 1 -- 172.21.15.40:0/1199396039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 msgr2=0x7f4e681a2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:53.992 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.272+0000 7f4e6ec00700 1 --2- 172.21.15.40:0/1199396039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 0x7f4e681a2ff0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c003f40 tx=0x7f4e5c011890 comp rx=0 tx=0).stop 2024-04-26T23:49:53.992 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.273+0000 7f4e6ec00700 1 -- 172.21.15.40:0/1199396039 shutdown_connections 2024-04-26T23:49:53.992 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.273+0000 7f4e6ec00700 1 --2- 172.21.15.40:0/1199396039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e6807ef00 0x7f4e681a2ff0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:53.992 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.273+0000 7f4e6ec00700 1 -- 172.21.15.40:0/1199396039 >> 172.21.15.40:0/1199396039 conn(0x7f4e681040c0 msgr2=0x7f4e681064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:53.992 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.273+0000 7f4e6ec00700 1 -- 172.21.15.40:0/1199396039 shutdown_connections 2024-04-26T23:49:53.992 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:53.273+0000 7f4e6ec00700 1 -- 172.21.15.40:0/1199396039 wait complete. 2024-04-26T23:49:53.992 INFO:teuthology.orchestra.run.smithi040.stdout:mgr not available, waiting (3/15)... 2024-04-26T23:49:54.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: Activating manager daemon x 2024-04-26T23:49:54.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: mgrmap e2: x(active, starting, since 0.0127383s) 2024-04-26T23:49:54.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: from='mgr.14100 172.21.15.40:0/3810314488' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T23:49:54.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: from='mgr.14100 172.21.15.40:0/3810314488' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T23:49:54.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: from='mgr.14100 172.21.15.40:0/3810314488' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T23:49:54.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: from='mgr.14100 172.21.15.40:0/3810314488' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:49:54.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: from='mgr.14100 172.21.15.40:0/3810314488' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T23:49:54.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: Manager daemon x is now available 2024-04-26T23:49:54.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: from='mgr.14100 172.21.15.40:0/3810314488' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T23:49:54.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: from='mgr.14100 172.21.15.40:0/3810314488' entity='mgr.x' 2024-04-26T23:49:54.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: from='mgr.14100 172.21.15.40:0/3810314488' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T23:49:54.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: from='mgr.14100 172.21.15.40:0/3810314488' entity='mgr.x' 2024-04-26T23:49:54.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:54 smithi040 ceph-mon[28095]: from='mgr.14100 172.21.15.40:0/3810314488' entity='mgr.x' 2024-04-26T23:49:56.148 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:55 smithi040 ceph-mon[28095]: mgrmap e3: x(active, since 1.02202s) 2024-04-26T23:49:57.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:56 smithi040 ceph-mon[28095]: mgrmap e4: x(active, since 2s) 2024-04-26T23:49:57.644 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:57.644 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "78cff6ea-0427-11ef-bc93-c7b262605968", 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:49:57.645 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-04-26T23:49:57.646 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T23:49:57.646 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:57.646 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T23:49:57.646 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T23:49:57.646 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:57.646 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T23:49:57.646 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:57.646 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T23:49:57.646 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T23:49:57.646 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T23:49:57.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T23:49:57.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T23:49:57.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T23:49:57.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:57.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T23:49:57.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T23:49:57.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T23:49:57.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T23:49:57.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T23:49:57.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T23:49:57.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T23:49:57.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T23:49:57.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T23:49:26.449500+0000", 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.598+0000 7fe519400700 1 Processor -- start 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.598+0000 7fe519400700 1 -- start start 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.599+0000 7fe519400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 0x7fe51410b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:57.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.599+0000 7fe519400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe51410b6e0 con 0x7fe51410acf0 2024-04-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.599+0000 7fe513e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 0x7fe51410b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.599+0000 7fe513e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 0x7fe51410b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51840/0 (socket says 172.21.15.40:51840) 2024-04-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.599+0000 7fe513e00700 1 -- 172.21.15.40:0/2145959877 learned_addr learned my addr 172.21.15.40:0/2145959877 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.600+0000 7fe513e00700 1 -- 172.21.15.40:0/2145959877 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51410bef0 con 0x7fe51410acf0 2024-04-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.600+0000 7fe513e00700 1 --2- 172.21.15.40:0/2145959877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 0x7fe51410b110 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe5040098a0 tx=0x7fe504009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca98023e591c50c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.600+0000 7fe512a00700 1 -- 172.21.15.40:0/2145959877 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe504004160 con 0x7fe51410acf0 2024-04-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.600+0000 7fe512a00700 1 -- 172.21.15.40:0/2145959877 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe504004300 con 0x7fe51410acf0 2024-04-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.600+0000 7fe512a00700 1 -- 172.21.15.40:0/2145959877 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe504011af0 con 0x7fe51410acf0 2024-04-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.601+0000 7fe519400700 1 -- 172.21.15.40:0/2145959877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 msgr2=0x7fe51410b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.601+0000 7fe519400700 1 --2- 172.21.15.40:0/2145959877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 0x7fe51410b110 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe5040098a0 tx=0x7fe504009bb0 comp rx=0 tx=0).stop 2024-04-26T23:49:57.658 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.601+0000 7fe519400700 1 -- 172.21.15.40:0/2145959877 shutdown_connections 2024-04-26T23:49:57.659 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.601+0000 7fe519400700 1 --2- 172.21.15.40:0/2145959877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 0x7fe51410b110 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:57.659 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.601+0000 7fe519400700 1 -- 172.21.15.40:0/2145959877 >> 172.21.15.40:0/2145959877 conn(0x7fe51407e0a0 msgr2=0x7fe51407e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:57.659 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.601+0000 7fe519400700 1 -- 172.21.15.40:0/2145959877 shutdown_connections 2024-04-26T23:49:57.659 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.601+0000 7fe519400700 1 -- 172.21.15.40:0/2145959877 wait complete. 2024-04-26T23:49:57.659 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.602+0000 7fe519400700 1 Processor -- start 2024-04-26T23:49:57.659 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.602+0000 7fe519400700 1 -- start start 2024-04-26T23:49:57.659 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.603+0000 7fe519400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 0x7fe51419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:57.659 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.603+0000 7fe519400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe51410f2a0 con 0x7fe51410acf0 2024-04-26T23:49:57.659 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.603+0000 7fe513e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 0x7fe51419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:57.659 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.603+0000 7fe513e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 0x7fe51419ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51850/0 (socket says 172.21.15.40:51850) 2024-04-26T23:49:57.659 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.603+0000 7fe513e00700 1 -- 172.21.15.40:0/1550795982 learned_addr learned my addr 172.21.15.40:0/1550795982 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:57.659 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.603+0000 7fe513e00700 1 -- 172.21.15.40:0/1550795982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51419f210 con 0x7fe51410acf0 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.604+0000 7fe513e00700 1 --2- 172.21.15.40:0/1550795982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 0x7fe51419ecd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe504009fb0 tx=0x7fe504011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.604+0000 7fe510c00700 1 -- 172.21.15.40:0/1550795982 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5040036a0 con 0x7fe51410acf0 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.604+0000 7fe510c00700 1 -- 172.21.15.40:0/1550795982 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe50401a430 con 0x7fe51410acf0 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.604+0000 7fe519400700 1 -- 172.21.15.40:0/1550795982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51419f4a0 con 0x7fe51410acf0 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.604+0000 7fe510c00700 1 -- 172.21.15.40:0/1550795982 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe504019550 con 0x7fe51410acf0 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.604+0000 7fe519400700 1 -- 172.21.15.40:0/1550795982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5141a2180 con 0x7fe51410acf0 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.605+0000 7fe519400700 1 -- 172.21.15.40:0/1550795982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe514064a80 con 0x7fe51410acf0 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.605+0000 7fe510c00700 1 -- 172.21.15.40:0/1550795982 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fe504003840 con 0x7fe51410acf0 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.606+0000 7fe510c00700 1 --2- 172.21.15.40:0/1550795982 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fe4f8039560 0x7fe4f803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.606+0000 7fe510c00700 1 -- 172.21.15.40:0/1550795982 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe50404d850 con 0x7fe51410acf0 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.606+0000 7fe513400700 1 --2- 172.21.15.40:0/1550795982 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fe4f8039560 0x7fe4f803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.607+0000 7fe513400700 1 --2- 172.21.15.40:0/1550795982 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fe4f8039560 0x7fe4f803ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe500006fd0 tx=0x7fe500006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:57.660 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.612+0000 7fe510c00700 1 -- 172.21.15.40:0/1550795982 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe50404d270 con 0x7fe51410acf0 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.931+0000 7fe519400700 1 -- 172.21.15.40:0/1550795982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fe5141a2350 con 0x7fe51410acf0 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.932+0000 7fe510c00700 1 -- 172.21.15.40:0/1550795982 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fe5141a2350 con 0x7fe51410acf0 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.936+0000 7fe519400700 1 -- 172.21.15.40:0/1550795982 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fe4f8039560 msgr2=0x7fe4f803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.936+0000 7fe519400700 1 --2- 172.21.15.40:0/1550795982 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fe4f8039560 0x7fe4f803ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe500006fd0 tx=0x7fe500006e40 comp rx=0 tx=0).stop 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.936+0000 7fe519400700 1 -- 172.21.15.40:0/1550795982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 msgr2=0x7fe51419ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.936+0000 7fe519400700 1 --2- 172.21.15.40:0/1550795982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 0x7fe51419ecd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe504009fb0 tx=0x7fe504011940 comp rx=0 tx=0).stop 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.936+0000 7fe519400700 1 -- 172.21.15.40:0/1550795982 shutdown_connections 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.937+0000 7fe519400700 1 --2- 172.21.15.40:0/1550795982 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fe4f8039560 0x7fe4f803ba20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.937+0000 7fe519400700 1 --2- 172.21.15.40:0/1550795982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe51410acf0 0x7fe51419ecd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.937+0000 7fe519400700 1 -- 172.21.15.40:0/1550795982 >> 172.21.15.40:0/1550795982 conn(0x7fe51407e0a0 msgr2=0x7fe514108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.937+0000 7fe519400700 1 -- 172.21.15.40:0/1550795982 shutdown_connections 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:56.937+0000 7fe519400700 1 -- 172.21.15.40:0/1550795982 wait complete. 2024-04-26T23:49:57.661 INFO:teuthology.orchestra.run.smithi040.stdout:mgr is available 2024-04-26T23:49:57.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:57 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1550795982' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T23:49:59.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:49:58 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/477101622' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T23:49:59.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:59.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T23:49:59.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout fsid = 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:49:59.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T23:49:59.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T23:49:59.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.286+0000 7f0203e00700 1 Processor -- start 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.286+0000 7f0203e00700 1 -- start start 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.287+0000 7f0203e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 0x7f01fc07d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.287+0000 7f0203e00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc07d930 con 0x7f01fc07ef00 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.287+0000 7f0201400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 0x7f01fc07d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.288+0000 7f0201400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 0x7f01fc07d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51862/0 (socket says 172.21.15.40:51862) 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.288+0000 7f0201400700 1 -- 172.21.15.40:0/3194740314 learned_addr learned my addr 172.21.15.40:0/3194740314 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.288+0000 7f0201400700 1 -- 172.21.15.40:0/3194740314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01fc07dab0 con 0x7f01fc07ef00 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.288+0000 7f0201400700 1 --2- 172.21.15.40:0/3194740314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 0x7f01fc07d360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f01f00098a0 tx=0x7f01f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebdacf992585fdd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.289+0000 7f01fbe00700 1 -- 172.21.15.40:0/3194740314 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01f0004160 con 0x7f01fc07ef00 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.289+0000 7f01fbe00700 1 -- 172.21.15.40:0/3194740314 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f01f0004300 con 0x7f01fc07ef00 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.289+0000 7f01fbe00700 1 -- 172.21.15.40:0/3194740314 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01f0011af0 con 0x7f01fc07ef00 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.289+0000 7f0203e00700 1 -- 172.21.15.40:0/3194740314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 msgr2=0x7f01fc07d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.289+0000 7f0203e00700 1 --2- 172.21.15.40:0/3194740314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 0x7f01fc07d360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f01f00098a0 tx=0x7f01f0009bb0 comp rx=0 tx=0).stop 2024-04-26T23:49:59.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.290+0000 7f0203e00700 1 -- 172.21.15.40:0/3194740314 shutdown_connections 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.290+0000 7f0203e00700 1 --2- 172.21.15.40:0/3194740314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 0x7f01fc07d360 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.290+0000 7f0203e00700 1 -- 172.21.15.40:0/3194740314 >> 172.21.15.40:0/3194740314 conn(0x7f01fc1042a0 msgr2=0x7f01fc1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.290+0000 7f0203e00700 1 -- 172.21.15.40:0/3194740314 shutdown_connections 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.290+0000 7f0203e00700 1 -- 172.21.15.40:0/3194740314 wait complete. 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.291+0000 7f0203e00700 1 Processor -- start 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.291+0000 7f0203e00700 1 -- start start 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.291+0000 7f0203e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 0x7f01fc19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.291+0000 7f0203e00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc10f2a0 con 0x7f01fc07ef00 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.292+0000 7f0201400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 0x7f01fc19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.292+0000 7f0201400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 0x7f01fc19ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51872/0 (socket says 172.21.15.40:51872) 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.292+0000 7f0201400700 1 -- 172.21.15.40:0/477101622 learned_addr learned my addr 172.21.15.40:0/477101622 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.292+0000 7f0201400700 1 -- 172.21.15.40:0/477101622 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01fc19f350 con 0x7f01fc07ef00 2024-04-26T23:49:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.293+0000 7f0201400700 1 --2- 172.21.15.40:0/477101622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 0x7f01fc19ee10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f01f0000c00 tx=0x7f01f0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.293+0000 7f01fa000700 1 -- 172.21.15.40:0/477101622 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01f0003b20 con 0x7f01fc07ef00 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.293+0000 7f01fa000700 1 -- 172.21.15.40:0/477101622 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f01f001a8c0 con 0x7f01fc07ef00 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.293+0000 7f01fa000700 1 -- 172.21.15.40:0/477101622 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01f0024940 con 0x7f01fc07ef00 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.293+0000 7f0203e00700 1 -- 172.21.15.40:0/477101622 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01fc19f5e0 con 0x7f01fc07ef00 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.293+0000 7f0203e00700 1 -- 172.21.15.40:0/477101622 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01fc1a22d0 con 0x7f01fc07ef00 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.294+0000 7f01fa000700 1 -- 172.21.15.40:0/477101622 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f01f0022020 con 0x7f01fc07ef00 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.294+0000 7f0203e00700 1 -- 172.21.15.40:0/477101622 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01fc064a80 con 0x7f01fc07ef00 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.295+0000 7f01fa000700 1 --2- 172.21.15.40:0/477101622 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f01e40395b0 0x7f01e403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.295+0000 7f01fa000700 1 -- 172.21.15.40:0/477101622 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f01f004d450 con 0x7f01fc07ef00 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.295+0000 7f0200a00700 1 --2- 172.21.15.40:0/477101622 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f01e40395b0 0x7f01e403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.296+0000 7f0200a00700 1 --2- 172.21.15.40:0/477101622 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f01e40395b0 0x7f01e403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f01e8006fd0 tx=0x7f01e8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.300+0000 7f01fa000700 1 -- 172.21.15.40:0/477101622 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f01f000be50 con 0x7f01fc07ef00 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.533+0000 7f0203e00700 1 -- 172.21.15.40:0/477101622 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f01fc1a24a0 con 0x7f01fc07ef00 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.534+0000 7f01fa000700 1 -- 172.21.15.40:0/477101622 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f01fc1a24a0 con 0x7f01fc07ef00 2024-04-26T23:49:59.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.537+0000 7f0203e00700 1 -- 172.21.15.40:0/477101622 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f01e40395b0 msgr2=0x7f01e403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:59.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.537+0000 7f0203e00700 1 --2- 172.21.15.40:0/477101622 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f01e40395b0 0x7f01e403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f01e8006fd0 tx=0x7f01e8006e40 comp rx=0 tx=0).stop 2024-04-26T23:49:59.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.537+0000 7f0203e00700 1 -- 172.21.15.40:0/477101622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 msgr2=0x7f01fc19ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:49:59.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.537+0000 7f0203e00700 1 --2- 172.21.15.40:0/477101622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 0x7f01fc19ee10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f01f0000c00 tx=0x7f01f0011b70 comp rx=0 tx=0).stop 2024-04-26T23:49:59.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.537+0000 7f0203e00700 1 -- 172.21.15.40:0/477101622 shutdown_connections 2024-04-26T23:49:59.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.537+0000 7f0203e00700 1 --2- 172.21.15.40:0/477101622 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f01e40395b0 0x7f01e403ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:59.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.537+0000 7f0203e00700 1 --2- 172.21.15.40:0/477101622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc07ef00 0x7f01fc19ee10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:49:59.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.537+0000 7f0203e00700 1 -- 172.21.15.40:0/477101622 >> 172.21.15.40:0/477101622 conn(0x7f01fc1042a0 msgr2=0x7f01fc104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:49:59.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.537+0000 7f0203e00700 1 -- 172.21.15.40:0/477101622 shutdown_connections 2024-04-26T23:49:59.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:58.538+0000 7f0203e00700 1 -- 172.21.15.40:0/477101622 wait complete. 2024-04-26T23:49:59.204 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling cephadm module... 2024-04-26T23:50:00.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:00 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1062905447' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T23:50:00.470 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:00 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: ignoring --setuser ceph since I am not root 2024-04-26T23:50:00.470 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:00 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: ignoring --setgroup ceph since I am not root 2024-04-26T23:50:00.865 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:00 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:00.485+0000 7fb635fc3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T23:50:00.865 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:00 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:00.727+0000 7fb635fc3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:50:00.876 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.838+0000 7fc215c00700 1 Processor -- start 2024-04-26T23:50:00.877 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.839+0000 7fc215c00700 1 -- start start 2024-04-26T23:50:00.877 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.839+0000 7fc215c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 0x7fc21007d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:00.877 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.839+0000 7fc215c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc21007d910 con 0x7fc21007eee0 2024-04-26T23:50:00.877 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.839+0000 7fc20fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 0x7fc21007d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:00.877 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.839+0000 7fc20fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 0x7fc21007d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51880/0 (socket says 172.21.15.40:51880) 2024-04-26T23:50:00.877 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.839+0000 7fc20fe00700 1 -- 172.21.15.40:0/58524391 learned_addr learned my addr 172.21.15.40:0/58524391 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:00.880 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.840+0000 7fc20fe00700 1 -- 172.21.15.40:0/58524391 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc21007da90 con 0x7fc21007eee0 2024-04-26T23:50:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.840+0000 7fc20fe00700 1 --2- 172.21.15.40:0/58524391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 0x7fc21007d340 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc2000098a0 tx=0x7fc200009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35754cb67a5186a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.841+0000 7fc20f400700 1 -- 172.21.15.40:0/58524391 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc200004160 con 0x7fc21007eee0 2024-04-26T23:50:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.841+0000 7fc20f400700 1 -- 172.21.15.40:0/58524391 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc200004300 con 0x7fc21007eee0 2024-04-26T23:50:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.841+0000 7fc20f400700 1 -- 172.21.15.40:0/58524391 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc200011af0 con 0x7fc21007eee0 2024-04-26T23:50:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.841+0000 7fc215c00700 1 -- 172.21.15.40:0/58524391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 msgr2=0x7fc21007d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.841+0000 7fc215c00700 1 --2- 172.21.15.40:0/58524391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 0x7fc21007d340 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc2000098a0 tx=0x7fc200009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.842+0000 7fc215c00700 1 -- 172.21.15.40:0/58524391 shutdown_connections 2024-04-26T23:50:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.842+0000 7fc215c00700 1 --2- 172.21.15.40:0/58524391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 0x7fc21007d340 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.842+0000 7fc215c00700 1 -- 172.21.15.40:0/58524391 >> 172.21.15.40:0/58524391 conn(0x7fc2101040c0 msgr2=0x7fc2101064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.842+0000 7fc215c00700 1 -- 172.21.15.40:0/58524391 shutdown_connections 2024-04-26T23:50:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.842+0000 7fc215c00700 1 -- 172.21.15.40:0/58524391 wait complete. 2024-04-26T23:50:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.843+0000 7fc215c00700 1 Processor -- start 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.843+0000 7fc215c00700 1 -- start start 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.843+0000 7fc215c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 0x7fc21019ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.843+0000 7fc215c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc21010f070 con 0x7fc21007eee0 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.844+0000 7fc20fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 0x7fc21019ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.844+0000 7fc20fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 0x7fc21019ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51888/0 (socket says 172.21.15.40:51888) 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.844+0000 7fc20fe00700 1 -- 172.21.15.40:0/1062905447 learned_addr learned my addr 172.21.15.40:0/1062905447 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.844+0000 7fc20fe00700 1 -- 172.21.15.40:0/1062905447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc21019f150 con 0x7fc21007eee0 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.844+0000 7fc20fe00700 1 --2- 172.21.15.40:0/1062905447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 0x7fc21019ec10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc200000c00 tx=0x7fc200011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.845+0000 7fc20d600700 1 -- 172.21.15.40:0/1062905447 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc200003b20 con 0x7fc21007eee0 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.845+0000 7fc20d600700 1 -- 172.21.15.40:0/1062905447 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc20001a8c0 con 0x7fc21007eee0 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.845+0000 7fc20d600700 1 -- 172.21.15.40:0/1062905447 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc200024940 con 0x7fc21007eee0 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.845+0000 7fc215c00700 1 -- 172.21.15.40:0/1062905447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc21019f3e0 con 0x7fc21007eee0 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.845+0000 7fc215c00700 1 -- 172.21.15.40:0/1062905447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2101a20d0 con 0x7fc21007eee0 2024-04-26T23:50:00.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.846+0000 7fc20d600700 1 -- 172.21.15.40:0/1062905447 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fc200022020 con 0x7fc21007eee0 2024-04-26T23:50:00.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.846+0000 7fc215c00700 1 -- 172.21.15.40:0/1062905447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc210064a80 con 0x7fc21007eee0 2024-04-26T23:50:00.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.847+0000 7fc20d600700 1 --2- 172.21.15.40:0/1062905447 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fc1f40395b0 0x7fc1f403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:00.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.847+0000 7fc20d600700 1 -- 172.21.15.40:0/1062905447 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc20004d490 con 0x7fc21007eee0 2024-04-26T23:50:00.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.847+0000 7fc207400700 1 --2- 172.21.15.40:0/1062905447 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fc1f40395b0 0x7fc1f403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:00.885 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.848+0000 7fc207400700 1 --2- 172.21.15.40:0/1062905447 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fc1f40395b0 0x7fc1f403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8006fd0 tx=0x7fc1f8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:49:59.852+0000 7fc20d600700 1 -- 172.21.15.40:0/1062905447 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc20004d1f0 con 0x7fc21007eee0 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.144+0000 7fc215c00700 1 -- 172.21.15.40:0/1062905447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fc2101a22a0 con 0x7fc21007eee0 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.197+0000 7fc20d600700 1 -- 172.21.15.40:0/1062905447 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fc2101a22a0 con 0x7fc21007eee0 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.200+0000 7fc20d600700 1 -- 172.21.15.40:0/1062905447 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fc20001aa70 con 0x7fc21007eee0 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.201+0000 7fc215c00700 1 -- 172.21.15.40:0/1062905447 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fc1f40395b0 msgr2=0x7fc1f403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.201+0000 7fc215c00700 1 --2- 172.21.15.40:0/1062905447 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fc1f40395b0 0x7fc1f403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8006fd0 tx=0x7fc1f8006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.201+0000 7fc215c00700 1 -- 172.21.15.40:0/1062905447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 msgr2=0x7fc21019ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.201+0000 7fc215c00700 1 --2- 172.21.15.40:0/1062905447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 0x7fc21019ec10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc200000c00 tx=0x7fc200011b70 comp rx=0 tx=0).stop 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.201+0000 7fc215c00700 1 -- 172.21.15.40:0/1062905447 shutdown_connections 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.201+0000 7fc215c00700 1 --2- 172.21.15.40:0/1062905447 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fc1f40395b0 0x7fc1f403ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.201+0000 7fc215c00700 1 --2- 172.21.15.40:0/1062905447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc21007eee0 0x7fc21019ec10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.201+0000 7fc215c00700 1 -- 172.21.15.40:0/1062905447 >> 172.21.15.40:0/1062905447 conn(0x7fc2101040c0 msgr2=0x7fc210104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.203+0000 7fc215c00700 1 -- 172.21.15.40:0/1062905447 shutdown_connections 2024-04-26T23:50:00.886 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:00.203+0000 7fc215c00700 1 -- 172.21.15.40:0/1062905447 wait complete. 2024-04-26T23:50:01.176 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:00 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:00.864+0000 7fb635fc3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T23:50:01.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:01 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1062905447' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T23:50:01.432 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:01 smithi040 ceph-mon[28095]: mgrmap e5: x(active, since 6s) 2024-04-26T23:50:01.433 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:01 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:01.276+0000 7fb635fc3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T23:50:01.433 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:01 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:01.391+0000 7fb635fc3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T23:50:02.412 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:02 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/931642668' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T23:50:02.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:50:02.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T23:50:02.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T23:50:02.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T23:50:02.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T23:50:02.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:50:02.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.473+0000 7fb85da00700 1 Processor -- start 2024-04-26T23:50:02.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.473+0000 7fb85da00700 1 -- start start 2024-04-26T23:50:02.435 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.473+0000 7fb85da00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 0x7fb85807d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:02.435 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.473+0000 7fb85da00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb85807d8d0 con 0x7fb85807eea0 2024-04-26T23:50:02.435 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.474+0000 7fb856a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 0x7fb85807d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:02.435 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.474+0000 7fb856a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 0x7fb85807d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32878/0 (socket says 172.21.15.40:32878) 2024-04-26T23:50:02.435 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.474+0000 7fb856a00700 1 -- 172.21.15.40:0/3282060085 learned_addr learned my addr 172.21.15.40:0/3282060085 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:02.435 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.474+0000 7fb856a00700 1 -- 172.21.15.40:0/3282060085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb85807da50 con 0x7fb85807eea0 2024-04-26T23:50:02.435 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.474+0000 7fb856a00700 1 --2- 172.21.15.40:0/3282060085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 0x7fb85807d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb8480098a0 tx=0x7fb848009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0c27c847e27d4d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:02.435 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.475+0000 7fb855600700 1 -- 172.21.15.40:0/3282060085 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb848004160 con 0x7fb85807eea0 2024-04-26T23:50:02.435 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.475+0000 7fb855600700 1 -- 172.21.15.40:0/3282060085 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb848004300 con 0x7fb85807eea0 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.475+0000 7fb855600700 1 -- 172.21.15.40:0/3282060085 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb848011af0 con 0x7fb85807eea0 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.476+0000 7fb85da00700 1 -- 172.21.15.40:0/3282060085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 msgr2=0x7fb85807d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.476+0000 7fb85da00700 1 --2- 172.21.15.40:0/3282060085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 0x7fb85807d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb8480098a0 tx=0x7fb848009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.476+0000 7fb85da00700 1 -- 172.21.15.40:0/3282060085 shutdown_connections 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.476+0000 7fb85da00700 1 --2- 172.21.15.40:0/3282060085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 0x7fb85807d300 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.476+0000 7fb85da00700 1 -- 172.21.15.40:0/3282060085 >> 172.21.15.40:0/3282060085 conn(0x7fb8581042a0 msgr2=0x7fb8581066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.476+0000 7fb85da00700 1 -- 172.21.15.40:0/3282060085 shutdown_connections 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.476+0000 7fb85da00700 1 -- 172.21.15.40:0/3282060085 wait complete. 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.477+0000 7fb85da00700 1 Processor -- start 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.477+0000 7fb85da00700 1 -- start start 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.478+0000 7fb85da00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 0x7fb85819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.478+0000 7fb85da00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb85810f2b0 con 0x7fb85807eea0 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.478+0000 7fb856a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 0x7fb85819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.478+0000 7fb856a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 0x7fb85819edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32894/0 (socket says 172.21.15.40:32894) 2024-04-26T23:50:02.436 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.478+0000 7fb856a00700 1 -- 172.21.15.40:0/931642668 learned_addr learned my addr 172.21.15.40:0/931642668 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.478+0000 7fb856a00700 1 -- 172.21.15.40:0/931642668 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb85819f2f0 con 0x7fb85807eea0 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.479+0000 7fb856a00700 1 --2- 172.21.15.40:0/931642668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 0x7fb85819edb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb848011ac0 tx=0x7fb848011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.479+0000 7fb84f400700 1 -- 172.21.15.40:0/931642668 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb848003b20 con 0x7fb85807eea0 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.479+0000 7fb85da00700 1 -- 172.21.15.40:0/931642668 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb85819f580 con 0x7fb85807eea0 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.479+0000 7fb84f400700 1 -- 172.21.15.40:0/931642668 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb84801a8c0 con 0x7fb85807eea0 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.479+0000 7fb84f400700 1 -- 172.21.15.40:0/931642668 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8480248d0 con 0x7fb85807eea0 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.479+0000 7fb85da00700 1 -- 172.21.15.40:0/931642668 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8581a2270 con 0x7fb85807eea0 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.480+0000 7fb84f400700 1 -- 172.21.15.40:0/931642668 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fb848022020 con 0x7fb85807eea0 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.480+0000 7fb85da00700 1 -- 172.21.15.40:0/931642668 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb858064a80 con 0x7fb85807eea0 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.481+0000 7fb84f400700 1 --2- 172.21.15.40:0/931642668 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fb83c039660 0x7fb83c03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.481+0000 7fb84f400700 1 -- 172.21.15.40:0/931642668 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb84804d4f0 con 0x7fb85807eea0 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.481+0000 7fb856000700 1 -- 172.21.15.40:0/931642668 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fb83c039660 msgr2=0x7fb83c03bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/673764112 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.481+0000 7fb856000700 1 --2- 172.21.15.40:0/931642668 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fb83c039660 0x7fb83c03bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:50:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.486+0000 7fb84f400700 1 -- 172.21.15.40:0/931642668 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb84800f2e0 con 0x7fb85807eea0 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.682+0000 7fb856000700 1 -- 172.21.15.40:0/931642668 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fb83c039660 msgr2=0x7fb83c03bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/673764112 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.682+0000 7fb856000700 1 --2- 172.21.15.40:0/931642668 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fb83c039660 0x7fb83c03bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.771+0000 7fb85da00700 1 -- 172.21.15.40:0/931642668 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fb8581a2440 con 0x7fb85807eea0 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.772+0000 7fb84f400700 1 -- 172.21.15.40:0/931642668 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fb8581a2440 con 0x7fb85807eea0 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.775+0000 7fb85da00700 1 -- 172.21.15.40:0/931642668 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fb83c039660 msgr2=0x7fb83c03bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.775+0000 7fb85da00700 1 --2- 172.21.15.40:0/931642668 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fb83c039660 0x7fb83c03bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.775+0000 7fb85da00700 1 -- 172.21.15.40:0/931642668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 msgr2=0x7fb85819edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.775+0000 7fb85da00700 1 --2- 172.21.15.40:0/931642668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 0x7fb85819edb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb848011ac0 tx=0x7fb848011ba0 comp rx=0 tx=0).stop 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.776+0000 7fb85da00700 1 -- 172.21.15.40:0/931642668 shutdown_connections 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.776+0000 7fb85da00700 1 --2- 172.21.15.40:0/931642668 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7fb83c039660 0x7fb83c03bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.776+0000 7fb85da00700 1 --2- 172.21.15.40:0/931642668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb85807eea0 0x7fb85819edb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.776+0000 7fb85da00700 1 -- 172.21.15.40:0/931642668 >> 172.21.15.40:0/931642668 conn(0x7fb8581042a0 msgr2=0x7fb858104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.776+0000 7fb85da00700 1 -- 172.21.15.40:0/931642668 shutdown_connections 2024-04-26T23:50:02.438 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:01.776+0000 7fb85da00700 1 -- 172.21.15.40:0/931642668 wait complete. 2024-04-26T23:50:02.439 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for the mgr to restart... 2024-04-26T23:50:02.439 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr epoch 5... 2024-04-26T23:50:03.053 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:02 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:02.990+0000 7fb635fc3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T23:50:03.449 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:03 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:03.224+0000 7fb635fc3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T23:50:03.719 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:03 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:03.444+0000 7fb635fc3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T23:50:04.219 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:03 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:03.833+0000 7fb635fc3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T23:50:04.220 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:03 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:03.959+0000 7fb635fc3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T23:50:04.481 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:04 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:04.216+0000 7fb635fc3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T23:50:04.970 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:04 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:04.476+0000 7fb635fc3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:50:05.720 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:05 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:05.303+0000 7fb635fc3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T23:50:06.220 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:05 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:05.872+0000 7fb635fc3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T23:50:06.220 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:06 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:06.008+0000 7fb635fc3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T23:50:06.719 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:06 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:06.281+0000 7fb635fc3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T23:50:07.441 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:06 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:06.973+0000 7fb635fc3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T23:50:07.441 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:07 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:07.101+0000 7fb635fc3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T23:50:07.719 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:07 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:07.436+0000 7fb635fc3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T23:50:07.720 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:07 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:07.618+0000 7fb635fc3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T23:50:08.219 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:07 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:07.751+0000 7fb635fc3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T23:50:08.220 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:07 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:07.908+0000 7fb635fc3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T23:50:08.720 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:08 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:08.311+0000 7fb635fc3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T23:50:09.219 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:09.097+0000 7fb635fc3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T23:50:09.720 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:09.395+0000 7fb635fc3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T23:50:09.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-mon[28095]: Active manager daemon x restarted 2024-04-26T23:50:09.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-mon[28095]: Activating manager daemon x 2024-04-26T23:50:09.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-mon[28095]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T23:50:09.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-mon[28095]: mgrmap e6: x(active, starting, since 0.00992151s) 2024-04-26T23:50:09.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:50:09.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T23:50:09.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T23:50:09.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T23:50:09.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T23:50:09.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-mon[28095]: Manager daemon x is now available 2024-04-26T23:50:09.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:09 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:10.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:10 smithi040 ceph-mon[28095]: Found migration_current of "None". Setting to last migration. 2024-04-26T23:50:10.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:10 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:10.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:10 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:50:10.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:10 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:50:10.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:10 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T23:50:10.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:10 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T23:50:10.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:10 smithi040 ceph-mon[28095]: mgrmap e7: x(active, since 1.01716s) 2024-04-26T23:50:11.147 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:50:11.147 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T23:50:11.147 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T23:50:11.147 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:50:11.147 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.048+0000 7f1fd3800700 1 Processor -- start 2024-04-26T23:50:11.147 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.048+0000 7f1fd3800700 1 -- start start 2024-04-26T23:50:11.147 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.049+0000 7f1fd3800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 0x7f1fcc10bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:11.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.049+0000 7f1fd3800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc07ced0 con 0x7f1fcc109740 2024-04-26T23:50:11.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.049+0000 7f1fd0e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 0x7f1fcc10bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:11.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.049+0000 7f1fd0e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 0x7f1fcc10bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32898/0 (socket says 172.21.15.40:32898) 2024-04-26T23:50:11.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.049+0000 7f1fd0e00700 1 -- 172.21.15.40:0/813134498 learned_addr learned my addr 172.21.15.40:0/813134498 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:11.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.050+0000 7f1fd0e00700 1 -- 172.21.15.40:0/813134498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fcc10c070 con 0x7f1fcc109740 2024-04-26T23:50:11.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.050+0000 7f1fd0e00700 1 --2- 172.21.15.40:0/813134498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 0x7f1fcc10bb30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0009e80 tx=0x7f1fc000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a74aa20d8ca7a84 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:11.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.050+0000 7f1fcb400700 1 -- 172.21.15.40:0/813134498 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fc0017070 con 0x7f1fcc109740 2024-04-26T23:50:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.051+0000 7f1fcb400700 1 -- 172.21.15.40:0/813134498 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1fc000b8c0 con 0x7f1fcc109740 2024-04-26T23:50:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.051+0000 7f1fcb400700 1 -- 172.21.15.40:0/813134498 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fc0004490 con 0x7f1fcc109740 2024-04-26T23:50:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.051+0000 7f1fd3800700 1 -- 172.21.15.40:0/813134498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 msgr2=0x7f1fcc10bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:11.155 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.051+0000 7f1fd3800700 1 --2- 172.21.15.40:0/813134498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 0x7f1fcc10bb30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0009e80 tx=0x7f1fc000b330 comp rx=0 tx=0).stop 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.051+0000 7f1fd3800700 1 -- 172.21.15.40:0/813134498 shutdown_connections 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.051+0000 7f1fd3800700 1 --2- 172.21.15.40:0/813134498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 0x7f1fcc10bb30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.051+0000 7f1fd3800700 1 -- 172.21.15.40:0/813134498 >> 172.21.15.40:0/813134498 conn(0x7f1fcc1031d0 msgr2=0x7f1fcc1055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.052+0000 7f1fd3800700 1 -- 172.21.15.40:0/813134498 shutdown_connections 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.052+0000 7f1fd3800700 1 -- 172.21.15.40:0/813134498 wait complete. 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.053+0000 7f1fd3800700 1 Processor -- start 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.053+0000 7f1fd3800700 1 -- start start 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.053+0000 7f1fd3800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 0x7f1fcc19cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.053+0000 7f1fd3800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc10d040 con 0x7f1fcc109740 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.054+0000 7f1fd0e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 0x7f1fcc19cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.054+0000 7f1fd0e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 0x7f1fcc19cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32910/0 (socket says 172.21.15.40:32910) 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.054+0000 7f1fd0e00700 1 -- 172.21.15.40:0/2324373240 learned_addr learned my addr 172.21.15.40:0/2324373240 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:11.156 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.054+0000 7f1fd0e00700 1 -- 172.21.15.40:0/2324373240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fcc19d120 con 0x7f1fcc109740 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.055+0000 7f1fd0e00700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 0x7f1fcc19cbe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000b7b0 tx=0x7f1fc000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.055+0000 7f1fc9600700 1 -- 172.21.15.40:0/2324373240 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fc0020020 con 0x7f1fcc109740 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.055+0000 7f1fc9600700 1 -- 172.21.15.40:0/2324373240 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1fc0003d10 con 0x7f1fcc109740 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.055+0000 7f1fd3800700 1 -- 172.21.15.40:0/2324373240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fcc19d3b0 con 0x7f1fcc109740 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.055+0000 7f1fc9600700 1 -- 172.21.15.40:0/2324373240 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fc0017040 con 0x7f1fcc109740 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.055+0000 7f1fd3800700 1 -- 172.21.15.40:0/2324373240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fcc1a00a0 con 0x7f1fcc109740 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.057+0000 7f1fc9600700 1 -- 172.21.15.40:0/2324373240 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f1fc0023020 con 0x7f1fcc109740 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.057+0000 7f1fc9600700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 0x7f1fb403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.057+0000 7f1fc9600700 1 -- 172.21.15.40:0/2324373240 --> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1fb403c180 con 0x7f1fb40395b0 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.057+0000 7f1fc9600700 1 -- 172.21.15.40:0/2324373240 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1fc000f050 con 0x7f1fcc109740 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.057+0000 7f1fcbe00700 1 -- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 msgr2=0x7f1fb403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/673764112 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.057+0000 7f1fcbe00700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 0x7f1fb403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.258+0000 7f1fcbe00700 1 -- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 msgr2=0x7f1fb403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/673764112 2024-04-26T23:50:11.157 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.258+0000 7f1fcbe00700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 0x7f1fb403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.659+0000 7f1fcbe00700 1 -- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 msgr2=0x7f1fb403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/673764112 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:03.659+0000 7f1fcbe00700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 0x7f1fb403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:04.460+0000 7f1fcbe00700 1 -- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 msgr2=0x7f1fb403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/673764112 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:04.460+0000 7f1fcbe00700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 0x7f1fb403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:06.062+0000 7f1fcbe00700 1 -- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 msgr2=0x7f1fb403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/673764112 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:06.062+0000 7f1fcbe00700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 0x7f1fb403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:09.266+0000 7f1fcbe00700 1 -- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 msgr2=0x7f1fb403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/673764112 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:09.266+0000 7f1fcbe00700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 0x7f1fb403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:09.405+0000 7f1fc9600700 1 -- 172.21.15.40:0/2324373240 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f1fc0012900 con 0x7f1fcc109740 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:09.405+0000 7f1fc9600700 1 -- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 msgr2=0x7f1fb403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:09.405+0000 7f1fc9600700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 0x7f1fb403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.409+0000 7f1fc9600700 1 -- 172.21.15.40:0/2324373240 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f1fc004e7a0 con 0x7f1fcc109740 2024-04-26T23:50:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.409+0000 7f1fc9600700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f1fb403cea0 0x7f1fb403f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.409+0000 7f1fc9600700 1 -- 172.21.15.40:0/2324373240 --> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1fc0022e70 con 0x7f1fb403cea0 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.411+0000 7f1fcbe00700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f1fb403cea0 0x7f1fb403f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.414+0000 7f1fcbe00700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f1fb403cea0 0x7f1fb403f290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8003a10 tx=0x7f1fb8009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.416+0000 7f1fc9600700 1 -- 172.21.15.40:0/2324373240 <== mgr.14120 v2:172.21.15.40:6800/1568598447 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f1fc0022e70 con 0x7f1fb403cea0 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.427+0000 7f1fd3800700 1 -- 172.21.15.40:0/2324373240 --> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f1fcc064a80 con 0x7f1fb403cea0 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.428+0000 7f1fc9600700 1 -- 172.21.15.40:0/2324373240 <== mgr.14120 v2:172.21.15.40:6800/1568598447 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f1fcc064a80 con 0x7f1fb403cea0 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.428+0000 7f1fd3800700 1 -- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f1fb403cea0 msgr2=0x7f1fb403f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.428+0000 7f1fd3800700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f1fb403cea0 0x7f1fb403f290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8003a10 tx=0x7f1fb8009290 comp rx=0 tx=0).stop 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.428+0000 7f1fd3800700 1 -- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 msgr2=0x7f1fcc19cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.428+0000 7f1fd3800700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 0x7f1fcc19cbe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000b7b0 tx=0x7f1fc000be90 comp rx=0 tx=0).stop 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.428+0000 7f1fd3800700 1 -- 172.21.15.40:0/2324373240 shutdown_connections 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.428+0000 7f1fd3800700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f1fb403cea0 0x7f1fb403f290 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.428+0000 7f1fd3800700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc109740 0x7f1fcc19cbe0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:11.159 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.428+0000 7f1fd3800700 1 --2- 172.21.15.40:0/2324373240 >> [v2:172.21.15.40:6800/673764112,v1:172.21.15.40:6801/673764112] conn(0x7f1fb40395b0 0x7f1fb403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.428+0000 7f1fd3800700 1 -- 172.21.15.40:0/2324373240 >> 172.21.15.40:0/2324373240 conn(0x7f1fcc1031d0 msgr2=0x7f1fcc103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.429+0000 7f1fd3800700 1 -- 172.21.15.40:0/2324373240 shutdown_connections 2024-04-26T23:50:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:10.429+0000 7f1fd3800700 1 -- 172.21.15.40:0/2324373240 wait complete. 2024-04-26T23:50:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:mgr epoch 5 is available 2024-04-26T23:50:11.160 INFO:teuthology.orchestra.run.smithi040.stdout:Setting orchestrator backend to cephadm... 2024-04-26T23:50:12.428 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:12 smithi040 ceph-mon[28095]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T23:50:12.429 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:12 smithi040 ceph-mon[28095]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T23:50:12.429 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:12 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:12.429 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:12 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:12.429 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:12 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:12.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.760+0000 7f144d200700 1 Processor -- start 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.760+0000 7f144d200700 1 -- start start 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.760+0000 7f144d200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 0x7f1448109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.760+0000 7f144d200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1448109700 con 0x7f1448108d10 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.761+0000 7f1445e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 0x7f1448109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.761+0000 7f1445e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 0x7f1448109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47984/0 (socket says 172.21.15.40:47984) 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.761+0000 7f1445e00700 1 -- 172.21.15.40:0/2175947156 learned_addr learned my addr 172.21.15.40:0/2175947156 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.761+0000 7f1445e00700 1 -- 172.21.15.40:0/2175947156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1448109f10 con 0x7f1448108d10 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.762+0000 7f1445e00700 1 --2- 172.21.15.40:0/2175947156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 0x7f1448109130 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f14380098a0 tx=0x7f1438009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667b048112df01a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.762+0000 7f1444a00700 1 -- 172.21.15.40:0/2175947156 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1438004160 con 0x7f1448108d10 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.762+0000 7f1444a00700 1 -- 172.21.15.40:0/2175947156 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1438004300 con 0x7f1448108d10 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.762+0000 7f1444a00700 1 -- 172.21.15.40:0/2175947156 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1438011af0 con 0x7f1448108d10 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.763+0000 7f144d200700 1 -- 172.21.15.40:0/2175947156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 msgr2=0x7f1448109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:12.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.763+0000 7f144d200700 1 --2- 172.21.15.40:0/2175947156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 0x7f1448109130 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f14380098a0 tx=0x7f1438009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:12.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.763+0000 7f144d200700 1 -- 172.21.15.40:0/2175947156 shutdown_connections 2024-04-26T23:50:12.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.763+0000 7f144d200700 1 --2- 172.21.15.40:0/2175947156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 0x7f1448109130 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:12.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.763+0000 7f144d200700 1 -- 172.21.15.40:0/2175947156 >> 172.21.15.40:0/2175947156 conn(0x7f14481042f0 msgr2=0x7f1448106710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:12.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.763+0000 7f144d200700 1 -- 172.21.15.40:0/2175947156 shutdown_connections 2024-04-26T23:50:12.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.763+0000 7f144d200700 1 -- 172.21.15.40:0/2175947156 wait complete. 2024-04-26T23:50:12.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.764+0000 7f144d200700 1 Processor -- start 2024-04-26T23:50:12.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.764+0000 7f144d200700 1 -- start start 2024-04-26T23:50:12.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.765+0000 7f144d200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 0x7f144819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:12.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.765+0000 7f144d200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f144810d2c0 con 0x7f1448108d10 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.765+0000 7f1445e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 0x7f144819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.765+0000 7f1445e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 0x7f144819ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48000/0 (socket says 172.21.15.40:48000) 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.765+0000 7f1445e00700 1 -- 172.21.15.40:0/2861936451 learned_addr learned my addr 172.21.15.40:0/2861936451 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.765+0000 7f1445e00700 1 -- 172.21.15.40:0/2861936451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144819d200 con 0x7f1448108d10 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.766+0000 7f1445e00700 1 --2- 172.21.15.40:0/2861936451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 0x7f144819ccc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f14380117b0 tx=0x7f1438011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.766+0000 7f143ea00700 1 -- 172.21.15.40:0/2861936451 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1438003af0 con 0x7f1448108d10 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.766+0000 7f143ea00700 1 -- 172.21.15.40:0/2861936451 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1438020070 con 0x7f1448108d10 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.766+0000 7f143ea00700 1 -- 172.21.15.40:0/2861936451 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1438019630 con 0x7f1448108d10 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.766+0000 7f144d200700 1 -- 172.21.15.40:0/2861936451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144819d490 con 0x7f1448108d10 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.766+0000 7f144d200700 1 -- 172.21.15.40:0/2861936451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14481a0180 con 0x7f1448108d10 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.767+0000 7f144d200700 1 -- 172.21.15.40:0/2861936451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1448064a80 con 0x7f1448108d10 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.767+0000 7f143ea00700 1 -- 172.21.15.40:0/2861936451 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f143801a9c0 con 0x7f1448108d10 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.768+0000 7f143ea00700 1 --2- 172.21.15.40:0/2861936451 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f142c039490 0x7f142c03b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:12.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.768+0000 7f143ea00700 1 -- 172.21.15.40:0/2861936451 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f143804da00 con 0x7f1448108d10 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.768+0000 7f1445400700 1 --2- 172.21.15.40:0/2861936451 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f142c039490 0x7f142c03b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.769+0000 7f1445400700 1 --2- 172.21.15.40:0/2861936451 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f142c039490 0x7f142c03b950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1434006fd0 tx=0x7f1434006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:11.774+0000 7f143ea00700 1 -- 172.21.15.40:0/2861936451 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f143800e260 con 0x7f1448108d10 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:12.018+0000 7f144d200700 1 -- 172.21.15.40:0/2861936451 --> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f1448105560 con 0x7f142c039490 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:12.029+0000 7f143ea00700 1 -- 172.21.15.40:0/2861936451 <== mgr.14120 v2:172.21.15.40:6800/1568598447 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1448105560 con 0x7f142c039490 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:12.032+0000 7f144d200700 1 -- 172.21.15.40:0/2861936451 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f142c039490 msgr2=0x7f142c03b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:12.032+0000 7f144d200700 1 --2- 172.21.15.40:0/2861936451 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f142c039490 0x7f142c03b950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1434006fd0 tx=0x7f1434006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:12.032+0000 7f144d200700 1 -- 172.21.15.40:0/2861936451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 msgr2=0x7f144819ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:12.032+0000 7f144d200700 1 --2- 172.21.15.40:0/2861936451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 0x7f144819ccc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f14380117b0 tx=0x7f1438011b70 comp rx=0 tx=0).stop 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:12.032+0000 7f144d200700 1 -- 172.21.15.40:0/2861936451 shutdown_connections 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:12.032+0000 7f144d200700 1 --2- 172.21.15.40:0/2861936451 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f142c039490 0x7f142c03b950 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:12.032+0000 7f144d200700 1 --2- 172.21.15.40:0/2861936451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1448108d10 0x7f144819ccc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:12.032+0000 7f144d200700 1 -- 172.21.15.40:0/2861936451 >> 172.21.15.40:0/2861936451 conn(0x7f14481042f0 msgr2=0x7f1448104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:12.741 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:12.033+0000 7f144d200700 1 -- 172.21.15.40:0/2861936451 shutdown_connections 2024-04-26T23:50:12.742 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:12.033+0000 7f144d200700 1 -- 172.21.15.40:0/2861936451 wait complete. 2024-04-26T23:50:13.311 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:13 smithi040 ceph-mon[28095]: [26/Apr/2024:23:50:11] ENGINE Bus STARTING 2024-04-26T23:50:13.311 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:13 smithi040 ceph-mon[28095]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:50:13.311 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:13 smithi040 ceph-mon[28095]: mgrmap e8: x(active, since 2s) 2024-04-26T23:50:13.311 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:13 smithi040 ceph-mon[28095]: [26/Apr/2024:23:50:12] ENGINE Serving on https://172.21.15.40:7150 2024-04-26T23:50:13.311 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:13 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:50:13.311 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:13 smithi040 ceph-mon[28095]: [26/Apr/2024:23:50:12] ENGINE Serving on http://172.21.15.40:8765 2024-04-26T23:50:13.311 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:13 smithi040 ceph-mon[28095]: [26/Apr/2024:23:50:12] ENGINE Bus STARTED 2024-04-26T23:50:13.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:13 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:50:14.209 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T23:50:14.210 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.306+0000 7f5845600700 1 Processor -- start 2024-04-26T23:50:14.210 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.306+0000 7f5845600700 1 -- start start 2024-04-26T23:50:14.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.306+0000 7f5845600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 0x7f5840108f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:14.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.306+0000 7f5845600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58401094d0 con 0x7f5840108ae0 2024-04-26T23:50:14.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.307+0000 7f583fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 0x7f5840108f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:14.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.307+0000 7f583fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 0x7f5840108f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48004/0 (socket says 172.21.15.40:48004) 2024-04-26T23:50:14.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.307+0000 7f583fe00700 1 -- 172.21.15.40:0/3854207015 learned_addr learned my addr 172.21.15.40:0/3854207015 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:14.213 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.307+0000 7f583fe00700 1 -- 172.21.15.40:0/3854207015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5840109ce0 con 0x7f5840108ae0 2024-04-26T23:50:14.214 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.308+0000 7f583fe00700 1 --2- 172.21.15.40:0/3854207015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 0x7f5840108f00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f58300098a0 tx=0x7f5830009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd3c7814a5002aa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:14.214 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.308+0000 7f583ea00700 1 -- 172.21.15.40:0/3854207015 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5830004160 con 0x7f5840108ae0 2024-04-26T23:50:14.214 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.308+0000 7f583ea00700 1 -- 172.21.15.40:0/3854207015 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5830004300 con 0x7f5840108ae0 2024-04-26T23:50:14.214 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.308+0000 7f583ea00700 1 -- 172.21.15.40:0/3854207015 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5830011af0 con 0x7f5840108ae0 2024-04-26T23:50:14.214 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.309+0000 7f5845600700 1 -- 172.21.15.40:0/3854207015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 msgr2=0x7f5840108f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:14.214 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.309+0000 7f5845600700 1 --2- 172.21.15.40:0/3854207015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 0x7f5840108f00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f58300098a0 tx=0x7f5830009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:14.214 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.309+0000 7f5845600700 1 -- 172.21.15.40:0/3854207015 shutdown_connections 2024-04-26T23:50:14.215 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.309+0000 7f5845600700 1 --2- 172.21.15.40:0/3854207015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 0x7f5840108f00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:14.215 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.309+0000 7f5845600700 1 -- 172.21.15.40:0/3854207015 >> 172.21.15.40:0/3854207015 conn(0x7f58401040c0 msgr2=0x7f58401064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:14.215 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.309+0000 7f5845600700 1 -- 172.21.15.40:0/3854207015 shutdown_connections 2024-04-26T23:50:14.215 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.309+0000 7f5845600700 1 -- 172.21.15.40:0/3854207015 wait complete. 2024-04-26T23:50:14.215 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.310+0000 7f5845600700 1 Processor -- start 2024-04-26T23:50:14.216 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.310+0000 7f5845600700 1 -- start start 2024-04-26T23:50:14.216 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.311+0000 7f5845600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 0x7f584007eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:14.216 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.311+0000 7f5845600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f584010d090 con 0x7f5840108ae0 2024-04-26T23:50:14.217 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.311+0000 7f583fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 0x7f584007eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:14.217 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.311+0000 7f583fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 0x7f584007eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48014/0 (socket says 172.21.15.40:48014) 2024-04-26T23:50:14.217 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.311+0000 7f583fe00700 1 -- 172.21.15.40:0/143414101 learned_addr learned my addr 172.21.15.40:0/143414101 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:14.217 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.311+0000 7f583fe00700 1 -- 172.21.15.40:0/143414101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584007f0d0 con 0x7f5840108ae0 2024-04-26T23:50:14.217 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.312+0000 7f583fe00700 1 --2- 172.21.15.40:0/143414101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 0x7f584007eb90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5830000c00 tx=0x7f5830011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:14.217 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.312+0000 7f583cc00700 1 -- 172.21.15.40:0/143414101 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5830003b20 con 0x7f5840108ae0 2024-04-26T23:50:14.218 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.312+0000 7f583cc00700 1 -- 172.21.15.40:0/143414101 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f583001a8c0 con 0x7f5840108ae0 2024-04-26T23:50:14.218 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.312+0000 7f583cc00700 1 -- 172.21.15.40:0/143414101 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5830024940 con 0x7f5840108ae0 2024-04-26T23:50:14.218 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.312+0000 7f5845600700 1 -- 172.21.15.40:0/143414101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584007d2c0 con 0x7f5840108ae0 2024-04-26T23:50:14.218 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.312+0000 7f5845600700 1 -- 172.21.15.40:0/143414101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f584007d800 con 0x7f5840108ae0 2024-04-26T23:50:14.218 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.313+0000 7f583cc00700 1 -- 172.21.15.40:0/143414101 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5830022020 con 0x7f5840108ae0 2024-04-26T23:50:14.220 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.313+0000 7f5845600700 1 -- 172.21.15.40:0/143414101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5840064a80 con 0x7f5840108ae0 2024-04-26T23:50:14.220 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.314+0000 7f583cc00700 1 --2- 172.21.15.40:0/143414101 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f58240395b0 0x7f582403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:14.220 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.314+0000 7f583cc00700 1 -- 172.21.15.40:0/143414101 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f583004d8f0 con 0x7f5840108ae0 2024-04-26T23:50:14.220 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.314+0000 7f583f400700 1 --2- 172.21.15.40:0/143414101 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f58240395b0 0x7f582403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:14.220 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.315+0000 7f583f400700 1 --2- 172.21.15.40:0/143414101 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f58240395b0 0x7f582403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f582c006fd0 tx=0x7f582c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:14.220 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.320+0000 7f583cc00700 1 -- 172.21.15.40:0/143414101 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f583000be50 con 0x7f5840108ae0 2024-04-26T23:50:14.220 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.547+0000 7f5845600700 1 -- 172.21.15.40:0/143414101 --> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f58401052c0 con 0x7f58240395b0 2024-04-26T23:50:14.220 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.548+0000 7f583cc00700 1 -- 172.21.15.40:0/143414101 <== mgr.14120 v2:172.21.15.40:6800/1568598447 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f58401052c0 con 0x7f58240395b0 2024-04-26T23:50:14.221 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.551+0000 7f5845600700 1 -- 172.21.15.40:0/143414101 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f58240395b0 msgr2=0x7f582403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:14.221 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.551+0000 7f5845600700 1 --2- 172.21.15.40:0/143414101 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f58240395b0 0x7f582403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f582c006fd0 tx=0x7f582c006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:14.221 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.551+0000 7f5845600700 1 -- 172.21.15.40:0/143414101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 msgr2=0x7f584007eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:14.221 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.551+0000 7f5845600700 1 --2- 172.21.15.40:0/143414101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 0x7f584007eb90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5830000c00 tx=0x7f5830011b70 comp rx=0 tx=0).stop 2024-04-26T23:50:14.221 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.551+0000 7f5845600700 1 -- 172.21.15.40:0/143414101 shutdown_connections 2024-04-26T23:50:14.221 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.551+0000 7f5845600700 1 --2- 172.21.15.40:0/143414101 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f58240395b0 0x7f582403ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:14.221 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.552+0000 7f5845600700 1 --2- 172.21.15.40:0/143414101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5840108ae0 0x7f584007eb90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:14.222 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.552+0000 7f5845600700 1 -- 172.21.15.40:0/143414101 >> 172.21.15.40:0/143414101 conn(0x7f58401040c0 msgr2=0x7f5840104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:14.222 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.552+0000 7f5845600700 1 -- 172.21.15.40:0/143414101 shutdown_connections 2024-04-26T23:50:14.222 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:13.552+0000 7f5845600700 1 -- 172.21.15.40:0/143414101 wait complete. 2024-04-26T23:50:14.222 INFO:teuthology.orchestra.run.smithi040.stdout:Generating ssh key... 2024-04-26T23:50:14.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:14 smithi040 ceph-mon[28095]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:50:15.510 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-mon[28095]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:50:15.511 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-mon[28095]: Generating ssh key... 2024-04-26T23:50:15.842 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: Generating public/private rsa key pair. 2024-04-26T23:50:15.843 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: Your identification has been saved in /tmp/tmpzsc969sy/key. 2024-04-26T23:50:15.843 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: Your public key has been saved in /tmp/tmpzsc969sy/key.pub. 2024-04-26T23:50:15.843 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: The key fingerprint is: 2024-04-26T23:50:15.843 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: SHA256:y5Qthtkb+f2Wnew8uNymYsMEsqv+HidIZJ5Ve2wlTiA ceph-78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:50:15.843 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: The key's randomart image is: 2024-04-26T23:50:15.843 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: +---[RSA 3072]----+ 2024-04-26T23:50:15.843 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: | E o.o . | 2024-04-26T23:50:15.843 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: | o = o | 2024-04-26T23:50:15.844 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: | o . . = | 2024-04-26T23:50:15.844 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: | + o= =o | 2024-04-26T23:50:15.844 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: | +o S o | 2024-04-26T23:50:15.844 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: | . .= * o | 2024-04-26T23:50:15.844 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: | . o=.+ . oo.| 2024-04-26T23:50:15.844 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: | .+ =.o==.| 2024-04-26T23:50:15.844 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: | .o+o . o=*+.| 2024-04-26T23:50:15.844 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: +----[SHA256]-----+ 2024-04-26T23:50:16.223 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.976+0000 7f3a23800700 1 Processor -- start 2024-04-26T23:50:16.223 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.976+0000 7f3a23800700 1 -- start start 2024-04-26T23:50:16.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.976+0000 7f3a23800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 0x7f3a1c07d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:16.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.976+0000 7f3a23800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c07d910 con 0x7f3a1c07eee0 2024-04-26T23:50:16.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.977+0000 7f3a20e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 0x7f3a1c07d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:16.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.977+0000 7f3a20e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 0x7f3a1c07d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48026/0 (socket says 172.21.15.40:48026) 2024-04-26T23:50:16.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.977+0000 7f3a20e00700 1 -- 172.21.15.40:0/406290697 learned_addr learned my addr 172.21.15.40:0/406290697 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:16.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.977+0000 7f3a20e00700 1 -- 172.21.15.40:0/406290697 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c07da90 con 0x7f3a1c07eee0 2024-04-26T23:50:16.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.978+0000 7f3a20e00700 1 --2- 172.21.15.40:0/406290697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 0x7f3a1c07d340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3a100098a0 tx=0x7f3a10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f9f73ab7c6bd6e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:16.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.978+0000 7f3a1b400700 1 -- 172.21.15.40:0/406290697 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a10004160 con 0x7f3a1c07eee0 2024-04-26T23:50:16.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.978+0000 7f3a1b400700 1 -- 172.21.15.40:0/406290697 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3a10004300 con 0x7f3a1c07eee0 2024-04-26T23:50:16.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.978+0000 7f3a1b400700 1 -- 172.21.15.40:0/406290697 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a10011af0 con 0x7f3a1c07eee0 2024-04-26T23:50:16.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.979+0000 7f3a23800700 1 -- 172.21.15.40:0/406290697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 msgr2=0x7f3a1c07d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:16.224 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.979+0000 7f3a23800700 1 --2- 172.21.15.40:0/406290697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 0x7f3a1c07d340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3a100098a0 tx=0x7f3a10009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.979+0000 7f3a23800700 1 -- 172.21.15.40:0/406290697 shutdown_connections 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.979+0000 7f3a23800700 1 --2- 172.21.15.40:0/406290697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 0x7f3a1c07d340 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.979+0000 7f3a23800700 1 -- 172.21.15.40:0/406290697 >> 172.21.15.40:0/406290697 conn(0x7f3a1c104280 msgr2=0x7f3a1c1066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.979+0000 7f3a23800700 1 -- 172.21.15.40:0/406290697 shutdown_connections 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.979+0000 7f3a23800700 1 -- 172.21.15.40:0/406290697 wait complete. 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.980+0000 7f3a23800700 1 Processor -- start 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.980+0000 7f3a23800700 1 -- start start 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.980+0000 7f3a23800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 0x7f3a1c19ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.980+0000 7f3a23800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c10f250 con 0x7f3a1c07eee0 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.981+0000 7f3a20e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 0x7f3a1c19ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.981+0000 7f3a20e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 0x7f3a1c19ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48038/0 (socket says 172.21.15.40:48038) 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.981+0000 7f3a20e00700 1 -- 172.21.15.40:0/65178888 learned_addr learned my addr 172.21.15.40:0/65178888 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.981+0000 7f3a20e00700 1 -- 172.21.15.40:0/65178888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c19f370 con 0x7f3a1c07eee0 2024-04-26T23:50:16.225 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.982+0000 7f3a20e00700 1 --2- 172.21.15.40:0/65178888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 0x7f3a1c19ee30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3a10000c00 tx=0x7f3a10011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.982+0000 7f3a19600700 1 -- 172.21.15.40:0/65178888 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a10003b20 con 0x7f3a1c07eee0 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.982+0000 7f3a19600700 1 -- 172.21.15.40:0/65178888 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3a1001a8c0 con 0x7f3a1c07eee0 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.982+0000 7f3a19600700 1 -- 172.21.15.40:0/65178888 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a10024940 con 0x7f3a1c07eee0 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.982+0000 7f3a23800700 1 -- 172.21.15.40:0/65178888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1c19f600 con 0x7f3a1c07eee0 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.982+0000 7f3a23800700 1 -- 172.21.15.40:0/65178888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a1c1a22f0 con 0x7f3a1c07eee0 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.983+0000 7f3a19600700 1 -- 172.21.15.40:0/65178888 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f3a10022020 con 0x7f3a1c07eee0 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.983+0000 7f3a23800700 1 -- 172.21.15.40:0/65178888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a1c064a80 con 0x7f3a1c07eee0 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.983+0000 7f3a19600700 1 --2- 172.21.15.40:0/65178888 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f3a04039600 0x7f3a0403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.984+0000 7f3a19600700 1 -- 172.21.15.40:0/65178888 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3a1004d7a0 con 0x7f3a1c07eee0 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.984+0000 7f3a1be00700 1 --2- 172.21.15.40:0/65178888 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f3a04039600 0x7f3a0403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.985+0000 7f3a1be00700 1 --2- 172.21.15.40:0/65178888 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f3a04039600 0x7f3a0403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c006fd0 tx=0x7f3a0c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:14.990+0000 7f3a19600700 1 -- 172.21.15.40:0/65178888 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3a1000f2e0 con 0x7f3a1c07eee0 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:15.222+0000 7f3a23800700 1 -- 172.21.15.40:0/65178888 --> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f3a1c105460 con 0x7f3a04039600 2024-04-26T23:50:16.226 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:15.528+0000 7f3a19600700 1 -- 172.21.15.40:0/65178888 <== mgr.14120 v2:172.21.15.40:6800/1568598447 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3a1c105460 con 0x7f3a04039600 2024-04-26T23:50:16.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:15.531+0000 7f3a23800700 1 -- 172.21.15.40:0/65178888 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f3a04039600 msgr2=0x7f3a0403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:16.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:15.531+0000 7f3a23800700 1 --2- 172.21.15.40:0/65178888 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f3a04039600 0x7f3a0403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c006fd0 tx=0x7f3a0c006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:16.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:15.531+0000 7f3a23800700 1 -- 172.21.15.40:0/65178888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 msgr2=0x7f3a1c19ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:16.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:15.531+0000 7f3a23800700 1 --2- 172.21.15.40:0/65178888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 0x7f3a1c19ee30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3a10000c00 tx=0x7f3a10011b70 comp rx=0 tx=0).stop 2024-04-26T23:50:16.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:15.531+0000 7f3a23800700 1 -- 172.21.15.40:0/65178888 shutdown_connections 2024-04-26T23:50:16.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:15.531+0000 7f3a23800700 1 --2- 172.21.15.40:0/65178888 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f3a04039600 0x7f3a0403bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:16.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:15.531+0000 7f3a23800700 1 --2- 172.21.15.40:0/65178888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a1c07eee0 0x7f3a1c19ee30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:16.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:15.531+0000 7f3a23800700 1 -- 172.21.15.40:0/65178888 >> 172.21.15.40:0/65178888 conn(0x7f3a1c104280 msgr2=0x7f3a1c104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:16.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:15.531+0000 7f3a23800700 1 -- 172.21.15.40:0/65178888 shutdown_connections 2024-04-26T23:50:16.227 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:15.531+0000 7f3a23800700 1 -- 172.21.15.40:0/65178888 wait complete. 2024-04-26T23:50:16.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:16 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:16.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:16 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:17.860 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:17 smithi040 ceph-mon[28095]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:50:17.875 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtKSNHcphSzzYxj2ic/R3850yAJOBUchuH7beZzqgJ18Tji5m+IpiFiDtxaLUpmpytT2sLKdZNS0xmzhXgb6l6jLQdN7cbFr0SLfMiD3WdKG+Se6QoxqcYN2pk+3B8UQ+ck6oeirYsLsqjxJ8D/uA8uYIBfuvzFKWAXeNFjC4yrflfakuQrUyorccUFbTAC+nc8TLHP/uB3GqghDkpHLamK3y+bahUexqTa83iSWCV/x5LNBKyOnidHhSL+1VQbLBX+yp7fCCLZ+G3KxHYVQIJTH9Tg+QlzoSNZAqMLmyUpkBmtBnUVGzPFab9Ek6/X3EQMPf1mph9oUrkjNW3L6UUNfCyLxfvnJQ38/Aj5vdT9ICREZhWkujJU0r+wTRShxgr8feKlj0S3h6zSWbtVmX8VvBmcWe5qvg6cF960Mlsi5e7dMKbhPJnHdmx2dNzRNP1JiIfZ842uLMk+1qrtlLZ81WVwEKdoyUQDINEJw3POb87kMCR5Xrx5sKSXXVbAbk= ceph-78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:50:17.875 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.863+0000 7fbd8ec00700 1 Processor -- start 2024-04-26T23:50:17.875 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.863+0000 7fbd8ec00700 1 -- start start 2024-04-26T23:50:17.875 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.864+0000 7fbd8ec00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 0x7fbd881090a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:17.875 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.864+0000 7fbd8ec00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbd88109670 con 0x7fbd88108c80 2024-04-26T23:50:17.875 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.864+0000 7fbd87e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 0x7fbd881090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:17.875 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.865+0000 7fbd87e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 0x7fbd881090a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48040/0 (socket says 172.21.15.40:48040) 2024-04-26T23:50:17.875 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.865+0000 7fbd87e00700 1 -- 172.21.15.40:0/2810326814 learned_addr learned my addr 172.21.15.40:0/2810326814 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:17.880 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.865+0000 7fbd87e00700 1 -- 172.21.15.40:0/2810326814 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd88109e80 con 0x7fbd88108c80 2024-04-26T23:50:17.880 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.865+0000 7fbd87e00700 1 --2- 172.21.15.40:0/2810326814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 0x7fbd881090a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbd780098a0 tx=0x7fbd78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b1f0bd67a09f60a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:17.880 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.866+0000 7fbd86a00700 1 -- 172.21.15.40:0/2810326814 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd78004160 con 0x7fbd88108c80 2024-04-26T23:50:17.880 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.866+0000 7fbd86a00700 1 -- 172.21.15.40:0/2810326814 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbd78004300 con 0x7fbd88108c80 2024-04-26T23:50:17.880 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.866+0000 7fbd86a00700 1 -- 172.21.15.40:0/2810326814 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd78011af0 con 0x7fbd88108c80 2024-04-26T23:50:17.880 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.867+0000 7fbd8ec00700 1 -- 172.21.15.40:0/2810326814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 msgr2=0x7fbd881090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.867+0000 7fbd8ec00700 1 --2- 172.21.15.40:0/2810326814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 0x7fbd881090a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbd780098a0 tx=0x7fbd78009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.867+0000 7fbd8ec00700 1 -- 172.21.15.40:0/2810326814 shutdown_connections 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.867+0000 7fbd8ec00700 1 --2- 172.21.15.40:0/2810326814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 0x7fbd881090a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.867+0000 7fbd8ec00700 1 -- 172.21.15.40:0/2810326814 >> 172.21.15.40:0/2810326814 conn(0x7fbd88104260 msgr2=0x7fbd88106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.867+0000 7fbd8ec00700 1 -- 172.21.15.40:0/2810326814 shutdown_connections 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.867+0000 7fbd8ec00700 1 -- 172.21.15.40:0/2810326814 wait complete. 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.868+0000 7fbd8ec00700 1 Processor -- start 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.868+0000 7fbd8ec00700 1 -- start start 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.869+0000 7fbd8ec00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 0x7fbd8819cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.869+0000 7fbd8ec00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8810d230 con 0x7fbd88108c80 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.869+0000 7fbd87e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 0x7fbd8819cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.869+0000 7fbd87e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 0x7fbd8819cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48050/0 (socket says 172.21.15.40:48050) 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.869+0000 7fbd87e00700 1 -- 172.21.15.40:0/3701230400 learned_addr learned my addr 172.21.15.40:0/3701230400 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:17.881 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.869+0000 7fbd87e00700 1 -- 172.21.15.40:0/3701230400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8819d130 con 0x7fbd88108c80 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.869+0000 7fbd87e00700 1 --2- 172.21.15.40:0/3701230400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 0x7fbd8819cbf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbd78000c00 tx=0x7fbd78011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.870+0000 7fbd84c00700 1 -- 172.21.15.40:0/3701230400 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd78003b20 con 0x7fbd88108c80 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.870+0000 7fbd84c00700 1 -- 172.21.15.40:0/3701230400 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbd7801a8c0 con 0x7fbd88108c80 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.870+0000 7fbd84c00700 1 -- 172.21.15.40:0/3701230400 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd78024940 con 0x7fbd88108c80 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.870+0000 7fbd8ec00700 1 -- 172.21.15.40:0/3701230400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd8819d3c0 con 0x7fbd88108c80 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.870+0000 7fbd8ec00700 1 -- 172.21.15.40:0/3701230400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd881a00b0 con 0x7fbd88108c80 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.871+0000 7fbd84c00700 1 -- 172.21.15.40:0/3701230400 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fbd78022020 con 0x7fbd88108c80 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.871+0000 7fbd8ec00700 1 -- 172.21.15.40:0/3701230400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd88064a80 con 0x7fbd88108c80 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.872+0000 7fbd84c00700 1 --2- 172.21.15.40:0/3701230400 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7fbd6c039600 0x7fbd6c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.872+0000 7fbd84c00700 1 -- 172.21.15.40:0/3701230400 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbd7804d7e0 con 0x7fbd88108c80 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.872+0000 7fbd87400700 1 --2- 172.21.15.40:0/3701230400 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7fbd6c039600 0x7fbd6c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.873+0000 7fbd87400700 1 --2- 172.21.15.40:0/3701230400 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7fbd6c039600 0x7fbd6c03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbd74006fd0 tx=0x7fbd74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:17.882 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:16.877+0000 7fbd84c00700 1 -- 172.21.15.40:0/3701230400 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbd7800be50 con 0x7fbd88108c80 2024-04-26T23:50:17.883 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:17.107+0000 7fbd8ec00700 1 -- 172.21.15.40:0/3701230400 --> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fbd88105440 con 0x7fbd6c039600 2024-04-26T23:50:17.883 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:17.108+0000 7fbd84c00700 1 -- 172.21.15.40:0/3701230400 <== mgr.14120 v2:172.21.15.40:6800/1568598447 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fbd88105440 con 0x7fbd6c039600 2024-04-26T23:50:17.883 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:17.111+0000 7fbd8ec00700 1 -- 172.21.15.40:0/3701230400 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7fbd6c039600 msgr2=0x7fbd6c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:17.883 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:17.111+0000 7fbd8ec00700 1 --2- 172.21.15.40:0/3701230400 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7fbd6c039600 0x7fbd6c03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbd74006fd0 tx=0x7fbd74006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:17.883 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:17.111+0000 7fbd8ec00700 1 -- 172.21.15.40:0/3701230400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 msgr2=0x7fbd8819cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:17.883 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:17.111+0000 7fbd8ec00700 1 --2- 172.21.15.40:0/3701230400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 0x7fbd8819cbf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbd78000c00 tx=0x7fbd78011b70 comp rx=0 tx=0).stop 2024-04-26T23:50:17.883 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:17.111+0000 7fbd8ec00700 1 -- 172.21.15.40:0/3701230400 shutdown_connections 2024-04-26T23:50:17.883 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:17.111+0000 7fbd8ec00700 1 --2- 172.21.15.40:0/3701230400 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7fbd6c039600 0x7fbd6c03bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:17.883 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:17.111+0000 7fbd8ec00700 1 --2- 172.21.15.40:0/3701230400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd88108c80 0x7fbd8819cbf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:17.883 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:17.111+0000 7fbd8ec00700 1 -- 172.21.15.40:0/3701230400 >> 172.21.15.40:0/3701230400 conn(0x7fbd88104260 msgr2=0x7fbd88104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:17.883 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:17.111+0000 7fbd8ec00700 1 -- 172.21.15.40:0/3701230400 shutdown_connections 2024-04-26T23:50:17.883 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:17.111+0000 7fbd8ec00700 1 -- 172.21.15.40:0/3701230400 wait complete. 2024-04-26T23:50:17.884 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T23:50:17.884 INFO:teuthology.orchestra.run.smithi040.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T23:50:17.892 INFO:teuthology.orchestra.run.smithi040.stdout:Adding host smithi040... 2024-04-26T23:50:19.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:19 smithi040 ceph-mon[28095]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi040", "addr": "172.21.15.40", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:50:20.667 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:20 smithi040 ceph-mon[28095]: Deploying cephadm binary to smithi040 2024-04-26T23:50:25.736 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Added host 'smithi040' with addr '172.21.15.40' 2024-04-26T23:50:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.527+0000 7f25e9400700 1 Processor -- start 2024-04-26T23:50:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.528+0000 7f25e9400700 1 -- start start 2024-04-26T23:50:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.528+0000 7f25e9400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 0x7f25e410b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.528+0000 7f25e9400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25e410b700 con 0x7f25e410ad10 2024-04-26T23:50:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.529+0000 7f25e3e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 0x7f25e410b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.529+0000 7f25e3e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 0x7f25e410b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48064/0 (socket says 172.21.15.40:48064) 2024-04-26T23:50:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.529+0000 7f25e3e00700 1 -- 172.21.15.40:0/2371430339 learned_addr learned my addr 172.21.15.40:0/2371430339 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.529+0000 7f25e3e00700 1 -- 172.21.15.40:0/2371430339 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e410bf10 con 0x7f25e410ad10 2024-04-26T23:50:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.530+0000 7f25e3e00700 1 --2- 172.21.15.40:0/2371430339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 0x7f25e410b130 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f25d40098a0 tx=0x7f25d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7626164aa46c5278 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.530+0000 7f25e2a00700 1 -- 172.21.15.40:0/2371430339 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25d4004160 con 0x7f25e410ad10 2024-04-26T23:50:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.530+0000 7f25e2a00700 1 -- 172.21.15.40:0/2371430339 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f25d4004300 con 0x7f25e410ad10 2024-04-26T23:50:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.530+0000 7f25e2a00700 1 -- 172.21.15.40:0/2371430339 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25d4011af0 con 0x7f25e410ad10 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.531+0000 7f25e9400700 1 -- 172.21.15.40:0/2371430339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 msgr2=0x7f25e410b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.531+0000 7f25e9400700 1 --2- 172.21.15.40:0/2371430339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 0x7f25e410b130 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f25d40098a0 tx=0x7f25d4009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.531+0000 7f25e9400700 1 -- 172.21.15.40:0/2371430339 shutdown_connections 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.531+0000 7f25e9400700 1 --2- 172.21.15.40:0/2371430339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 0x7f25e410b130 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.531+0000 7f25e9400700 1 -- 172.21.15.40:0/2371430339 >> 172.21.15.40:0/2371430339 conn(0x7f25e407e0d0 msgr2=0x7f25e407e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.531+0000 7f25e9400700 1 -- 172.21.15.40:0/2371430339 shutdown_connections 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.531+0000 7f25e9400700 1 -- 172.21.15.40:0/2371430339 wait complete. 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.532+0000 7f25e9400700 1 Processor -- start 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.532+0000 7f25e9400700 1 -- start start 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.532+0000 7f25e9400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 0x7f25e419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.532+0000 7f25e9400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25e410f2c0 con 0x7f25e410ad10 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.533+0000 7f25e3e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 0x7f25e419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.533+0000 7f25e3e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 0x7f25e419eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48072/0 (socket says 172.21.15.40:48072) 2024-04-26T23:50:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.533+0000 7f25e3e00700 1 -- 172.21.15.40:0/62474688 learned_addr learned my addr 172.21.15.40:0/62474688 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.533+0000 7f25e3e00700 1 -- 172.21.15.40:0/62474688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e419f1e0 con 0x7f25e410ad10 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.533+0000 7f25e3e00700 1 --2- 172.21.15.40:0/62474688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 0x7f25e419eca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009fb0 tx=0x7f25d4011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.534+0000 7f25e0c00700 1 -- 172.21.15.40:0/62474688 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25d40036a0 con 0x7f25e410ad10 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.534+0000 7f25e0c00700 1 -- 172.21.15.40:0/62474688 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f25d401a430 con 0x7f25e410ad10 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.534+0000 7f25e0c00700 1 -- 172.21.15.40:0/62474688 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25d4019550 con 0x7f25e410ad10 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.534+0000 7f25e9400700 1 -- 172.21.15.40:0/62474688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e419f470 con 0x7f25e410ad10 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.534+0000 7f25e9400700 1 -- 172.21.15.40:0/62474688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e41a2160 con 0x7f25e410ad10 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.535+0000 7f25e9400700 1 -- 172.21.15.40:0/62474688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e4064a80 con 0x7f25e410ad10 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.535+0000 7f25e0c00700 1 -- 172.21.15.40:0/62474688 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f25d401a8e0 con 0x7f25e410ad10 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.536+0000 7f25e0c00700 1 --2- 172.21.15.40:0/62474688 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f25c80395b0 0x7f25c803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.536+0000 7f25e0c00700 1 -- 172.21.15.40:0/62474688 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f25d404d990 con 0x7f25e410ad10 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.536+0000 7f25e3400700 1 --2- 172.21.15.40:0/62474688 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f25c80395b0 0x7f25c803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.537+0000 7f25e3400700 1 --2- 172.21.15.40:0/62474688 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f25c80395b0 0x7f25c803ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f25d0006fd0 tx=0x7f25d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:25.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.541+0000 7f25e0c00700 1 -- 172.21.15.40:0/62474688 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f25d401a690 con 0x7f25e410ad10 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:18.777+0000 7f25e9400700 1 -- 172.21.15.40:0/62474688 --> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi040", "addr": "172.21.15.40", "target": ["mon-mgr", ""]}) -- 0x7f25e4108c70 con 0x7f25c80395b0 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:25.047+0000 7f25e0c00700 1 -- 172.21.15.40:0/62474688 <== mgr.14120 v2:172.21.15.40:6800/1568598447 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f25e4108c70 con 0x7f25c80395b0 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:25.051+0000 7f25e9400700 1 -- 172.21.15.40:0/62474688 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f25c80395b0 msgr2=0x7f25c803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:25.051+0000 7f25e9400700 1 --2- 172.21.15.40:0/62474688 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f25c80395b0 0x7f25c803ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f25d0006fd0 tx=0x7f25d0006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:25.051+0000 7f25e9400700 1 -- 172.21.15.40:0/62474688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 msgr2=0x7f25e419eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:25.051+0000 7f25e9400700 1 --2- 172.21.15.40:0/62474688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 0x7f25e419eca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009fb0 tx=0x7f25d4011940 comp rx=0 tx=0).stop 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:25.051+0000 7f25e9400700 1 -- 172.21.15.40:0/62474688 shutdown_connections 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:25.051+0000 7f25e9400700 1 --2- 172.21.15.40:0/62474688 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f25c80395b0 0x7f25c803ba70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:25.051+0000 7f25e9400700 1 --2- 172.21.15.40:0/62474688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25e410ad10 0x7f25e419eca0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:25.051+0000 7f25e9400700 1 -- 172.21.15.40:0/62474688 >> 172.21.15.40:0/62474688 conn(0x7f25e407e0d0 msgr2=0x7f25e4108560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:25.051+0000 7f25e9400700 1 -- 172.21.15.40:0/62474688 shutdown_connections 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:25.052+0000 7f25e9400700 1 -- 172.21.15.40:0/62474688 wait complete. 2024-04-26T23:50:25.740 INFO:teuthology.orchestra.run.smithi040.stdout:Deploying unmanaged mon service... 2024-04-26T23:50:26.372 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:26 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:26.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:26 smithi040 ceph-mon[28095]: Added host smithi040 2024-04-26T23:50:26.373 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:26 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:50:27.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T23:50:27.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.388+0000 7f9f90c00700 1 Processor -- start 2024-04-26T23:50:27.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.388+0000 7f9f90c00700 1 -- start start 2024-04-26T23:50:27.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.389+0000 7f9f90c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 0x7f9f8c109100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:27.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.389+0000 7f9f90c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c1096d0 con 0x7f9f8c108ce0 2024-04-26T23:50:27.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.389+0000 7f9f89e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 0x7f9f8c109100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:27.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.389+0000 7f9f89e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 0x7f9f8c109100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49382/0 (socket says 172.21.15.40:49382) 2024-04-26T23:50:27.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.389+0000 7f9f89e00700 1 -- 172.21.15.40:0/2096564218 learned_addr learned my addr 172.21.15.40:0/2096564218 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:27.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.390+0000 7f9f89e00700 1 -- 172.21.15.40:0/2096564218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c109ee0 con 0x7f9f8c108ce0 2024-04-26T23:50:27.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.390+0000 7f9f89e00700 1 --2- 172.21.15.40:0/2096564218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 0x7f9f8c109100 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0098a0 tx=0x7f9f7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda2671f0bba0f61 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:27.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.390+0000 7f9f88a00700 1 -- 172.21.15.40:0/2096564218 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f7c004160 con 0x7f9f8c108ce0 2024-04-26T23:50:27.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.390+0000 7f9f88a00700 1 -- 172.21.15.40:0/2096564218 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9f7c004300 con 0x7f9f8c108ce0 2024-04-26T23:50:27.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.390+0000 7f9f88a00700 1 -- 172.21.15.40:0/2096564218 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f7c011af0 con 0x7f9f8c108ce0 2024-04-26T23:50:27.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.391+0000 7f9f90c00700 1 -- 172.21.15.40:0/2096564218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 msgr2=0x7f9f8c109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:27.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.391+0000 7f9f90c00700 1 --2- 172.21.15.40:0/2096564218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 0x7f9f8c109100 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0098a0 tx=0x7f9f7c009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:27.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.391+0000 7f9f90c00700 1 -- 172.21.15.40:0/2096564218 shutdown_connections 2024-04-26T23:50:27.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.391+0000 7f9f90c00700 1 --2- 172.21.15.40:0/2096564218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 0x7f9f8c109100 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:27.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.391+0000 7f9f90c00700 1 -- 172.21.15.40:0/2096564218 >> 172.21.15.40:0/2096564218 conn(0x7f9f8c1042a0 msgr2=0x7f9f8c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:27.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.392+0000 7f9f90c00700 1 -- 172.21.15.40:0/2096564218 shutdown_connections 2024-04-26T23:50:27.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.392+0000 7f9f90c00700 1 -- 172.21.15.40:0/2096564218 wait complete. 2024-04-26T23:50:27.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.393+0000 7f9f90c00700 1 Processor -- start 2024-04-26T23:50:27.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.393+0000 7f9f90c00700 1 -- start start 2024-04-26T23:50:27.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.393+0000 7f9f90c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 0x7f9f8c19ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:27.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.393+0000 7f9f90c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c10d290 con 0x7f9f8c108ce0 2024-04-26T23:50:27.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.394+0000 7f9f89e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 0x7f9f8c19ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:27.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.394+0000 7f9f89e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 0x7f9f8c19ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49388/0 (socket says 172.21.15.40:49388) 2024-04-26T23:50:27.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.394+0000 7f9f89e00700 1 -- 172.21.15.40:0/333355244 learned_addr learned my addr 172.21.15.40:0/333355244 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:27.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.394+0000 7f9f89e00700 1 -- 172.21.15.40:0/333355244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c19f370 con 0x7f9f8c108ce0 2024-04-26T23:50:27.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.394+0000 7f9f89e00700 1 --2- 172.21.15.40:0/333355244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 0x7f9f8c19ee30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c000c00 tx=0x7f9f7c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:27.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.395+0000 7f9f82a00700 1 -- 172.21.15.40:0/333355244 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f7c003b20 con 0x7f9f8c108ce0 2024-04-26T23:50:27.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.395+0000 7f9f82a00700 1 -- 172.21.15.40:0/333355244 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9f7c01a8c0 con 0x7f9f8c108ce0 2024-04-26T23:50:27.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.395+0000 7f9f90c00700 1 -- 172.21.15.40:0/333355244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8c19f600 con 0x7f9f8c108ce0 2024-04-26T23:50:27.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.395+0000 7f9f82a00700 1 -- 172.21.15.40:0/333355244 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f7c024940 con 0x7f9f8c108ce0 2024-04-26T23:50:27.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.395+0000 7f9f90c00700 1 -- 172.21.15.40:0/333355244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8c1a22f0 con 0x7f9f8c108ce0 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.396+0000 7f9f82a00700 1 -- 172.21.15.40:0/333355244 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f9f7c022020 con 0x7f9f8c108ce0 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.396+0000 7f9f90c00700 1 -- 172.21.15.40:0/333355244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f8c064a80 con 0x7f9f8c108ce0 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.396+0000 7f9f82a00700 1 --2- 172.21.15.40:0/333355244 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9f70039660 0x7f9f7003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.397+0000 7f9f82a00700 1 -- 172.21.15.40:0/333355244 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9f7c04d8a0 con 0x7f9f8c108ce0 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.397+0000 7f9f89400700 1 --2- 172.21.15.40:0/333355244 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9f70039660 0x7f9f7003bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.397+0000 7f9f89400700 1 --2- 172.21.15.40:0/333355244 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9f70039660 0x7f9f7003bb20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9f78006fd0 tx=0x7f9f78006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.402+0000 7f9f82a00700 1 -- 172.21.15.40:0/333355244 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9f7c00be50 con 0x7f9f8c108ce0 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.652+0000 7f9f90c00700 1 -- 172.21.15.40:0/333355244 --> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f9f8c1054a0 con 0x7f9f70039660 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.662+0000 7f9f82a00700 1 -- 172.21.15.40:0/333355244 <== mgr.14120 v2:172.21.15.40:6800/1568598447 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9f8c1054a0 con 0x7f9f70039660 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.665+0000 7f9f90c00700 1 -- 172.21.15.40:0/333355244 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9f70039660 msgr2=0x7f9f7003bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.665+0000 7f9f90c00700 1 --2- 172.21.15.40:0/333355244 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9f70039660 0x7f9f7003bb20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9f78006fd0 tx=0x7f9f78006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.665+0000 7f9f90c00700 1 -- 172.21.15.40:0/333355244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 msgr2=0x7f9f8c19ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:27.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.665+0000 7f9f90c00700 1 --2- 172.21.15.40:0/333355244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 0x7f9f8c19ee30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c000c00 tx=0x7f9f7c011b70 comp rx=0 tx=0).stop 2024-04-26T23:50:27.573 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.665+0000 7f9f90c00700 1 -- 172.21.15.40:0/333355244 shutdown_connections 2024-04-26T23:50:27.573 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.665+0000 7f9f90c00700 1 --2- 172.21.15.40:0/333355244 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9f70039660 0x7f9f7003bb20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:27.573 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.665+0000 7f9f90c00700 1 --2- 172.21.15.40:0/333355244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f8c108ce0 0x7f9f8c19ee30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:27.573 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.665+0000 7f9f90c00700 1 -- 172.21.15.40:0/333355244 >> 172.21.15.40:0/333355244 conn(0x7f9f8c1042a0 msgr2=0x7f9f8c104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:27.573 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.665+0000 7f9f90c00700 1 -- 172.21.15.40:0/333355244 shutdown_connections 2024-04-26T23:50:27.573 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:26.665+0000 7f9f90c00700 1 -- 172.21.15.40:0/333355244 wait complete. 2024-04-26T23:50:27.573 INFO:teuthology.orchestra.run.smithi040.stdout:Deploying unmanaged mgr service... 2024-04-26T23:50:27.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:27 smithi040 ceph-mon[28095]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:50:27.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:27 smithi040 ceph-mon[28095]: Saving service mon spec with placement count:5 2024-04-26T23:50:27.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:27 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:29.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:29 smithi040 ceph-mon[28095]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:50:29.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:29 smithi040 ceph-mon[28095]: Saving service mgr spec with placement count:2 2024-04-26T23:50:29.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:29 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:29.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:29 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:29.835 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.314+0000 7f8bef000700 1 Processor -- start 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.314+0000 7f8bef000700 1 -- start start 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.315+0000 7f8bef000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 0x7f8be81120e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.315+0000 7f8bef000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8be8083500 con 0x7f8be810fcf0 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.315+0000 7f8be7e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 0x7f8be81120e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.315+0000 7f8be7e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 0x7f8be81120e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49398/0 (socket says 172.21.15.40:49398) 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.315+0000 7f8be7e00700 1 -- 172.21.15.40:0/2279875808 learned_addr learned my addr 172.21.15.40:0/2279875808 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.316+0000 7f8be7e00700 1 -- 172.21.15.40:0/2279875808 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be8112620 con 0x7f8be810fcf0 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.316+0000 7f8be7e00700 1 --2- 172.21.15.40:0/2279875808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 0x7f8be81120e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8bd80098a0 tx=0x7f8bd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af09fe8eb64ded3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.316+0000 7f8be6a00700 1 -- 172.21.15.40:0/2279875808 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bd8004160 con 0x7f8be810fcf0 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.316+0000 7f8be6a00700 1 -- 172.21.15.40:0/2279875808 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8bd8004300 con 0x7f8be810fcf0 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.316+0000 7f8be6a00700 1 -- 172.21.15.40:0/2279875808 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bd8011af0 con 0x7f8be810fcf0 2024-04-26T23:50:29.836 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.317+0000 7f8bef000700 1 -- 172.21.15.40:0/2279875808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 msgr2=0x7f8be81120e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.317+0000 7f8bef000700 1 --2- 172.21.15.40:0/2279875808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 0x7f8be81120e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8bd80098a0 tx=0x7f8bd8009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.317+0000 7f8bef000700 1 -- 172.21.15.40:0/2279875808 shutdown_connections 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.317+0000 7f8bef000700 1 --2- 172.21.15.40:0/2279875808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 0x7f8be81120e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.317+0000 7f8bef000700 1 -- 172.21.15.40:0/2279875808 >> 172.21.15.40:0/2279875808 conn(0x7f8be807e2a0 msgr2=0x7f8be80806c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.317+0000 7f8bef000700 1 -- 172.21.15.40:0/2279875808 shutdown_connections 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.317+0000 7f8bef000700 1 -- 172.21.15.40:0/2279875808 wait complete. 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.318+0000 7f8bef000700 1 Processor -- start 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.318+0000 7f8bef000700 1 -- start start 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.319+0000 7f8bef000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 0x7f8be81a3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.319+0000 7f8bef000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8be8113640 con 0x7f8be810fcf0 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.319+0000 7f8be7e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 0x7f8be81a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.319+0000 7f8be7e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 0x7f8be81a3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49402/0 (socket says 172.21.15.40:49402) 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.319+0000 7f8be7e00700 1 -- 172.21.15.40:0/2436152450 learned_addr learned my addr 172.21.15.40:0/2436152450 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:29.837 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.319+0000 7f8be7e00700 1 -- 172.21.15.40:0/2436152450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be81a36c0 con 0x7f8be810fcf0 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.320+0000 7f8be7e00700 1 --2- 172.21.15.40:0/2436152450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 0x7f8be81a3180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8003f40 tx=0x7f8bd8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.320+0000 7f8be4c00700 1 -- 172.21.15.40:0/2436152450 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bd80038b0 con 0x7f8be810fcf0 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.320+0000 7f8be4c00700 1 -- 172.21.15.40:0/2436152450 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8bd8003a50 con 0x7f8be810fcf0 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.320+0000 7f8bef000700 1 -- 172.21.15.40:0/2436152450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8be81a3950 con 0x7f8be810fcf0 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.320+0000 7f8be4c00700 1 -- 172.21.15.40:0/2436152450 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bd801a5c0 con 0x7f8be810fcf0 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.320+0000 7f8bef000700 1 -- 172.21.15.40:0/2436152450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8be81a6640 con 0x7f8be810fcf0 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.321+0000 7f8be4c00700 1 -- 172.21.15.40:0/2436152450 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f8bd8024430 con 0x7f8be810fcf0 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.321+0000 7f8bef000700 1 -- 172.21.15.40:0/2436152450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8be8064a80 con 0x7f8be810fcf0 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.322+0000 7f8be4c00700 1 --2- 172.21.15.40:0/2436152450 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8bcc0395b0 0x7f8bcc03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.322+0000 7f8be4c00700 1 -- 172.21.15.40:0/2436152450 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8bd804db00 con 0x7f8be810fcf0 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.322+0000 7f8be7400700 1 --2- 172.21.15.40:0/2436152450 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8bcc0395b0 0x7f8bcc03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.323+0000 7f8be7400700 1 --2- 172.21.15.40:0/2436152450 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8bcc0395b0 0x7f8bcc03ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4006fd0 tx=0x7f8bd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:29.838 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.328+0000 7f8be4c00700 1 -- 172.21.15.40:0/2436152450 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8bd802cc30 con 0x7f8be810fcf0 2024-04-26T23:50:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.580+0000 7f8bef000700 1 -- 172.21.15.40:0/2436152450 --> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f8be807f420 con 0x7f8bcc0395b0 2024-04-26T23:50:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.587+0000 7f8be4c00700 1 -- 172.21.15.40:0/2436152450 <== mgr.14120 v2:172.21.15.40:6800/1568598447 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8be807f420 con 0x7f8bcc0395b0 2024-04-26T23:50:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.590+0000 7f8bef000700 1 -- 172.21.15.40:0/2436152450 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8bcc0395b0 msgr2=0x7f8bcc03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.590+0000 7f8bef000700 1 --2- 172.21.15.40:0/2436152450 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8bcc0395b0 0x7f8bcc03ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4006fd0 tx=0x7f8bd4006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.590+0000 7f8bef000700 1 -- 172.21.15.40:0/2436152450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 msgr2=0x7f8be81a3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.590+0000 7f8bef000700 1 --2- 172.21.15.40:0/2436152450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 0x7f8be81a3180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8003f40 tx=0x7f8bd8011890 comp rx=0 tx=0).stop 2024-04-26T23:50:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.590+0000 7f8bef000700 1 -- 172.21.15.40:0/2436152450 shutdown_connections 2024-04-26T23:50:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.590+0000 7f8bef000700 1 --2- 172.21.15.40:0/2436152450 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8bcc0395b0 0x7f8bcc03ba70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.590+0000 7f8bef000700 1 --2- 172.21.15.40:0/2436152450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8be810fcf0 0x7f8be81a3180 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.590+0000 7f8bef000700 1 -- 172.21.15.40:0/2436152450 >> 172.21.15.40:0/2436152450 conn(0x7f8be807e2a0 msgr2=0x7f8be807ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.591+0000 7f8bef000700 1 -- 172.21.15.40:0/2436152450 shutdown_connections 2024-04-26T23:50:29.839 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:28.591+0000 7f8bef000700 1 -- 172.21.15.40:0/2436152450 wait complete. 2024-04-26T23:50:31.387 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:31 smithi040 ceph-mon[28095]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:50:31.387 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:31 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:31.388 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:31 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/913414635' entity='client.admin' 2024-04-26T23:50:31.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.416+0000 7f2488800700 1 Processor -- start 2024-04-26T23:50:31.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.416+0000 7f2488800700 1 -- start start 2024-04-26T23:50:31.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.416+0000 7f2488800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 0x7f248007d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:31.396 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.416+0000 7f2488800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f248007d8d0 con 0x7f248007eea0 2024-04-26T23:50:31.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.417+0000 7f2485e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 0x7f248007d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:31.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.417+0000 7f2485e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 0x7f248007d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44260/0 (socket says 172.21.15.40:44260) 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.417+0000 7f2485e00700 1 -- 172.21.15.40:0/618238409 learned_addr learned my addr 172.21.15.40:0/618238409 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.417+0000 7f2485e00700 1 -- 172.21.15.40:0/618238409 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248007da50 con 0x7f248007eea0 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.418+0000 7f2485e00700 1 --2- 172.21.15.40:0/618238409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 0x7f248007d300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f24740098a0 tx=0x7f2474009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f4e9dfea7a8c2f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.418+0000 7f2484a00700 1 -- 172.21.15.40:0/618238409 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2474004160 con 0x7f248007eea0 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.418+0000 7f2484a00700 1 -- 172.21.15.40:0/618238409 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2474004300 con 0x7f248007eea0 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.418+0000 7f2484a00700 1 -- 172.21.15.40:0/618238409 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2474011a20 con 0x7f248007eea0 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.419+0000 7f2488800700 1 -- 172.21.15.40:0/618238409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 msgr2=0x7f248007d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.419+0000 7f2488800700 1 --2- 172.21.15.40:0/618238409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 0x7f248007d300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f24740098a0 tx=0x7f2474009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.419+0000 7f2488800700 1 -- 172.21.15.40:0/618238409 shutdown_connections 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.419+0000 7f2488800700 1 --2- 172.21.15.40:0/618238409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 0x7f248007d300 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.419+0000 7f2488800700 1 -- 172.21.15.40:0/618238409 >> 172.21.15.40:0/618238409 conn(0x7f24801042a0 msgr2=0x7f24801066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.419+0000 7f2488800700 1 -- 172.21.15.40:0/618238409 shutdown_connections 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.419+0000 7f2488800700 1 -- 172.21.15.40:0/618238409 wait complete. 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.420+0000 7f2488800700 1 Processor -- start 2024-04-26T23:50:31.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.421+0000 7f2488800700 1 -- start start 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.421+0000 7f2488800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 0x7f248019ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.421+0000 7f2488800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f248010f300 con 0x7f248007eea0 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.421+0000 7f2485e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 0x7f248019ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.421+0000 7f2485e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 0x7f248019ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44262/0 (socket says 172.21.15.40:44262) 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.421+0000 7f2485e00700 1 -- 172.21.15.40:0/913414635 learned_addr learned my addr 172.21.15.40:0/913414635 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.422+0000 7f2485e00700 1 -- 172.21.15.40:0/913414635 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248019f280 con 0x7f248007eea0 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.422+0000 7f2485e00700 1 --2- 172.21.15.40:0/913414635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 0x7f248019ed40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2474000c00 tx=0x7f2474011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.422+0000 7f247aa00700 1 -- 172.21.15.40:0/913414635 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2474003aa0 con 0x7f248007eea0 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.422+0000 7f247aa00700 1 -- 172.21.15.40:0/913414635 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f247401b8f0 con 0x7f248007eea0 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.422+0000 7f2488800700 1 -- 172.21.15.40:0/913414635 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f248019f510 con 0x7f248007eea0 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.422+0000 7f247aa00700 1 -- 172.21.15.40:0/913414635 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2474023870 con 0x7f248007eea0 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.423+0000 7f2488800700 1 -- 172.21.15.40:0/913414635 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24801a2200 con 0x7f248007eea0 2024-04-26T23:50:31.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.424+0000 7f2488800700 1 -- 172.21.15.40:0/913414635 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2480064a80 con 0x7f248007eea0 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.424+0000 7f247aa00700 1 -- 172.21.15.40:0/913414635 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f247401b440 con 0x7f248007eea0 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.425+0000 7f247aa00700 1 --2- 172.21.15.40:0/913414635 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f2468039600 0x7f246803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.425+0000 7f247aa00700 1 -- 172.21.15.40:0/913414635 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f247404dec0 con 0x7f248007eea0 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.425+0000 7f2485400700 1 --2- 172.21.15.40:0/913414635 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f2468039600 0x7f246803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.426+0000 7f2485400700 1 --2- 172.21.15.40:0/913414635 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f2468039600 0x7f246803bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2470006fd0 tx=0x7f2470006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.431+0000 7f247aa00700 1 -- 172.21.15.40:0/913414635 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f247401f030 con 0x7f248007eea0 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.664+0000 7f2488800700 1 -- 172.21.15.40:0/913414635 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f24801a27f0 con 0x7f248007eea0 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.672+0000 7f247aa00700 1 -- 172.21.15.40:0/913414635 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f247400e260 con 0x7f248007eea0 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.675+0000 7f2488800700 1 -- 172.21.15.40:0/913414635 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f2468039600 msgr2=0x7f246803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.675+0000 7f2488800700 1 --2- 172.21.15.40:0/913414635 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f2468039600 0x7f246803bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2470006fd0 tx=0x7f2470006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.675+0000 7f2488800700 1 -- 172.21.15.40:0/913414635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 msgr2=0x7f248019ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.675+0000 7f2488800700 1 --2- 172.21.15.40:0/913414635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 0x7f248019ed40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2474000c00 tx=0x7f2474011aa0 comp rx=0 tx=0).stop 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.675+0000 7f2488800700 1 -- 172.21.15.40:0/913414635 shutdown_connections 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.675+0000 7f2488800700 1 --2- 172.21.15.40:0/913414635 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f2468039600 0x7f246803bac0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:31.405 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.675+0000 7f2488800700 1 --2- 172.21.15.40:0/913414635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f248007eea0 0x7f248019ed40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:31.406 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.675+0000 7f2488800700 1 -- 172.21.15.40:0/913414635 >> 172.21.15.40:0/913414635 conn(0x7f24801042a0 msgr2=0x7f2480104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:31.406 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.676+0000 7f2488800700 1 -- 172.21.15.40:0/913414635 shutdown_connections 2024-04-26T23:50:31.406 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:30.676+0000 7f2488800700 1 -- 172.21.15.40:0/913414635 wait complete. 2024-04-26T23:50:33.455 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:33 smithi040 ceph-mon[28095]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:50:33.456 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:33 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/605067225' entity='client.admin' 2024-04-26T23:50:33.456 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:33 smithi040 ceph-mon[28095]: from='mgr.14120 172.21.15.40:0/78522194' entity='mgr.x' 2024-04-26T23:50:33.469 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.117+0000 7f250b600700 1 Processor -- start 2024-04-26T23:50:33.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.117+0000 7f250b600700 1 -- start start 2024-04-26T23:50:33.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.118+0000 7f250b600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 0x7f2504109100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:33.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.118+0000 7f250b600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25041096d0 con 0x7f2504108ce0 2024-04-26T23:50:33.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.118+0000 7f2508c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 0x7f2504109100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:33.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.118+0000 7f2508c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 0x7f2504109100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44266/0 (socket says 172.21.15.40:44266) 2024-04-26T23:50:33.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.118+0000 7f2508c00700 1 -- 172.21.15.40:0/2603220316 learned_addr learned my addr 172.21.15.40:0/2603220316 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:33.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.119+0000 7f2508c00700 1 -- 172.21.15.40:0/2603220316 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2504109ee0 con 0x7f2504108ce0 2024-04-26T23:50:33.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.119+0000 7f2508c00700 1 --2- 172.21.15.40:0/2603220316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 0x7f2504109100 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f24f80098a0 tx=0x7f24f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa29ac37a337b2c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:33.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.119+0000 7f2503400700 1 -- 172.21.15.40:0/2603220316 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24f8004160 con 0x7f2504108ce0 2024-04-26T23:50:33.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.119+0000 7f2503400700 1 -- 172.21.15.40:0/2603220316 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24f8004300 con 0x7f2504108ce0 2024-04-26T23:50:33.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.119+0000 7f2503400700 1 -- 172.21.15.40:0/2603220316 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24f8011a20 con 0x7f2504108ce0 2024-04-26T23:50:33.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.120+0000 7f250b600700 1 -- 172.21.15.40:0/2603220316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 msgr2=0x7f2504109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.120+0000 7f250b600700 1 --2- 172.21.15.40:0/2603220316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 0x7f2504109100 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f24f80098a0 tx=0x7f24f8009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.120+0000 7f250b600700 1 -- 172.21.15.40:0/2603220316 shutdown_connections 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.120+0000 7f250b600700 1 --2- 172.21.15.40:0/2603220316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 0x7f2504109100 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.120+0000 7f250b600700 1 -- 172.21.15.40:0/2603220316 >> 172.21.15.40:0/2603220316 conn(0x7f25041042a0 msgr2=0x7f25041066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.120+0000 7f250b600700 1 -- 172.21.15.40:0/2603220316 shutdown_connections 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.120+0000 7f250b600700 1 -- 172.21.15.40:0/2603220316 wait complete. 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.121+0000 7f250b600700 1 Processor -- start 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.122+0000 7f250b600700 1 -- start start 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.122+0000 7f250b600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 0x7f250419eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.122+0000 7f250b600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f250410d290 con 0x7f2504108ce0 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.122+0000 7f2508c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 0x7f250419eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.122+0000 7f2508c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 0x7f250419eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44274/0 (socket says 172.21.15.40:44274) 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.122+0000 7f2508c00700 1 -- 172.21.15.40:0/605067225 learned_addr learned my addr 172.21.15.40:0/605067225 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:33.471 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.123+0000 7f2508c00700 1 -- 172.21.15.40:0/605067225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f250419f2e0 con 0x7f2504108ce0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.123+0000 7f2508c00700 1 --2- 172.21.15.40:0/605067225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 0x7f250419eda0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009520 tx=0x7f24f8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.123+0000 7f2501600700 1 -- 172.21.15.40:0/605067225 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24f80039e0 con 0x7f2504108ce0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.123+0000 7f2501600700 1 -- 172.21.15.40:0/605067225 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24f801b8f0 con 0x7f2504108ce0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.123+0000 7f2501600700 1 -- 172.21.15.40:0/605067225 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24f8019770 con 0x7f2504108ce0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.124+0000 7f250b600700 1 -- 172.21.15.40:0/605067225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f250419f570 con 0x7f2504108ce0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.124+0000 7f250b600700 1 -- 172.21.15.40:0/605067225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25041a2260 con 0x7f2504108ce0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.125+0000 7f250b600700 1 -- 172.21.15.40:0/605067225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2504064a80 con 0x7f2504108ce0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.125+0000 7f2501600700 1 -- 172.21.15.40:0/605067225 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f24f801b440 con 0x7f2504108ce0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.126+0000 7f2501600700 1 --2- 172.21.15.40:0/605067225 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f24ec0395b0 0x7f24ec03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.126+0000 7f2501600700 1 -- 172.21.15.40:0/605067225 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f24f804dee0 con 0x7f2504108ce0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.126+0000 7f2503e00700 1 --2- 172.21.15.40:0/605067225 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f24ec0395b0 0x7f24ec03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.127+0000 7f2503e00700 1 --2- 172.21.15.40:0/605067225 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f24ec0395b0 0x7f24ec03ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f24f0006fd0 tx=0x7f24f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.131+0000 7f2501600700 1 -- 172.21.15.40:0/605067225 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f24f801f030 con 0x7f2504108ce0 2024-04-26T23:50:33.472 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.372+0000 7f250b600700 1 -- 172.21.15.40:0/605067225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f25041a28c0 con 0x7f2504108ce0 2024-04-26T23:50:33.473 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.381+0000 7f2501600700 1 -- 172.21.15.40:0/605067225 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f24f8010380 con 0x7f2504108ce0 2024-04-26T23:50:33.473 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.384+0000 7f250b600700 1 -- 172.21.15.40:0/605067225 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f24ec0395b0 msgr2=0x7f24ec03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:33.473 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.384+0000 7f250b600700 1 --2- 172.21.15.40:0/605067225 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f24ec0395b0 0x7f24ec03ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f24f0006fd0 tx=0x7f24f0006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:33.473 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.384+0000 7f250b600700 1 -- 172.21.15.40:0/605067225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 msgr2=0x7f250419eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:33.473 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.384+0000 7f250b600700 1 --2- 172.21.15.40:0/605067225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 0x7f250419eda0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009520 tx=0x7f24f8011aa0 comp rx=0 tx=0).stop 2024-04-26T23:50:33.473 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.385+0000 7f250b600700 1 -- 172.21.15.40:0/605067225 shutdown_connections 2024-04-26T23:50:33.473 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.385+0000 7f250b600700 1 --2- 172.21.15.40:0/605067225 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f24ec0395b0 0x7f24ec03ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:33.473 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.385+0000 7f250b600700 1 --2- 172.21.15.40:0/605067225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2504108ce0 0x7f250419eda0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:33.473 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.385+0000 7f250b600700 1 -- 172.21.15.40:0/605067225 >> 172.21.15.40:0/605067225 conn(0x7f25041042a0 msgr2=0x7f2504104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:33.473 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.386+0000 7f250b600700 1 -- 172.21.15.40:0/605067225 shutdown_connections 2024-04-26T23:50:33.473 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:32.386+0000 7f250b600700 1 -- 172.21.15.40:0/605067225 wait complete. 2024-04-26T23:50:33.473 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling the dashboard module... 2024-04-26T23:50:35.625 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:35 smithi040 ceph-mon[28095]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:50:35.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:35 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1933691509' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T23:50:35.626 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:35 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: ignoring --setuser ceph since I am not root 2024-04-26T23:50:35.626 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:35 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: ignoring --setgroup ceph since I am not root 2024-04-26T23:50:35.626 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:35 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:35.620+0000 7f35676ac200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T23:50:35.883 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:35 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:35.878+0000 7f35676ac200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:50:36.129 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.052+0000 7fc832600700 1 Processor -- start 2024-04-26T23:50:36.130 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.053+0000 7fc832600700 1 -- start start 2024-04-26T23:50:36.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.053+0000 7fc832600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 0x7fc82c10bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:36.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.053+0000 7fc832600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c07ced0 con 0x7fc82c109740 2024-04-26T23:50:36.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.053+0000 7fc82b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 0x7fc82c10bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:36.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.054+0000 7fc82b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 0x7fc82c10bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44284/0 (socket says 172.21.15.40:44284) 2024-04-26T23:50:36.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.054+0000 7fc82b400700 1 -- 172.21.15.40:0/2787588278 learned_addr learned my addr 172.21.15.40:0/2787588278 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:36.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.054+0000 7fc82b400700 1 -- 172.21.15.40:0/2787588278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82c10c070 con 0x7fc82c109740 2024-04-26T23:50:36.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.054+0000 7fc82b400700 1 --2- 172.21.15.40:0/2787588278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 0x7fc82c10bb30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0098a0 tx=0x7fc81c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c69186afde02eb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:36.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.055+0000 7fc82a000700 1 -- 172.21.15.40:0/2787588278 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc81c004160 con 0x7fc82c109740 2024-04-26T23:50:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.055+0000 7fc82a000700 1 -- 172.21.15.40:0/2787588278 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc81c004300 con 0x7fc82c109740 2024-04-26T23:50:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.055+0000 7fc82a000700 1 -- 172.21.15.40:0/2787588278 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc81c011a20 con 0x7fc82c109740 2024-04-26T23:50:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.056+0000 7fc832600700 1 -- 172.21.15.40:0/2787588278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 msgr2=0x7fc82c10bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.056+0000 7fc832600700 1 --2- 172.21.15.40:0/2787588278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 0x7fc82c10bb30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0098a0 tx=0x7fc81c009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.056+0000 7fc832600700 1 -- 172.21.15.40:0/2787588278 shutdown_connections 2024-04-26T23:50:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.056+0000 7fc832600700 1 --2- 172.21.15.40:0/2787588278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 0x7fc82c10bb30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.056+0000 7fc832600700 1 -- 172.21.15.40:0/2787588278 >> 172.21.15.40:0/2787588278 conn(0x7fc82c1031d0 msgr2=0x7fc82c1055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.056+0000 7fc832600700 1 -- 172.21.15.40:0/2787588278 shutdown_connections 2024-04-26T23:50:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.056+0000 7fc832600700 1 -- 172.21.15.40:0/2787588278 wait complete. 2024-04-26T23:50:36.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.057+0000 7fc832600700 1 Processor -- start 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.057+0000 7fc832600700 1 -- start start 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.057+0000 7fc832600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 0x7fc82c19cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.057+0000 7fc832600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c10d080 con 0x7fc82c109740 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.058+0000 7fc82b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 0x7fc82c19cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.058+0000 7fc82b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 0x7fc82c19cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44298/0 (socket says 172.21.15.40:44298) 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.058+0000 7fc82b400700 1 -- 172.21.15.40:0/1933691509 learned_addr learned my addr 172.21.15.40:0/1933691509 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.058+0000 7fc82b400700 1 -- 172.21.15.40:0/1933691509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82c19d120 con 0x7fc82c109740 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.059+0000 7fc82b400700 1 --2- 172.21.15.40:0/1933691509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 0x7fc82c19cbe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc81c004660 tx=0x7fc81c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.059+0000 7fc823e00700 1 -- 172.21.15.40:0/1933691509 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc81c003aa0 con 0x7fc82c109740 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.059+0000 7fc823e00700 1 -- 172.21.15.40:0/1933691509 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc81c01b8f0 con 0x7fc82c109740 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.059+0000 7fc832600700 1 -- 172.21.15.40:0/1933691509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc82c19d3b0 con 0x7fc82c109740 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.059+0000 7fc823e00700 1 -- 172.21.15.40:0/1933691509 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc81c023900 con 0x7fc82c109740 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.059+0000 7fc832600700 1 -- 172.21.15.40:0/1933691509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc82c1a00a0 con 0x7fc82c109740 2024-04-26T23:50:36.137 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.060+0000 7fc832600700 1 -- 172.21.15.40:0/1933691509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc82c064a80 con 0x7fc82c109740 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.061+0000 7fc823e00700 1 -- 172.21.15.40:0/1933691509 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc81c01baa0 con 0x7fc82c109740 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.061+0000 7fc823e00700 1 --2- 172.21.15.40:0/1933691509 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7fc810039600 0x7fc81003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.061+0000 7fc823e00700 1 -- 172.21.15.40:0/1933691509 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc81c04dec0 con 0x7fc82c109740 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.062+0000 7fc82aa00700 1 --2- 172.21.15.40:0/1933691509 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7fc810039600 0x7fc81003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.062+0000 7fc82aa00700 1 --2- 172.21.15.40:0/1933691509 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7fc810039600 0x7fc81003bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc818006fd0 tx=0x7fc818006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.071+0000 7fc823e00700 1 -- 172.21.15.40:0/1933691509 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc81c01f030 con 0x7fc82c109740 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:34.371+0000 7fc832600700 1 -- 172.21.15.40:0/1933691509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fc82c1a0690 con 0x7fc82c109740 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:35.304+0000 7fc823e00700 1 -- 172.21.15.40:0/1933691509 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fc81c01a3d0 con 0x7fc82c109740 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:35.308+0000 7fc823e00700 1 -- 172.21.15.40:0/1933691509 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fc81c052070 con 0x7fc82c109740 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:35.308+0000 7fc832600700 1 -- 172.21.15.40:0/1933691509 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7fc810039600 msgr2=0x7fc81003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:35.308+0000 7fc832600700 1 --2- 172.21.15.40:0/1933691509 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7fc810039600 0x7fc81003bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc818006fd0 tx=0x7fc818006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:35.308+0000 7fc832600700 1 -- 172.21.15.40:0/1933691509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 msgr2=0x7fc82c19cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:36.138 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:35.308+0000 7fc832600700 1 --2- 172.21.15.40:0/1933691509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 0x7fc82c19cbe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc81c004660 tx=0x7fc81c011aa0 comp rx=0 tx=0).stop 2024-04-26T23:50:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:35.309+0000 7fc832600700 1 -- 172.21.15.40:0/1933691509 shutdown_connections 2024-04-26T23:50:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:35.309+0000 7fc832600700 1 --2- 172.21.15.40:0/1933691509 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7fc810039600 0x7fc81003bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:35.309+0000 7fc832600700 1 --2- 172.21.15.40:0/1933691509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc82c109740 0x7fc82c19cbe0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:35.309+0000 7fc832600700 1 -- 172.21.15.40:0/1933691509 >> 172.21.15.40:0/1933691509 conn(0x7fc82c1031d0 msgr2=0x7fc82c103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:35.310+0000 7fc832600700 1 -- 172.21.15.40:0/1933691509 shutdown_connections 2024-04-26T23:50:36.139 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:35.310+0000 7fc832600700 1 -- 172.21.15.40:0/1933691509 wait complete. 2024-04-26T23:50:36.219 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:36 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:36.020+0000 7f35676ac200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T23:50:36.555 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:36 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:36.441+0000 7f35676ac200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T23:50:36.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:36 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1933691509' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T23:50:36.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:36 smithi040 ceph-mon[28095]: mgrmap e9: x(active, since 25s) 2024-04-26T23:50:36.848 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:36 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:36.561+0000 7f35676ac200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T23:50:37.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:37 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1347546484' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.843+0000 7f8cdae00700 1 Processor -- start 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.843+0000 7f8cdae00700 1 -- start start 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.844+0000 7f8cdae00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 0x7f8cd4109100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.844+0000 7f8cdae00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd41096d0 con 0x7f8cd4108ce0 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.844+0000 7f8cd3e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 0x7f8cd4109100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.844+0000 7f8cd3e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 0x7f8cd4109100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44316/0 (socket says 172.21.15.40:44316) 2024-04-26T23:50:38.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.844+0000 7f8cd3e00700 1 -- 172.21.15.40:0/1360713076 learned_addr learned my addr 172.21.15.40:0/1360713076 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.845+0000 7f8cd3e00700 1 -- 172.21.15.40:0/1360713076 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd4109ee0 con 0x7f8cd4108ce0 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.845+0000 7f8cd3e00700 1 --2- 172.21.15.40:0/1360713076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 0x7f8cd4109100 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40098a0 tx=0x7f8cc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff47908069a55ea2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.845+0000 7f8cd2a00700 1 -- 172.21.15.40:0/1360713076 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8cc4004160 con 0x7f8cd4108ce0 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.846+0000 7f8cd2a00700 1 -- 172.21.15.40:0/1360713076 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cc4004300 con 0x7f8cd4108ce0 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.846+0000 7f8cd2a00700 1 -- 172.21.15.40:0/1360713076 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8cc4011a20 con 0x7f8cd4108ce0 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.846+0000 7f8cdae00700 1 -- 172.21.15.40:0/1360713076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 msgr2=0x7f8cd4109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.846+0000 7f8cdae00700 1 --2- 172.21.15.40:0/1360713076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 0x7f8cd4109100 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40098a0 tx=0x7f8cc4009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.846+0000 7f8cdae00700 1 -- 172.21.15.40:0/1360713076 shutdown_connections 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.846+0000 7f8cdae00700 1 --2- 172.21.15.40:0/1360713076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 0x7f8cd4109100 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.846+0000 7f8cdae00700 1 -- 172.21.15.40:0/1360713076 >> 172.21.15.40:0/1360713076 conn(0x7f8cd41042a0 msgr2=0x7f8cd41066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.847+0000 7f8cdae00700 1 -- 172.21.15.40:0/1360713076 shutdown_connections 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.847+0000 7f8cdae00700 1 -- 172.21.15.40:0/1360713076 wait complete. 2024-04-26T23:50:38.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.848+0000 7f8cdae00700 1 Processor -- start 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.848+0000 7f8cdae00700 1 -- start start 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.848+0000 7f8cdae00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 0x7f8cd419cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.848+0000 7f8cdae00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd410d290 con 0x7f8cd4108ce0 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.848+0000 7f8cd3e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 0x7f8cd419cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.848+0000 7f8cd3e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 0x7f8cd419cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44326/0 (socket says 172.21.15.40:44326) 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.848+0000 7f8cd3e00700 1 -- 172.21.15.40:0/1347546484 learned_addr learned my addr 172.21.15.40:0/1347546484 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.849+0000 7f8cd3e00700 1 -- 172.21.15.40:0/1347546484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd419d0c0 con 0x7f8cd4108ce0 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.849+0000 7f8cd3e00700 1 --2- 172.21.15.40:0/1347546484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 0x7f8cd419cb80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009520 tx=0x7f8cc4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.849+0000 7f8cd0c00700 1 -- 172.21.15.40:0/1347546484 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8cc40039e0 con 0x7f8cd4108ce0 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.849+0000 7f8cd0c00700 1 -- 172.21.15.40:0/1347546484 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cc401b8f0 con 0x7f8cd4108ce0 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.849+0000 7f8cd0c00700 1 -- 172.21.15.40:0/1347546484 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8cc4019770 con 0x7f8cd4108ce0 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.849+0000 7f8cdae00700 1 -- 172.21.15.40:0/1347546484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cd419d350 con 0x7f8cd4108ce0 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.850+0000 7f8cdae00700 1 -- 172.21.15.40:0/1347546484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cd41a0040 con 0x7f8cd4108ce0 2024-04-26T23:50:38.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.851+0000 7f8cdae00700 1 -- 172.21.15.40:0/1347546484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cd4064a80 con 0x7f8cd4108ce0 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.851+0000 7f8cd0c00700 1 -- 172.21.15.40:0/1347546484 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f8cc401b440 con 0x7f8cd4108ce0 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.851+0000 7f8cd0c00700 1 --2- 172.21.15.40:0/1347546484 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8cb8039650 0x7f8cb803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.851+0000 7f8cd0c00700 1 -- 172.21.15.40:0/1347546484 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8cc404df70 con 0x7f8cd4108ce0 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.852+0000 7f8cd3400700 1 -- 172.21.15.40:0/1347546484 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8cb8039650 msgr2=0x7f8cb803bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1568598447 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.852+0000 7f8cd3400700 1 --2- 172.21.15.40:0/1347546484 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8cb8039650 0x7f8cb803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:36.857+0000 7f8cd0c00700 1 -- 172.21.15.40:0/1347546484 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8cc401f030 con 0x7f8cd4108ce0 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.053+0000 7f8cd3400700 1 -- 172.21.15.40:0/1347546484 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8cb8039650 msgr2=0x7f8cb803bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1568598447 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.053+0000 7f8cd3400700 1 --2- 172.21.15.40:0/1347546484 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8cb8039650 0x7f8cb803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.182+0000 7f8cdae00700 1 -- 172.21.15.40:0/1347546484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f8cd41a0630 con 0x7f8cd4108ce0 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.183+0000 7f8cd0c00700 1 -- 172.21.15.40:0/1347546484 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f8cc4010380 con 0x7f8cd4108ce0 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.186+0000 7f8cdae00700 1 -- 172.21.15.40:0/1347546484 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8cb8039650 msgr2=0x7f8cb803bb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.186+0000 7f8cdae00700 1 --2- 172.21.15.40:0/1347546484 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8cb8039650 0x7f8cb803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.186+0000 7f8cdae00700 1 -- 172.21.15.40:0/1347546484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 msgr2=0x7f8cd419cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:38.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.186+0000 7f8cdae00700 1 --2- 172.21.15.40:0/1347546484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 0x7f8cd419cb80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009520 tx=0x7f8cc4011aa0 comp rx=0 tx=0).stop 2024-04-26T23:50:38.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.186+0000 7f8cdae00700 1 -- 172.21.15.40:0/1347546484 shutdown_connections 2024-04-26T23:50:38.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.186+0000 7f8cdae00700 1 --2- 172.21.15.40:0/1347546484 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f8cb8039650 0x7f8cb803bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:38.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.186+0000 7f8cdae00700 1 --2- 172.21.15.40:0/1347546484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cd4108ce0 0x7f8cd419cb80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:38.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.186+0000 7f8cdae00700 1 -- 172.21.15.40:0/1347546484 >> 172.21.15.40:0/1347546484 conn(0x7f8cd41042a0 msgr2=0x7f8cd4104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:38.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.186+0000 7f8cdae00700 1 -- 172.21.15.40:0/1347546484 shutdown_connections 2024-04-26T23:50:38.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:37.186+0000 7f8cdae00700 1 -- 172.21.15.40:0/1347546484 wait complete. 2024-04-26T23:50:38.135 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for the mgr to restart... 2024-04-26T23:50:38.135 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr epoch 9... 2024-04-26T23:50:38.404 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:38 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:38.162+0000 7f35676ac200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T23:50:38.687 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:38 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:38.399+0000 7f35676ac200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T23:50:38.687 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:38 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:38.626+0000 7f35676ac200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T23:50:39.219 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:39 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:39.021+0000 7f35676ac200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T23:50:39.220 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:39 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:39.147+0000 7f35676ac200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T23:50:39.674 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:39 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:39.408+0000 7f35676ac200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T23:50:39.675 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:39 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:39.669+0000 7f35676ac200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:50:40.969 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:40 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:40.483+0000 7f35676ac200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T23:50:41.466 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:41 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:41.053+0000 7f35676ac200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T23:50:41.466 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:41 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:41.186+0000 7f35676ac200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T23:50:41.467 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:41 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:41.461+0000 7f35676ac200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T23:50:42.469 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:42 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:42.129+0000 7f35676ac200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T23:50:42.470 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:42 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:42.260+0000 7f35676ac200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T23:50:42.790 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:42 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:42.599+0000 7f35676ac200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T23:50:42.791 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:42 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:42.786+0000 7f35676ac200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T23:50:43.089 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:42 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:42.923+0000 7f35676ac200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T23:50:43.467 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:43 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:43.084+0000 7f35676ac200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T23:50:43.719 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:43 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:43.462+0000 7f35676ac200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T23:50:44.547 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:44 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:44.248+0000 7f35676ac200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T23:50:44.969 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:50:44 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:50:44.542+0000 7f35676ac200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T23:50:44.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:44 smithi040 ceph-mon[28095]: Active manager daemon x restarted 2024-04-26T23:50:44.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:44 smithi040 ceph-mon[28095]: Activating manager daemon x 2024-04-26T23:50:44.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:44 smithi040 ceph-mon[28095]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T23:50:44.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:44 smithi040 ceph-mon[28095]: mgrmap e10: x(active, starting, since 0.00943497s) 2024-04-26T23:50:44.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:50:44.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T23:50:44.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T23:50:44.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T23:50:44.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T23:50:44.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:44 smithi040 ceph-mon[28095]: Manager daemon x is now available 2024-04-26T23:50:45.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:50:45.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T23:50:45.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T23:50:45.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:45 smithi040 ceph-mon[28095]: [26/Apr/2024:23:50:45] ENGINE Bus STARTING 2024-04-26T23:50:45.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:45 smithi040 ceph-mon[28095]: [26/Apr/2024:23:50:45] ENGINE Serving on https://172.21.15.40:7150 2024-04-26T23:50:45.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:45 smithi040 ceph-mon[28095]: [26/Apr/2024:23:50:45] ENGINE Serving on http://172.21.15.40:8765 2024-04-26T23:50:45.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:45 smithi040 ceph-mon[28095]: [26/Apr/2024:23:50:45] ENGINE Bus STARTED 2024-04-26T23:50:45.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:45 smithi040 ceph-mon[28095]: mgrmap e11: x(active, since 1.01206s) 2024-04-26T23:50:46.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:50:46.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.745+0000 7f9ba2e00700 1 Processor -- start 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.745+0000 7f9ba2e00700 1 -- start start 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.745+0000 7f9ba2e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 0x7f9b9c07d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.745+0000 7f9ba2e00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c07d930 con 0x7f9b9c07ef00 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.745+0000 7f9b9be00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 0x7f9b9c07d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.746+0000 7f9b9be00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 0x7f9b9c07d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44338/0 (socket says 172.21.15.40:44338) 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.746+0000 7f9b9be00700 1 -- 172.21.15.40:0/1313653692 learned_addr learned my addr 172.21.15.40:0/1313653692 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.746+0000 7f9b9be00700 1 -- 172.21.15.40:0/1313653692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9c07dab0 con 0x7f9b9c07ef00 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.746+0000 7f9b9be00700 1 --2- 172.21.15.40:0/1313653692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 0x7f9b9c07d360 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0098a0 tx=0x7f9b8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc4c50cdfb90c2f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.747+0000 7f9b9aa00700 1 -- 172.21.15.40:0/1313653692 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b8c004160 con 0x7f9b9c07ef00 2024-04-26T23:50:46.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.747+0000 7f9b9aa00700 1 -- 172.21.15.40:0/1313653692 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b8c004300 con 0x7f9b9c07ef00 2024-04-26T23:50:46.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.747+0000 7f9b9aa00700 1 -- 172.21.15.40:0/1313653692 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b8c011a20 con 0x7f9b9c07ef00 2024-04-26T23:50:46.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.747+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1313653692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 msgr2=0x7f9b9c07d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:46.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.747+0000 7f9ba2e00700 1 --2- 172.21.15.40:0/1313653692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 0x7f9b9c07d360 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0098a0 tx=0x7f9b8c009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:46.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.747+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1313653692 shutdown_connections 2024-04-26T23:50:46.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.747+0000 7f9ba2e00700 1 --2- 172.21.15.40:0/1313653692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 0x7f9b9c07d360 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:46.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.747+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1313653692 >> 172.21.15.40:0/1313653692 conn(0x7f9b9c1042a0 msgr2=0x7f9b9c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:46.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.748+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1313653692 shutdown_connections 2024-04-26T23:50:46.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.748+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1313653692 wait complete. 2024-04-26T23:50:46.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.748+0000 7f9ba2e00700 1 Processor -- start 2024-04-26T23:50:46.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.749+0000 7f9ba2e00700 1 -- start start 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.749+0000 7f9ba2e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 0x7f9b9c19edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.749+0000 7f9ba2e00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c10f2f0 con 0x7f9b9c07ef00 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.749+0000 7f9b9be00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 0x7f9b9c19edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.749+0000 7f9b9be00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 0x7f9b9c19edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44344/0 (socket says 172.21.15.40:44344) 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.749+0000 7f9b9be00700 1 -- 172.21.15.40:0/1851721700 learned_addr learned my addr 172.21.15.40:0/1851721700 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.750+0000 7f9b9be00700 1 -- 172.21.15.40:0/1851721700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9c19f300 con 0x7f9b9c07ef00 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.750+0000 7f9b9be00700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 0x7f9b9c19edc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009520 tx=0x7f9b8c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.750+0000 7f9b98c00700 1 -- 172.21.15.40:0/1851721700 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b8c003af0 con 0x7f9b9c07ef00 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.750+0000 7f9b98c00700 1 -- 172.21.15.40:0/1851721700 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b8c01a8e0 con 0x7f9b9c07ef00 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.750+0000 7f9b98c00700 1 -- 172.21.15.40:0/1851721700 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b8c0247a0 con 0x7f9b9c07ef00 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.750+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1851721700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b9c19f590 con 0x7f9b9c07ef00 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.750+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1851721700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b9c1a2280 con 0x7f9b9c07ef00 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.751+0000 7f9b98c00700 1 -- 172.21.15.40:0/1851721700 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f9b8c022020 con 0x7f9b9c07ef00 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.752+0000 7f9b98c00700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 0x7f9b8003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.752+0000 7f9b98c00700 1 -- 172.21.15.40:0/1851721700 --> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9b8003c230 con 0x7f9b80039660 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.752+0000 7f9b98c00700 1 -- 172.21.15.40:0/1851721700 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9b8c04da60 con 0x7f9b9c07ef00 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.752+0000 7f9b9b400700 1 -- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 msgr2=0x7f9b8003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1568598447 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.752+0000 7f9b9b400700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 0x7f9b8003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.953+0000 7f9b9b400700 1 -- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 msgr2=0x7f9b8003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1568598447 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:38.953+0000 7f9b9b400700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 0x7f9b8003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:39.354+0000 7f9b9b400700 1 -- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 msgr2=0x7f9b8003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1568598447 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:39.354+0000 7f9b9b400700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 0x7f9b8003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:40.155+0000 7f9b9b400700 1 -- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 msgr2=0x7f9b8003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1568598447 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:40.155+0000 7f9b9b400700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 0x7f9b8003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:41.757+0000 7f9b9b400700 1 -- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 msgr2=0x7f9b8003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1568598447 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:41.757+0000 7f9b9b400700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 0x7f9b8003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:44.552+0000 7f9b98c00700 1 -- 172.21.15.40:0/1851721700 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f9b8c024a60 con 0x7f9b9c07ef00 2024-04-26T23:50:46.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:44.552+0000 7f9b98c00700 1 -- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 msgr2=0x7f9b8003bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:44.552+0000 7f9b98c00700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 0x7f9b8003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.556+0000 7f9b98c00700 1 -- 172.21.15.40:0/1851721700 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f9b8c02ca00 con 0x7f9b9c07ef00 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.556+0000 7f9b98c00700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9b8003cff0 0x7f9b8003f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.556+0000 7f9b98c00700 1 -- 172.21.15.40:0/1851721700 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9b8003c230 con 0x7f9b8003cff0 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.556+0000 7f9b9b400700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9b8003cff0 0x7f9b8003f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.558+0000 7f9b9b400700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9b8003cff0 0x7f9b8003f3e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9b88003a10 tx=0x7f9b88009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.559+0000 7f9b98c00700 1 -- 172.21.15.40:0/1851721700 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f9b8003c230 con 0x7f9b8003cff0 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.567+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1851721700 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9b9c064a80 con 0x7f9b8003cff0 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.567+0000 7f9b98c00700 1 -- 172.21.15.40:0/1851721700 <== mgr.14152 v2:172.21.15.40:6800/3777720410 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f9b9c064a80 con 0x7f9b8003cff0 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.569+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9b8003cff0 msgr2=0x7f9b8003f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.569+0000 7f9ba2e00700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9b8003cff0 0x7f9b8003f3e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9b88003a10 tx=0x7f9b88009270 comp rx=0 tx=0).stop 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.569+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 msgr2=0x7f9b9c19edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.569+0000 7f9ba2e00700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 0x7f9b9c19edc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009520 tx=0x7f9b8c011aa0 comp rx=0 tx=0).stop 2024-04-26T23:50:46.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.569+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1851721700 shutdown_connections 2024-04-26T23:50:46.319 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.569+0000 7f9ba2e00700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9b8003cff0 0x7f9b8003f3e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:46.319 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.569+0000 7f9ba2e00700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b9c07ef00 0x7f9b9c19edc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:46.319 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.569+0000 7f9ba2e00700 1 --2- 172.21.15.40:0/1851721700 >> [v2:172.21.15.40:6800/1568598447,v1:172.21.15.40:6801/1568598447] conn(0x7f9b80039660 0x7f9b8003bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:46.319 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.569+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1851721700 >> 172.21.15.40:0/1851721700 conn(0x7f9b9c1042a0 msgr2=0x7f9b9c104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:46.319 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.569+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1851721700 shutdown_connections 2024-04-26T23:50:46.319 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:45.569+0000 7f9ba2e00700 1 -- 172.21.15.40:0/1851721700 wait complete. 2024-04-26T23:50:46.319 INFO:teuthology.orchestra.run.smithi040.stdout:mgr epoch 9 is available 2024-04-26T23:50:46.319 INFO:teuthology.orchestra.run.smithi040.stdout:Generating a dashboard self-signed certificate... 2024-04-26T23:50:46.927 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:46 smithi040 ceph-mon[28095]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T23:50:46.927 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:46 smithi040 ceph-mon[28095]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T23:50:47.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:47 smithi040 ceph-mon[28095]: mgrmap e12: x(active, since 2s) 2024-04-26T23:50:47.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:47 smithi040 ceph-mon[28095]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:50:47.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:47 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:50:47.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:47 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:50:48.350 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T23:50:48.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.964+0000 7fb283400700 1 Processor -- start 2024-04-26T23:50:48.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.965+0000 7fb283400700 1 -- start start 2024-04-26T23:50:48.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.965+0000 7fb283400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 0x7fb27c10b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:48.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.965+0000 7fb283400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c10b7f0 con 0x7fb27c10ae90 2024-04-26T23:50:48.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.965+0000 7fb280a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 0x7fb27c10b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:48.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.965+0000 7fb280a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 0x7fb27c10b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42538/0 (socket says 172.21.15.40:42538) 2024-04-26T23:50:48.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.966+0000 7fb280a00700 1 -- 172.21.15.40:0/555649647 learned_addr learned my addr 172.21.15.40:0/555649647 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:48.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.966+0000 7fb280a00700 1 -- 172.21.15.40:0/555649647 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27c10c080 con 0x7fb27c10ae90 2024-04-26T23:50:48.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.966+0000 7fb280a00700 1 --2- 172.21.15.40:0/555649647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 0x7fb27c10b2b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb2700098a0 tx=0x7fb270009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54de431ec499a19d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:48.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.967+0000 7fb27b400700 1 -- 172.21.15.40:0/555649647 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb270004160 con 0x7fb27c10ae90 2024-04-26T23:50:48.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.967+0000 7fb27b400700 1 -- 172.21.15.40:0/555649647 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb270004300 con 0x7fb27c10ae90 2024-04-26T23:50:48.351 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.967+0000 7fb27b400700 1 -- 172.21.15.40:0/555649647 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb270011a20 con 0x7fb27c10ae90 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.967+0000 7fb283400700 1 -- 172.21.15.40:0/555649647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 msgr2=0x7fb27c10b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.967+0000 7fb283400700 1 --2- 172.21.15.40:0/555649647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 0x7fb27c10b2b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb2700098a0 tx=0x7fb270009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.967+0000 7fb283400700 1 -- 172.21.15.40:0/555649647 shutdown_connections 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.967+0000 7fb283400700 1 --2- 172.21.15.40:0/555649647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 0x7fb27c10b2b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.967+0000 7fb283400700 1 -- 172.21.15.40:0/555649647 >> 172.21.15.40:0/555649647 conn(0x7fb27c07e300 msgr2=0x7fb27c07e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.968+0000 7fb283400700 1 -- 172.21.15.40:0/555649647 shutdown_connections 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.968+0000 7fb283400700 1 -- 172.21.15.40:0/555649647 wait complete. 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.969+0000 7fb283400700 1 Processor -- start 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.969+0000 7fb283400700 1 -- start start 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.969+0000 7fb283400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 0x7fb27c19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.969+0000 7fb283400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c10f340 con 0x7fb27c10ae90 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.970+0000 7fb280a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 0x7fb27c19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:48.352 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.970+0000 7fb280a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 0x7fb27c19ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42542/0 (socket says 172.21.15.40:42542) 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.970+0000 7fb280a00700 1 -- 172.21.15.40:0/1655634458 learned_addr learned my addr 172.21.15.40:0/1655634458 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.970+0000 7fb280a00700 1 -- 172.21.15.40:0/1655634458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27c19f350 con 0x7fb27c10ae90 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.970+0000 7fb280a00700 1 --2- 172.21.15.40:0/1655634458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 0x7fb27c19ee10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb270000c00 tx=0x7fb270011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.971+0000 7fb279600700 1 -- 172.21.15.40:0/1655634458 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb270003aa0 con 0x7fb27c10ae90 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.971+0000 7fb279600700 1 -- 172.21.15.40:0/1655634458 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb27001b8f0 con 0x7fb27c10ae90 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.971+0000 7fb283400700 1 -- 172.21.15.40:0/1655634458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27c19f5e0 con 0x7fb27c10ae90 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.971+0000 7fb279600700 1 -- 172.21.15.40:0/1655634458 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb270023890 con 0x7fb27c10ae90 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.971+0000 7fb283400700 1 -- 172.21.15.40:0/1655634458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb27c1a22d0 con 0x7fb27c10ae90 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.972+0000 7fb283400700 1 -- 172.21.15.40:0/1655634458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb27c064a80 con 0x7fb27c10ae90 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.972+0000 7fb279600700 1 -- 172.21.15.40:0/1655634458 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fb27001b440 con 0x7fb27c10ae90 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.972+0000 7fb279600700 1 --2- 172.21.15.40:0/1655634458 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb264039600 0x7fb26403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.972+0000 7fb279600700 1 -- 172.21.15.40:0/1655634458 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb27004df70 con 0x7fb27c10ae90 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.973+0000 7fb27be00700 1 --2- 172.21.15.40:0/1655634458 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb264039600 0x7fb26403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:48.353 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.974+0000 7fb27be00700 1 --2- 172.21.15.40:0/1655634458 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb264039600 0x7fb26403bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb268006fd0 tx=0x7fb268006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:48.354 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:46.979+0000 7fb279600700 1 -- 172.21.15.40:0/1655634458 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb27001f030 con 0x7fb27c10ae90 2024-04-26T23:50:48.354 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:47.215+0000 7fb283400700 1 -- 172.21.15.40:0/1655634458 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fb27c108b70 con 0x7fb264039600 2024-04-26T23:50:48.354 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:47.341+0000 7fb279600700 1 -- 172.21.15.40:0/1655634458 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fb27c108b70 con 0x7fb264039600 2024-04-26T23:50:48.354 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:47.345+0000 7fb283400700 1 -- 172.21.15.40:0/1655634458 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb264039600 msgr2=0x7fb26403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:48.354 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:47.345+0000 7fb283400700 1 --2- 172.21.15.40:0/1655634458 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb264039600 0x7fb26403bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb268006fd0 tx=0x7fb268006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:48.354 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:47.345+0000 7fb283400700 1 -- 172.21.15.40:0/1655634458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 msgr2=0x7fb27c19ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:48.354 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:47.345+0000 7fb283400700 1 --2- 172.21.15.40:0/1655634458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 0x7fb27c19ee10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb270000c00 tx=0x7fb270011aa0 comp rx=0 tx=0).stop 2024-04-26T23:50:48.354 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:47.345+0000 7fb283400700 1 -- 172.21.15.40:0/1655634458 shutdown_connections 2024-04-26T23:50:48.354 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:47.345+0000 7fb283400700 1 --2- 172.21.15.40:0/1655634458 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb264039600 0x7fb26403bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:48.354 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:47.345+0000 7fb283400700 1 --2- 172.21.15.40:0/1655634458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb27c10ae90 0x7fb27c19ee10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:48.354 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:47.345+0000 7fb283400700 1 -- 172.21.15.40:0/1655634458 >> 172.21.15.40:0/1655634458 conn(0x7fb27c07e300 msgr2=0x7fb27c108460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:48.357 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:47.345+0000 7fb283400700 1 -- 172.21.15.40:0/1655634458 shutdown_connections 2024-04-26T23:50:48.357 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:47.345+0000 7fb283400700 1 -- 172.21.15.40:0/1655634458 wait complete. 2024-04-26T23:50:48.358 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial admin user... 2024-04-26T23:50:49.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:50:50.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:50 smithi040 ceph-mon[28095]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:50:50.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:50 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:50:50.523 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$3nBhxI4IYk6Yh5TXr2w8O.tP6tBMqzARwl174O5K9tht9TSImxYuW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714175449, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T23:50:50.523 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.132+0000 7f5719400700 1 Processor -- start 2024-04-26T23:50:50.523 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.132+0000 7f5719400700 1 -- start start 2024-04-26T23:50:50.523 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.133+0000 7f5719400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 0x7f571410bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:50.523 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.133+0000 7f5719400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f571407ced0 con 0x7f5714109740 2024-04-26T23:50:50.523 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.133+0000 7f5712800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 0x7f571410bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:50.523 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.133+0000 7f5712800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 0x7f571410bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42552/0 (socket says 172.21.15.40:42552) 2024-04-26T23:50:50.523 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.133+0000 7f5712800700 1 -- 172.21.15.40:0/1253204925 learned_addr learned my addr 172.21.15.40:0/1253204925 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.134+0000 7f5712800700 1 -- 172.21.15.40:0/1253204925 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571410c070 con 0x7f5714109740 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.134+0000 7f5712800700 1 --2- 172.21.15.40:0/1253204925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 0x7f571410bb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f57040098a0 tx=0x7f5704009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8032e5bc9a482d08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.134+0000 7f5711400700 1 -- 172.21.15.40:0/1253204925 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5704004160 con 0x7f5714109740 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.134+0000 7f5711400700 1 -- 172.21.15.40:0/1253204925 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5704004300 con 0x7f5714109740 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.134+0000 7f5711400700 1 -- 172.21.15.40:0/1253204925 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5704011a20 con 0x7f5714109740 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.135+0000 7f5719400700 1 -- 172.21.15.40:0/1253204925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 msgr2=0x7f571410bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.135+0000 7f5719400700 1 --2- 172.21.15.40:0/1253204925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 0x7f571410bb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f57040098a0 tx=0x7f5704009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.136+0000 7f5719400700 1 -- 172.21.15.40:0/1253204925 shutdown_connections 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.136+0000 7f5719400700 1 --2- 172.21.15.40:0/1253204925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 0x7f571410bb30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.136+0000 7f5719400700 1 -- 172.21.15.40:0/1253204925 >> 172.21.15.40:0/1253204925 conn(0x7f57141031d0 msgr2=0x7f57141055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.136+0000 7f5719400700 1 -- 172.21.15.40:0/1253204925 shutdown_connections 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.136+0000 7f5719400700 1 -- 172.21.15.40:0/1253204925 wait complete. 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.137+0000 7f5719400700 1 Processor -- start 2024-04-26T23:50:50.524 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.138+0000 7f5719400700 1 -- start start 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.138+0000 7f5719400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 0x7f571419cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.138+0000 7f5719400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f571410d080 con 0x7f5714109740 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.138+0000 7f5712800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 0x7f571419cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.139+0000 7f5712800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 0x7f571419cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42564/0 (socket says 172.21.15.40:42564) 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.139+0000 7f5712800700 1 -- 172.21.15.40:0/3450944599 learned_addr learned my addr 172.21.15.40:0/3450944599 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.139+0000 7f5712800700 1 -- 172.21.15.40:0/3450944599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571419d140 con 0x7f5714109740 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.139+0000 7f5712800700 1 --2- 172.21.15.40:0/3450944599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 0x7f571419cc00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5704004660 tx=0x7f5704011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.140+0000 7f570b400700 1 -- 172.21.15.40:0/3450944599 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5704003aa0 con 0x7f5714109740 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.140+0000 7f5719400700 1 -- 172.21.15.40:0/3450944599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f571419d3d0 con 0x7f5714109740 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.140+0000 7f570b400700 1 -- 172.21.15.40:0/3450944599 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f570401b8f0 con 0x7f5714109740 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.140+0000 7f570b400700 1 -- 172.21.15.40:0/3450944599 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5704023900 con 0x7f5714109740 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.140+0000 7f5719400700 1 -- 172.21.15.40:0/3450944599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57141a00c0 con 0x7f5714109740 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.141+0000 7f5719400700 1 -- 172.21.15.40:0/3450944599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5714064a80 con 0x7f5714109740 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.141+0000 7f570b400700 1 -- 172.21.15.40:0/3450944599 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f570401baa0 con 0x7f5714109740 2024-04-26T23:50:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.142+0000 7f570b400700 1 --2- 172.21.15.40:0/3450944599 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f56f8039660 0x7f56f803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.142+0000 7f570b400700 1 -- 172.21.15.40:0/3450944599 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f570404df30 con 0x7f5714109740 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.142+0000 7f5711e00700 1 --2- 172.21.15.40:0/3450944599 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f56f8039660 0x7f56f803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.143+0000 7f5711e00700 1 --2- 172.21.15.40:0/3450944599 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f56f8039660 0x7f56f803bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5700006fd0 tx=0x7f5700006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.147+0000 7f570b400700 1 -- 172.21.15.40:0/3450944599 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f570401f030 con 0x7f5714109740 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.397+0000 7f5719400700 1 -- 172.21.15.40:0/3450944599 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f57141a0350 con 0x7f56f8039660 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.676+0000 7f570b400700 1 -- 172.21.15.40:0/3450944599 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f57141a0350 con 0x7f56f8039660 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.679+0000 7f5719400700 1 -- 172.21.15.40:0/3450944599 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f56f8039660 msgr2=0x7f56f803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.679+0000 7f5719400700 1 --2- 172.21.15.40:0/3450944599 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f56f8039660 0x7f56f803bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5700006fd0 tx=0x7f5700006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.679+0000 7f5719400700 1 -- 172.21.15.40:0/3450944599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 msgr2=0x7f571419cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.679+0000 7f5719400700 1 --2- 172.21.15.40:0/3450944599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 0x7f571419cc00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5704004660 tx=0x7f5704011aa0 comp rx=0 tx=0).stop 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.679+0000 7f5719400700 1 -- 172.21.15.40:0/3450944599 shutdown_connections 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.679+0000 7f5719400700 1 --2- 172.21.15.40:0/3450944599 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f56f8039660 0x7f56f803bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.680+0000 7f5719400700 1 --2- 172.21.15.40:0/3450944599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5714109740 0x7f571419cc00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:50.527 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.680+0000 7f5719400700 1 -- 172.21.15.40:0/3450944599 >> 172.21.15.40:0/3450944599 conn(0x7f57141031d0 msgr2=0x7f5714103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:50.527 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.680+0000 7f5719400700 1 -- 172.21.15.40:0/3450944599 shutdown_connections 2024-04-26T23:50:50.527 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:49.680+0000 7f5719400700 1 -- 172.21.15.40:0/3450944599 wait complete. 2024-04-26T23:50:50.527 INFO:teuthology.orchestra.run.smithi040.stdout:Fetching dashboard port number... 2024-04-26T23:50:51.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:51 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:50:51.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:51 smithi040 ceph-mon[28095]: mgrmap e13: x(active, since 6s) 2024-04-26T23:50:51.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:51 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2739781798' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T23:50:52.280 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T23:50:52.281 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.203+0000 7f443e800700 1 Processor -- start 2024-04-26T23:50:52.281 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.203+0000 7f443e800700 1 -- start start 2024-04-26T23:50:52.281 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.203+0000 7f443e800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 0x7f443810bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:52.281 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.203+0000 7f443e800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f443807ce10 con 0x7f4438109800 2024-04-26T23:50:52.281 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.204+0000 7f4437400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 0x7f443810bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:52.281 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.204+0000 7f4437400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 0x7f443810bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59120/0 (socket says 172.21.15.40:59120) 2024-04-26T23:50:52.281 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.204+0000 7f4437400700 1 -- 172.21.15.40:0/635754507 learned_addr learned my addr 172.21.15.40:0/635754507 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:52.281 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.205+0000 7f4437400700 1 -- 172.21.15.40:0/635754507 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443810c130 con 0x7f4438109800 2024-04-26T23:50:52.281 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.205+0000 7f4437400700 1 --2- 172.21.15.40:0/635754507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 0x7f443810bbf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f44280098a0 tx=0x7f4428009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f9227040a7036e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:52.285 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.205+0000 7f4436000700 1 -- 172.21.15.40:0/635754507 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4428004160 con 0x7f4438109800 2024-04-26T23:50:52.285 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.205+0000 7f4436000700 1 -- 172.21.15.40:0/635754507 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4428004300 con 0x7f4438109800 2024-04-26T23:50:52.285 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.205+0000 7f4436000700 1 -- 172.21.15.40:0/635754507 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4428011a20 con 0x7f4438109800 2024-04-26T23:50:52.286 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.206+0000 7f443e800700 1 -- 172.21.15.40:0/635754507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 msgr2=0x7f443810bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:52.286 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.206+0000 7f443e800700 1 --2- 172.21.15.40:0/635754507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 0x7f443810bbf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f44280098a0 tx=0x7f4428009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:52.286 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.206+0000 7f443e800700 1 -- 172.21.15.40:0/635754507 shutdown_connections 2024-04-26T23:50:52.286 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.206+0000 7f443e800700 1 --2- 172.21.15.40:0/635754507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 0x7f443810bbf0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:52.286 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.206+0000 7f443e800700 1 -- 172.21.15.40:0/635754507 >> 172.21.15.40:0/635754507 conn(0x7f44381031d0 msgr2=0x7f44381055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:52.286 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.206+0000 7f443e800700 1 -- 172.21.15.40:0/635754507 shutdown_connections 2024-04-26T23:50:52.286 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.206+0000 7f443e800700 1 -- 172.21.15.40:0/635754507 wait complete. 2024-04-26T23:50:52.286 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.207+0000 7f443e800700 1 Processor -- start 2024-04-26T23:50:52.286 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.208+0000 7f443e800700 1 -- start start 2024-04-26T23:50:52.286 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.208+0000 7f443e800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 0x7f443819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:52.286 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.208+0000 7f443e800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f443810d080 con 0x7f4438109800 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.208+0000 7f4437400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 0x7f443819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.208+0000 7f4437400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 0x7f443819edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59126/0 (socket says 172.21.15.40:59126) 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.208+0000 7f4437400700 1 -- 172.21.15.40:0/2739781798 learned_addr learned my addr 172.21.15.40:0/2739781798 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.209+0000 7f4437400700 1 -- 172.21.15.40:0/2739781798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443819f2f0 con 0x7f4438109800 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.209+0000 7f4437400700 1 --2- 172.21.15.40:0/2739781798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 0x7f443819edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4428004660 tx=0x7f4428011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.209+0000 7f442fe00700 1 -- 172.21.15.40:0/2739781798 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4428003aa0 con 0x7f4438109800 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.209+0000 7f443e800700 1 -- 172.21.15.40:0/2739781798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443819f580 con 0x7f4438109800 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.209+0000 7f442fe00700 1 -- 172.21.15.40:0/2739781798 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f442801b8f0 con 0x7f4438109800 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.209+0000 7f442fe00700 1 -- 172.21.15.40:0/2739781798 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4428023900 con 0x7f4438109800 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.210+0000 7f443e800700 1 -- 172.21.15.40:0/2739781798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44381a2270 con 0x7f4438109800 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.211+0000 7f443e800700 1 -- 172.21.15.40:0/2739781798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4438064a80 con 0x7f4438109800 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.211+0000 7f442fe00700 1 -- 172.21.15.40:0/2739781798 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4428023b70 con 0x7f4438109800 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.211+0000 7f442fe00700 1 --2- 172.21.15.40:0/2739781798 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f441c039700 0x7f441c03bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.211+0000 7f442fe00700 1 -- 172.21.15.40:0/2739781798 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f442804e0e0 con 0x7f4438109800 2024-04-26T23:50:52.287 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.212+0000 7f4436a00700 1 --2- 172.21.15.40:0/2739781798 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f441c039700 0x7f441c03bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:52.288 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.212+0000 7f4436a00700 1 --2- 172.21.15.40:0/2739781798 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f441c039700 0x7f441c03bbc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4424006fd0 tx=0x7f4424006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:52.288 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.217+0000 7f442fe00700 1 -- 172.21.15.40:0/2739781798 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f442801f030 con 0x7f4438109800 2024-04-26T23:50:52.288 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.460+0000 7f443e800700 1 -- 172.21.15.40:0/2739781798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f44381a2440 con 0x7f4438109800 2024-04-26T23:50:52.288 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.460+0000 7f442fe00700 1 -- 172.21.15.40:0/2739781798 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f44280292a0 con 0x7f4438109800 2024-04-26T23:50:52.288 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.463+0000 7f443e800700 1 -- 172.21.15.40:0/2739781798 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f441c039700 msgr2=0x7f441c03bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:52.288 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.463+0000 7f443e800700 1 --2- 172.21.15.40:0/2739781798 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f441c039700 0x7f441c03bbc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4424006fd0 tx=0x7f4424006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:52.288 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.463+0000 7f443e800700 1 -- 172.21.15.40:0/2739781798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 msgr2=0x7f443819edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:52.288 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.463+0000 7f443e800700 1 --2- 172.21.15.40:0/2739781798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 0x7f443819edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4428004660 tx=0x7f4428011aa0 comp rx=0 tx=0).stop 2024-04-26T23:50:52.288 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.463+0000 7f443e800700 1 -- 172.21.15.40:0/2739781798 shutdown_connections 2024-04-26T23:50:52.289 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.463+0000 7f443e800700 1 --2- 172.21.15.40:0/2739781798 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f441c039700 0x7f441c03bbc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:52.289 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.463+0000 7f443e800700 1 --2- 172.21.15.40:0/2739781798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4438109800 0x7f443819edb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:52.289 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.463+0000 7f443e800700 1 -- 172.21.15.40:0/2739781798 >> 172.21.15.40:0/2739781798 conn(0x7f44381031d0 msgr2=0x7f4438103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:52.289 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.464+0000 7f443e800700 1 -- 172.21.15.40:0/2739781798 shutdown_connections 2024-04-26T23:50:52.289 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:51.464+0000 7f443e800700 1 -- 172.21.15.40:0/2739781798 wait complete. 2024-04-26T23:50:52.325 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T23:50:52.326 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-04-26T23:50:52.352 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T23:50:52.353 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-04-26T23:50:52.353 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-04-26T23:50:52.353 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T23:50:52.360 INFO:teuthology.orchestra.run.smithi040.stdout:Ceph Dashboard is now available at: 2024-04-26T23:50:52.360 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:50:52.360 INFO:teuthology.orchestra.run.smithi040.stdout: URL: https://smithi040.front.sepia.ceph.com:8443/ 2024-04-26T23:50:52.360 INFO:teuthology.orchestra.run.smithi040.stdout: User: admin 2024-04-26T23:50:52.360 INFO:teuthology.orchestra.run.smithi040.stdout: Password: augjtv6n6b 2024-04-26T23:50:52.360 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:50:52.361 INFO:teuthology.orchestra.run.smithi040.stdout:Saving cluster configuration to /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config directory 2024-04-26T23:50:52.364 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling autotune for osd_memory_target 2024-04-26T23:50:54.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.406+0000 7f5df4000700 1 Processor -- start 2024-04-26T23:50:54.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.406+0000 7f5df4000700 1 -- start start 2024-04-26T23:50:54.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.407+0000 7f5df4000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 0x7f5dec10b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:54.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.407+0000 7f5df4000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec10b7f0 con 0x7f5dec10ae90 2024-04-26T23:50:54.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.407+0000 7f5df1600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 0x7f5dec10b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:54.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.407+0000 7f5df1600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 0x7f5dec10b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59136/0 (socket says 172.21.15.40:59136) 2024-04-26T23:50:54.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.407+0000 7f5df1600700 1 -- 172.21.15.40:0/294765038 learned_addr learned my addr 172.21.15.40:0/294765038 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:54.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.408+0000 7f5df1600700 1 -- 172.21.15.40:0/294765038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dec10bff0 con 0x7f5dec10ae90 2024-04-26T23:50:54.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.409+0000 7f5df1600700 1 --2- 172.21.15.40:0/294765038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 0x7f5dec10b2b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5de0009e80 tx=0x7f5de000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb7387caee1ce6f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:54.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.409+0000 7f5debe00700 1 -- 172.21.15.40:0/294765038 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5de0017070 con 0x7f5dec10ae90 2024-04-26T23:50:54.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.409+0000 7f5debe00700 1 -- 172.21.15.40:0/294765038 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5de000b8c0 con 0x7f5dec10ae90 2024-04-26T23:50:54.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.409+0000 7f5debe00700 1 -- 172.21.15.40:0/294765038 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5de0004490 con 0x7f5dec10ae90 2024-04-26T23:50:54.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.410+0000 7f5df4000700 1 -- 172.21.15.40:0/294765038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 msgr2=0x7f5dec10b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:54.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.410+0000 7f5df4000700 1 --2- 172.21.15.40:0/294765038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 0x7f5dec10b2b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5de0009e80 tx=0x7f5de000b330 comp rx=0 tx=0).stop 2024-04-26T23:50:54.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.410+0000 7f5df4000700 1 -- 172.21.15.40:0/294765038 shutdown_connections 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.410+0000 7f5df4000700 1 --2- 172.21.15.40:0/294765038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 0x7f5dec10b2b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.410+0000 7f5df4000700 1 -- 172.21.15.40:0/294765038 >> 172.21.15.40:0/294765038 conn(0x7f5dec07e2a0 msgr2=0x7f5dec07e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.410+0000 7f5df4000700 1 -- 172.21.15.40:0/294765038 shutdown_connections 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.410+0000 7f5df4000700 1 -- 172.21.15.40:0/294765038 wait complete. 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.411+0000 7f5df4000700 1 Processor -- start 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.411+0000 7f5df4000700 1 -- start start 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.411+0000 7f5df4000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 0x7f5dec19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.411+0000 7f5df4000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec10f430 con 0x7f5dec10ae90 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.412+0000 7f5df1600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 0x7f5dec19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.412+0000 7f5df1600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 0x7f5dec19edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59146/0 (socket says 172.21.15.40:59146) 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.412+0000 7f5df1600700 1 -- 172.21.15.40:0/1523367603 learned_addr learned my addr 172.21.15.40:0/1523367603 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.412+0000 7f5df1600700 1 -- 172.21.15.40:0/1523367603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dec19f310 con 0x7f5dec10ae90 2024-04-26T23:50:54.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.412+0000 7f5df1600700 1 --2- 172.21.15.40:0/1523367603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 0x7f5dec19edd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5de0004220 tx=0x7f5de000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.413+0000 7f5dea000700 1 -- 172.21.15.40:0/1523367603 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5de0017040 con 0x7f5dec10ae90 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.413+0000 7f5dea000700 1 -- 172.21.15.40:0/1523367603 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5de0022070 con 0x7f5dec10ae90 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.413+0000 7f5dea000700 1 -- 172.21.15.40:0/1523367603 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5de0012430 con 0x7f5dec10ae90 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.413+0000 7f5df4000700 1 -- 172.21.15.40:0/1523367603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dec19f5a0 con 0x7f5dec10ae90 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.413+0000 7f5df4000700 1 -- 172.21.15.40:0/1523367603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dec1a2290 con 0x7f5dec10ae90 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.414+0000 7f5dea000700 1 -- 172.21.15.40:0/1523367603 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5de001d440 con 0x7f5dec10ae90 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.414+0000 7f5df4000700 1 -- 172.21.15.40:0/1523367603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dec10f5b0 con 0x7f5dec10ae90 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.415+0000 7f5dea000700 1 --2- 172.21.15.40:0/1523367603 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5dd4039700 0x7f5dd403bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.415+0000 7f5dea000700 1 -- 172.21.15.40:0/1523367603 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5de000f050 con 0x7f5dec10ae90 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.415+0000 7f5df0c00700 1 --2- 172.21.15.40:0/1523367603 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5dd4039700 0x7f5dd403bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.416+0000 7f5df0c00700 1 --2- 172.21.15.40:0/1523367603 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5dd4039700 0x7f5dd403bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8006fd0 tx=0x7f5dd8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.422+0000 7f5dea000700 1 -- 172.21.15.40:0/1523367603 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5de0011390 con 0x7f5dec10ae90 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.659+0000 7f5df4000700 1 -- 172.21.15.40:0/1523367603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f5dec1a28b0 con 0x7f5dec10ae90 2024-04-26T23:50:54.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.660+0000 7f5dea000700 1 -- 172.21.15.40:0/1523367603 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f5de002d250 con 0x7f5dec10ae90 2024-04-26T23:50:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.662+0000 7f5df4000700 1 -- 172.21.15.40:0/1523367603 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5dd4039700 msgr2=0x7f5dd403bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.662+0000 7f5df4000700 1 --2- 172.21.15.40:0/1523367603 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5dd4039700 0x7f5dd403bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8006fd0 tx=0x7f5dd8006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.662+0000 7f5df4000700 1 -- 172.21.15.40:0/1523367603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 msgr2=0x7f5dec19edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.662+0000 7f5df4000700 1 --2- 172.21.15.40:0/1523367603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 0x7f5dec19edd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5de0004220 tx=0x7f5de000bd50 comp rx=0 tx=0).stop 2024-04-26T23:50:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.662+0000 7f5df4000700 1 -- 172.21.15.40:0/1523367603 shutdown_connections 2024-04-26T23:50:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.662+0000 7f5df4000700 1 --2- 172.21.15.40:0/1523367603 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5dd4039700 0x7f5dd403bbc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.662+0000 7f5df4000700 1 --2- 172.21.15.40:0/1523367603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dec10ae90 0x7f5dec19edd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.662+0000 7f5df4000700 1 -- 172.21.15.40:0/1523367603 >> 172.21.15.40:0/1523367603 conn(0x7f5dec07e2a0 msgr2=0x7f5dec108470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.663+0000 7f5df4000700 1 -- 172.21.15.40:0/1523367603 shutdown_connections 2024-04-26T23:50:54.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:53.663+0000 7f5df4000700 1 -- 172.21.15.40:0/1523367603 wait complete. 2024-04-26T23:50:56.292 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.266+0000 7f4cab200700 1 Processor -- start 2024-04-26T23:50:56.293 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.266+0000 7f4cab200700 1 -- start start 2024-04-26T23:50:56.293 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.267+0000 7f4cab200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 0x7f4ca407d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:56.293 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.267+0000 7f4cab200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca407d930 con 0x7f4ca407ef00 2024-04-26T23:50:56.293 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.267+0000 7f4ca9e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 0x7f4ca407d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:56.293 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.267+0000 7f4ca9e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 0x7f4ca407d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59154/0 (socket says 172.21.15.40:59154) 2024-04-26T23:50:56.293 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.267+0000 7f4ca9e00700 1 -- 172.21.15.40:0/3011448148 learned_addr learned my addr 172.21.15.40:0/3011448148 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:56.293 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.268+0000 7f4ca9e00700 1 -- 172.21.15.40:0/3011448148 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca407dab0 con 0x7f4ca407ef00 2024-04-26T23:50:56.293 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.268+0000 7f4ca9e00700 1 --2- 172.21.15.40:0/3011448148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 0x7f4ca407d360 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4c980098a0 tx=0x7f4c98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d07cc0ca1ce3e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:56.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.268+0000 7f4ca8a00700 1 -- 172.21.15.40:0/3011448148 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c98004160 con 0x7f4ca407ef00 2024-04-26T23:50:56.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.268+0000 7f4ca8a00700 1 -- 172.21.15.40:0/3011448148 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c98004300 con 0x7f4ca407ef00 2024-04-26T23:50:56.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.268+0000 7f4ca8a00700 1 -- 172.21.15.40:0/3011448148 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c98011a20 con 0x7f4ca407ef00 2024-04-26T23:50:56.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.269+0000 7f4cab200700 1 -- 172.21.15.40:0/3011448148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 msgr2=0x7f4ca407d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:56.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.269+0000 7f4cab200700 1 --2- 172.21.15.40:0/3011448148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 0x7f4ca407d360 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4c980098a0 tx=0x7f4c98009bb0 comp rx=0 tx=0).stop 2024-04-26T23:50:56.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.269+0000 7f4cab200700 1 -- 172.21.15.40:0/3011448148 shutdown_connections 2024-04-26T23:50:56.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.269+0000 7f4cab200700 1 --2- 172.21.15.40:0/3011448148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 0x7f4ca407d360 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:56.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.269+0000 7f4cab200700 1 -- 172.21.15.40:0/3011448148 >> 172.21.15.40:0/3011448148 conn(0x7f4ca41040c0 msgr2=0x7f4ca41064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:56.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.269+0000 7f4cab200700 1 -- 172.21.15.40:0/3011448148 shutdown_connections 2024-04-26T23:50:56.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.269+0000 7f4cab200700 1 -- 172.21.15.40:0/3011448148 wait complete. 2024-04-26T23:50:56.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.270+0000 7f4cab200700 1 Processor -- start 2024-04-26T23:50:56.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.271+0000 7f4cab200700 1 -- start start 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.271+0000 7f4cab200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 0x7f4ca41a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.271+0000 7f4cab200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca410ace0 con 0x7f4ca407ef00 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.271+0000 7f4ca9e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 0x7f4ca41a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.271+0000 7f4ca9e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 0x7f4ca41a2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59164/0 (socket says 172.21.15.40:59164) 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.271+0000 7f4ca9e00700 1 -- 172.21.15.40:0/338906019 learned_addr learned my addr 172.21.15.40:0/338906019 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.272+0000 7f4ca9e00700 1 -- 172.21.15.40:0/338906019 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca41a34f0 con 0x7f4ca407ef00 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.272+0000 7f4ca9e00700 1 --2- 172.21.15.40:0/338906019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 0x7f4ca41a2fb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009520 tx=0x7f4c98011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.272+0000 7f4c9ea00700 1 -- 172.21.15.40:0/338906019 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c98003af0 con 0x7f4ca407ef00 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.272+0000 7f4c9ea00700 1 -- 172.21.15.40:0/338906019 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c9801a8e0 con 0x7f4ca407ef00 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.272+0000 7f4c9ea00700 1 -- 172.21.15.40:0/338906019 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c980247a0 con 0x7f4ca407ef00 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.272+0000 7f4cab200700 1 -- 172.21.15.40:0/338906019 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca41a3780 con 0x7f4ca407ef00 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.272+0000 7f4cab200700 1 -- 172.21.15.40:0/338906019 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca41a6470 con 0x7f4ca407ef00 2024-04-26T23:50:56.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.274+0000 7f4cab200700 1 -- 172.21.15.40:0/338906019 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca4064a80 con 0x7f4ca407ef00 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.274+0000 7f4c9ea00700 1 -- 172.21.15.40:0/338906019 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4c98022020 con 0x7f4ca407ef00 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.274+0000 7f4c9ea00700 1 --2- 172.21.15.40:0/338906019 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4c8c039700 0x7f4c8c03bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.274+0000 7f4c9ea00700 1 -- 172.21.15.40:0/338906019 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4c9804db10 con 0x7f4ca407ef00 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.274+0000 7f4ca9400700 1 --2- 172.21.15.40:0/338906019 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4c8c039700 0x7f4c8c03bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.275+0000 7f4ca9400700 1 --2- 172.21.15.40:0/338906019 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4c8c039700 0x7f4c8c03bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4c94006fd0 tx=0x7f4c94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.279+0000 7f4c9ea00700 1 -- 172.21.15.40:0/338906019 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4c9800b3e0 con 0x7f4ca407ef00 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.633+0000 7f4cab200700 1 -- 172.21.15.40:0/338906019 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f4ca402e720 con 0x7f4ca407ef00 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.639+0000 7f4c9ea00700 1 -- 172.21.15.40:0/338906019 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f4c98016260 con 0x7f4ca407ef00 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.642+0000 7f4cab200700 1 -- 172.21.15.40:0/338906019 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4c8c039700 msgr2=0x7f4c8c03bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.642+0000 7f4cab200700 1 --2- 172.21.15.40:0/338906019 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4c8c039700 0x7f4c8c03bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4c94006fd0 tx=0x7f4c94006e40 comp rx=0 tx=0).stop 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.642+0000 7f4cab200700 1 -- 172.21.15.40:0/338906019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 msgr2=0x7f4ca41a2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.642+0000 7f4cab200700 1 --2- 172.21.15.40:0/338906019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 0x7f4ca41a2fb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009520 tx=0x7f4c98011aa0 comp rx=0 tx=0).stop 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.642+0000 7f4cab200700 1 -- 172.21.15.40:0/338906019 shutdown_connections 2024-04-26T23:50:56.300 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.642+0000 7f4cab200700 1 --2- 172.21.15.40:0/338906019 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4c8c039700 0x7f4c8c03bbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:56.301 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.642+0000 7f4cab200700 1 --2- 172.21.15.40:0/338906019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ca407ef00 0x7f4ca41a2fb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:50:56.301 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.642+0000 7f4cab200700 1 -- 172.21.15.40:0/338906019 >> 172.21.15.40:0/338906019 conn(0x7f4ca41040c0 msgr2=0x7f4ca41064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:50:56.301 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.642+0000 7f4cab200700 1 -- 172.21.15.40:0/338906019 shutdown_connections 2024-04-26T23:50:56.301 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-04-26T23:50:55.642+0000 7f4cab200700 1 -- 172.21.15.40:0/338906019 wait complete. 2024-04-26T23:50:56.301 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T23:50:56.302 INFO:teuthology.orchestra.run.smithi040.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T23:50:56.302 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:50:56.302 INFO:teuthology.orchestra.run.smithi040.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T23:50:56.302 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:50:56.302 INFO:teuthology.orchestra.run.smithi040.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T23:50:56.302 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:50:56.302 INFO:teuthology.orchestra.run.smithi040.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T23:50:56.302 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:50:56.302 INFO:teuthology.orchestra.run.smithi040.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T23:50:56.303 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:50:56.303 INFO:teuthology.orchestra.run.smithi040.stdout: ceph telemetry on 2024-04-26T23:50:56.303 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:50:56.303 INFO:teuthology.orchestra.run.smithi040.stdout:For more information see: 2024-04-26T23:50:56.303 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:50:56.303 INFO:teuthology.orchestra.run.smithi040.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T23:50:56.303 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:50:56.303 INFO:teuthology.orchestra.run.smithi040.stdout:Bootstrap complete. 2024-04-26T23:50:56.401 INFO:tasks.cephadm:Fetching config... 2024-04-26T23:50:56.401 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:50:56.401 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T23:50:56.454 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T23:50:56.454 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:50:56.454 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T23:50:56.500 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T23:50:56.500 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:50:56.500 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T23:50:56.612 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T23:50:56.612 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:50:56.613 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T23:50:56.705 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T23:50:56.706 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtKSNHcphSzzYxj2ic/R3850yAJOBUchuH7beZzqgJ18Tji5m+IpiFiDtxaLUpmpytT2sLKdZNS0xmzhXgb6l6jLQdN7cbFr0SLfMiD3WdKG+Se6QoxqcYN2pk+3B8UQ+ck6oeirYsLsqjxJ8D/uA8uYIBfuvzFKWAXeNFjC4yrflfakuQrUyorccUFbTAC+nc8TLHP/uB3GqghDkpHLamK3y+bahUexqTa83iSWCV/x5LNBKyOnidHhSL+1VQbLBX+yp7fCCLZ+G3KxHYVQIJTH9Tg+QlzoSNZAqMLmyUpkBmtBnUVGzPFab9Ek6/X3EQMPf1mph9oUrkjNW3L6UUNfCyLxfvnJQ38/Aj5vdT9ICREZhWkujJU0r+wTRShxgr8feKlj0S3h6zSWbtVmX8VvBmcWe5qvg6cF960Mlsi5e7dMKbhPJnHdmx2dNzRNP1JiIfZ842uLMk+1qrtlLZ81WVwEKdoyUQDINEJw3POb87kMCR5Xrx5sKSXXVbAbk= ceph-78cff6ea-0427-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T23:50:56.828 INFO:teuthology.orchestra.run.smithi040.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtKSNHcphSzzYxj2ic/R3850yAJOBUchuH7beZzqgJ18Tji5m+IpiFiDtxaLUpmpytT2sLKdZNS0xmzhXgb6l6jLQdN7cbFr0SLfMiD3WdKG+Se6QoxqcYN2pk+3B8UQ+ck6oeirYsLsqjxJ8D/uA8uYIBfuvzFKWAXeNFjC4yrflfakuQrUyorccUFbTAC+nc8TLHP/uB3GqghDkpHLamK3y+bahUexqTa83iSWCV/x5LNBKyOnidHhSL+1VQbLBX+yp7fCCLZ+G3KxHYVQIJTH9Tg+QlzoSNZAqMLmyUpkBmtBnUVGzPFab9Ek6/X3EQMPf1mph9oUrkjNW3L6UUNfCyLxfvnJQ38/Aj5vdT9ICREZhWkujJU0r+wTRShxgr8feKlj0S3h6zSWbtVmX8VvBmcWe5qvg6cF960Mlsi5e7dMKbhPJnHdmx2dNzRNP1JiIfZ842uLMk+1qrtlLZ81WVwEKdoyUQDINEJw3POb87kMCR5Xrx5sKSXXVbAbk= ceph-78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:50:56.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:50:56 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/338906019' entity='client.admin' 2024-04-26T23:50:56.862 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtKSNHcphSzzYxj2ic/R3850yAJOBUchuH7beZzqgJ18Tji5m+IpiFiDtxaLUpmpytT2sLKdZNS0xmzhXgb6l6jLQdN7cbFr0SLfMiD3WdKG+Se6QoxqcYN2pk+3B8UQ+ck6oeirYsLsqjxJ8D/uA8uYIBfuvzFKWAXeNFjC4yrflfakuQrUyorccUFbTAC+nc8TLHP/uB3GqghDkpHLamK3y+bahUexqTa83iSWCV/x5LNBKyOnidHhSL+1VQbLBX+yp7fCCLZ+G3KxHYVQIJTH9Tg+QlzoSNZAqMLmyUpkBmtBnUVGzPFab9Ek6/X3EQMPf1mph9oUrkjNW3L6UUNfCyLxfvnJQ38/Aj5vdT9ICREZhWkujJU0r+wTRShxgr8feKlj0S3h6zSWbtVmX8VvBmcWe5qvg6cF960Mlsi5e7dMKbhPJnHdmx2dNzRNP1JiIfZ842uLMk+1qrtlLZ81WVwEKdoyUQDINEJw3POb87kMCR5Xrx5sKSXXVbAbk= ceph-78cff6ea-0427-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T23:50:56.906 INFO:teuthology.orchestra.run.smithi080.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtKSNHcphSzzYxj2ic/R3850yAJOBUchuH7beZzqgJ18Tji5m+IpiFiDtxaLUpmpytT2sLKdZNS0xmzhXgb6l6jLQdN7cbFr0SLfMiD3WdKG+Se6QoxqcYN2pk+3B8UQ+ck6oeirYsLsqjxJ8D/uA8uYIBfuvzFKWAXeNFjC4yrflfakuQrUyorccUFbTAC+nc8TLHP/uB3GqghDkpHLamK3y+bahUexqTa83iSWCV/x5LNBKyOnidHhSL+1VQbLBX+yp7fCCLZ+G3KxHYVQIJTH9Tg+QlzoSNZAqMLmyUpkBmtBnUVGzPFab9Ek6/X3EQMPf1mph9oUrkjNW3L6UUNfCyLxfvnJQ38/Aj5vdT9ICREZhWkujJU0r+wTRShxgr8feKlj0S3h6zSWbtVmX8VvBmcWe5qvg6cF960Mlsi5e7dMKbhPJnHdmx2dNzRNP1JiIfZ842uLMk+1qrtlLZ81WVwEKdoyUQDINEJw3POb87kMCR5Xrx5sKSXXVbAbk= ceph-78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:50:56.919 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtKSNHcphSzzYxj2ic/R3850yAJOBUchuH7beZzqgJ18Tji5m+IpiFiDtxaLUpmpytT2sLKdZNS0xmzhXgb6l6jLQdN7cbFr0SLfMiD3WdKG+Se6QoxqcYN2pk+3B8UQ+ck6oeirYsLsqjxJ8D/uA8uYIBfuvzFKWAXeNFjC4yrflfakuQrUyorccUFbTAC+nc8TLHP/uB3GqghDkpHLamK3y+bahUexqTa83iSWCV/x5LNBKyOnidHhSL+1VQbLBX+yp7fCCLZ+G3KxHYVQIJTH9Tg+QlzoSNZAqMLmyUpkBmtBnUVGzPFab9Ek6/X3EQMPf1mph9oUrkjNW3L6UUNfCyLxfvnJQ38/Aj5vdT9ICREZhWkujJU0r+wTRShxgr8feKlj0S3h6zSWbtVmX8VvBmcWe5qvg6cF960Mlsi5e7dMKbhPJnHdmx2dNzRNP1JiIfZ842uLMk+1qrtlLZ81WVwEKdoyUQDINEJw3POb87kMCR5Xrx5sKSXXVbAbk= ceph-78cff6ea-0427-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T23:50:56.961 INFO:teuthology.orchestra.run.smithi103.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtKSNHcphSzzYxj2ic/R3850yAJOBUchuH7beZzqgJ18Tji5m+IpiFiDtxaLUpmpytT2sLKdZNS0xmzhXgb6l6jLQdN7cbFr0SLfMiD3WdKG+Se6QoxqcYN2pk+3B8UQ+ck6oeirYsLsqjxJ8D/uA8uYIBfuvzFKWAXeNFjC4yrflfakuQrUyorccUFbTAC+nc8TLHP/uB3GqghDkpHLamK3y+bahUexqTa83iSWCV/x5LNBKyOnidHhSL+1VQbLBX+yp7fCCLZ+G3KxHYVQIJTH9Tg+QlzoSNZAqMLmyUpkBmtBnUVGzPFab9Ek6/X3EQMPf1mph9oUrkjNW3L6UUNfCyLxfvnJQ38/Aj5vdT9ICREZhWkujJU0r+wTRShxgr8feKlj0S3h6zSWbtVmX8VvBmcWe5qvg6cF960Mlsi5e7dMKbhPJnHdmx2dNzRNP1JiIfZ842uLMk+1qrtlLZ81WVwEKdoyUQDINEJw3POb87kMCR5Xrx5sKSXXVbAbk= ceph-78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:50:56.972 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T23:50:57.431 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:51:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.126+0000 7f58ee600700 1 -- 172.21.15.40:0/4006590609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58e8077450 msgr2=0x7f58e80758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.126+0000 7f58ee600700 1 --2- 172.21.15.40:0/4006590609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58e8077450 0x7f58e80758b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f58d8009910 tx=0x7f58d8009c20 comp rx=0 tx=0).stop 2024-04-26T23:51:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.127+0000 7f58ee600700 1 -- 172.21.15.40:0/4006590609 shutdown_connections 2024-04-26T23:51:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.127+0000 7f58ee600700 1 --2- 172.21.15.40:0/4006590609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58e8077450 0x7f58e80758b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.127+0000 7f58ee600700 1 -- 172.21.15.40:0/4006590609 >> 172.21.15.40:0/4006590609 conn(0x7f58e80fe420 msgr2=0x7f58e8100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.127+0000 7f58ee600700 1 -- 172.21.15.40:0/4006590609 shutdown_connections 2024-04-26T23:51:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.127+0000 7f58ee600700 1 -- 172.21.15.40:0/4006590609 wait complete. 2024-04-26T23:51:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.128+0000 7f58ee600700 1 Processor -- start 2024-04-26T23:51:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.129+0000 7f58ee600700 1 -- start start 2024-04-26T23:51:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.129+0000 7f58ee600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58e8077450 0x7f58e819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.129+0000 7f58ee600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f58e810afc0 con 0x7f58e8077450 2024-04-26T23:51:00.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.129+0000 7f58e7400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58e8077450 0x7f58e819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:00.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.129+0000 7f58e7400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58e8077450 0x7f58e819a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47872/0 (socket says 172.21.15.40:47872) 2024-04-26T23:51:00.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.129+0000 7f58e7400700 1 -- 172.21.15.40:0/1991478070 learned_addr learned my addr 172.21.15.40:0/1991478070 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:51:00.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.130+0000 7f58e7400700 1 -- 172.21.15.40:0/1991478070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58e819abe0 con 0x7f58e8077450 2024-04-26T23:51:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.130+0000 7f58e7400700 1 --2- 172.21.15.40:0/1991478070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58e8077450 0x7f58e819a6a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f58d8005800 tx=0x7f58d8009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.130+0000 7f58e4c00700 1 -- 172.21.15.40:0/1991478070 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58d8027070 con 0x7f58e8077450 2024-04-26T23:51:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.130+0000 7f58e4c00700 1 -- 172.21.15.40:0/1991478070 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58d800be10 con 0x7f58e8077450 2024-04-26T23:51:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.130+0000 7f58ee600700 1 -- 172.21.15.40:0/1991478070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58e819ae70 con 0x7f58e8077450 2024-04-26T23:51:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.130+0000 7f58e4c00700 1 -- 172.21.15.40:0/1991478070 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58d800f460 con 0x7f58e8077450 2024-04-26T23:51:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.131+0000 7f58ee600700 1 -- 172.21.15.40:0/1991478070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58e819db60 con 0x7f58e8077450 2024-04-26T23:51:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.132+0000 7f58ee600700 1 -- 172.21.15.40:0/1991478070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58e810b740 con 0x7f58e8077450 2024-04-26T23:51:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.132+0000 7f58e4c00700 1 -- 172.21.15.40:0/1991478070 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f58d802a070 con 0x7f58e8077450 2024-04-26T23:51:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.132+0000 7f58e4c00700 1 --2- 172.21.15.40:0/1991478070 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f58cc039650 0x7f58cc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.132+0000 7f58e4c00700 1 -- 172.21.15.40:0/1991478070 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f58d801e030 con 0x7f58e8077450 2024-04-26T23:51:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.133+0000 7f58dea00700 1 --2- 172.21.15.40:0/1991478070 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f58cc039650 0x7f58cc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.133+0000 7f58dea00700 1 --2- 172.21.15.40:0/1991478070 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f58cc039650 0x7f58cc03bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f58d0006fd0 tx=0x7f58d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:00.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.138+0000 7f58e4c00700 1 -- 172.21.15.40:0/1991478070 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f58d8013070 con 0x7f58e8077450 2024-04-26T23:51:00.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.365+0000 7f58ee600700 1 -- 172.21.15.40:0/1991478070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f58e802e5a0 con 0x7f58e8077450 2024-04-26T23:51:00.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.371+0000 7f58e4c00700 1 -- 172.21.15.40:0/1991478070 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f58d80310c0 con 0x7f58e8077450 2024-04-26T23:51:00.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.374+0000 7f58ee600700 1 -- 172.21.15.40:0/1991478070 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f58cc039650 msgr2=0x7f58cc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:00.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.374+0000 7f58ee600700 1 --2- 172.21.15.40:0/1991478070 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f58cc039650 0x7f58cc03bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f58d0006fd0 tx=0x7f58d0006e40 comp rx=0 tx=0).stop 2024-04-26T23:51:00.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.374+0000 7f58ee600700 1 -- 172.21.15.40:0/1991478070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58e8077450 msgr2=0x7f58e819a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:00.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.374+0000 7f58ee600700 1 --2- 172.21.15.40:0/1991478070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58e8077450 0x7f58e819a6a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f58d8005800 tx=0x7f58d8009f60 comp rx=0 tx=0).stop 2024-04-26T23:51:00.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.375+0000 7f58ee600700 1 -- 172.21.15.40:0/1991478070 shutdown_connections 2024-04-26T23:51:00.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.375+0000 7f58ee600700 1 --2- 172.21.15.40:0/1991478070 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f58cc039650 0x7f58cc03bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:00.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.375+0000 7f58ee600700 1 --2- 172.21.15.40:0/1991478070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58e8077450 0x7f58e819a6a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:00.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.375+0000 7f58ee600700 1 -- 172.21.15.40:0/1991478070 >> 172.21.15.40:0/1991478070 conn(0x7f58e80fe420 msgr2=0x7f58e80fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:00.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.375+0000 7f58ee600700 1 -- 172.21.15.40:0/1991478070 shutdown_connections 2024-04-26T23:51:00.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:00.375+0000 7f58ee600700 1 -- 172.21.15.40:0/1991478070 wait complete. 2024-04-26T23:51:00.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:00.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:00.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:51:00.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:00.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:00.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:00.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:00 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1991478070' entity='client.admin' 2024-04-26T23:51:00.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:00.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:01.065 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T23:51:01.066 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T23:51:01.514 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:51:03.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.671+0000 7f2065600700 1 -- 172.21.15.40:0/1860426216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2060103910 msgr2=0x7f2060105d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.671+0000 7f2065600700 1 --2- 172.21.15.40:0/1860426216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2060103910 0x7f2060105d00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2050009910 tx=0x7f2050009c20 comp rx=0 tx=0).stop 2024-04-26T23:51:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.671+0000 7f2065600700 1 -- 172.21.15.40:0/1860426216 shutdown_connections 2024-04-26T23:51:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.671+0000 7f2065600700 1 --2- 172.21.15.40:0/1860426216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2060103910 0x7f2060105d00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.671+0000 7f2065600700 1 -- 172.21.15.40:0/1860426216 >> 172.21.15.40:0/1860426216 conn(0x7f20600fd320 msgr2=0x7f20600ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.671+0000 7f2065600700 1 -- 172.21.15.40:0/1860426216 shutdown_connections 2024-04-26T23:51:03.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.671+0000 7f2065600700 1 -- 172.21.15.40:0/1860426216 wait complete. 2024-04-26T23:51:03.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.672+0000 7f2065600700 1 Processor -- start 2024-04-26T23:51:03.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.672+0000 7f2065600700 1 -- start start 2024-04-26T23:51:03.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.673+0000 7f2065600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2060103910 0x7f2060103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:03.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.673+0000 7f2065600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2060108d60 con 0x7f2060103910 2024-04-26T23:51:03.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.673+0000 7f205ea00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2060103910 0x7f2060103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:03.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.673+0000 7f205ea00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2060103910 0x7f2060103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47886/0 (socket says 172.21.15.40:47886) 2024-04-26T23:51:03.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.673+0000 7f205ea00700 1 -- 172.21.15.40:0/3923741566 learned_addr learned my addr 172.21.15.40:0/3923741566 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:51:03.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.673+0000 7f205ea00700 1 -- 172.21.15.40:0/3923741566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20601036d0 con 0x7f2060103910 2024-04-26T23:51:03.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.674+0000 7f205ea00700 1 --2- 172.21.15.40:0/3923741566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2060103910 0x7f2060103190 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2050005190 tx=0x7f2050004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:03.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.674+0000 7f2057400700 1 -- 172.21.15.40:0/3923741566 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2050027070 con 0x7f2060103910 2024-04-26T23:51:03.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.674+0000 7f2057400700 1 -- 172.21.15.40:0/3923741566 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2050022470 con 0x7f2060103910 2024-04-26T23:51:03.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.674+0000 7f2057400700 1 -- 172.21.15.40:0/3923741566 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f205000f5f0 con 0x7f2060103910 2024-04-26T23:51:03.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.674+0000 7f2065600700 1 -- 172.21.15.40:0/3923741566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20601018c0 con 0x7f2060103910 2024-04-26T23:51:03.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.674+0000 7f2065600700 1 -- 172.21.15.40:0/3923741566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2060101e00 con 0x7f2060103910 2024-04-26T23:51:03.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.675+0000 7f2065600700 1 -- 172.21.15.40:0/3923741566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20601094e0 con 0x7f2060103910 2024-04-26T23:51:03.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.675+0000 7f2057400700 1 -- 172.21.15.40:0/3923741566 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f205002a070 con 0x7f2060103910 2024-04-26T23:51:03.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.676+0000 7f2057400700 1 --2- 172.21.15.40:0/3923741566 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f20440396b0 0x7f204403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:03.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.676+0000 7f2057400700 1 -- 172.21.15.40:0/3923741566 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f205001e030 con 0x7f2060103910 2024-04-26T23:51:03.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.676+0000 7f205e000700 1 --2- 172.21.15.40:0/3923741566 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f20440396b0 0x7f204403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:03.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.677+0000 7f205e000700 1 --2- 172.21.15.40:0/3923741566 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f20440396b0 0x7f204403bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f204c006fd0 tx=0x7f204c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:03.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.682+0000 7f2057400700 1 -- 172.21.15.40:0/3923741566 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2050013070 con 0x7f2060103910 2024-04-26T23:51:03.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.917+0000 7f2065600700 1 -- 172.21.15.40:0/3923741566 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f20601918e0 con 0x7f20440396b0 2024-04-26T23:51:03.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.925+0000 7f2057400700 1 -- 172.21.15.40:0/3923741566 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f20601918e0 con 0x7f20440396b0 2024-04-26T23:51:03.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.928+0000 7f2065600700 1 -- 172.21.15.40:0/3923741566 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f20440396b0 msgr2=0x7f204403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:03.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.928+0000 7f2065600700 1 --2- 172.21.15.40:0/3923741566 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f20440396b0 0x7f204403bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f204c006fd0 tx=0x7f204c006e40 comp rx=0 tx=0).stop 2024-04-26T23:51:03.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.928+0000 7f2065600700 1 -- 172.21.15.40:0/3923741566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2060103910 msgr2=0x7f2060103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:03.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.928+0000 7f2065600700 1 --2- 172.21.15.40:0/3923741566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2060103910 0x7f2060103190 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2050005190 tx=0x7f2050004f80 comp rx=0 tx=0).stop 2024-04-26T23:51:03.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.928+0000 7f2065600700 1 -- 172.21.15.40:0/3923741566 shutdown_connections 2024-04-26T23:51:03.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.928+0000 7f2065600700 1 --2- 172.21.15.40:0/3923741566 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f20440396b0 0x7f204403bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:03.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.928+0000 7f2065600700 1 --2- 172.21.15.40:0/3923741566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2060103910 0x7f2060103190 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:03.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.928+0000 7f2065600700 1 -- 172.21.15.40:0/3923741566 >> 172.21.15.40:0/3923741566 conn(0x7f20600fd320 msgr2=0x7f20600fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:03.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.928+0000 7f2065600700 1 -- 172.21.15.40:0/3923741566 shutdown_connections 2024-04-26T23:51:03.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:03.929+0000 7f2065600700 1 -- 172.21.15.40:0/3923741566 wait complete. 2024-04-26T23:51:04.885 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi080 2024-04-26T23:51:04.885 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-26T23:51:04.885 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/etc/ceph/ceph.conf 2024-04-26T23:51:04.904 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-26T23:51:04.904 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:51:04.962 INFO:tasks.cephadm:Adding host smithi080 to orchestrator... 2024-04-26T23:51:04.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch host add smithi080 2024-04-26T23:51:05.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:04 smithi040 ceph-mon[28095]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:51:05.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:05.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:05.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:05.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:51:05.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:04 smithi040 ceph-mon[28095]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-26T23:51:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:51:06.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:05 smithi040 ceph-mon[28095]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:06.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:05 smithi040 ceph-mon[28095]: Updating smithi040:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:06.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:05 smithi040 ceph-mon[28095]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:51:07.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:06 smithi040 ceph-mon[28095]: Updating smithi040:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T23:51:07.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:07.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:07.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:07.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.485+0000 7fc163a00700 1 -- 172.21.15.40:0/798055332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc15c1052b0 msgr2=0x7fc15c105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:07.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.485+0000 7fc163a00700 1 --2- 172.21.15.40:0/798055332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc15c1052b0 0x7fc15c105690 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc144009910 tx=0x7fc144009c20 comp rx=0 tx=0).stop 2024-04-26T23:51:07.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.486+0000 7fc163a00700 1 -- 172.21.15.40:0/798055332 shutdown_connections 2024-04-26T23:51:07.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.486+0000 7fc163a00700 1 --2- 172.21.15.40:0/798055332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc15c1052b0 0x7fc15c105690 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:07.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.486+0000 7fc163a00700 1 -- 172.21.15.40:0/798055332 >> 172.21.15.40:0/798055332 conn(0x7fc15c100b40 msgr2=0x7fc15c102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:07.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.486+0000 7fc163a00700 1 -- 172.21.15.40:0/798055332 shutdown_connections 2024-04-26T23:51:07.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.486+0000 7fc163a00700 1 -- 172.21.15.40:0/798055332 wait complete. 2024-04-26T23:51:07.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.487+0000 7fc163a00700 1 Processor -- start 2024-04-26T23:51:07.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.487+0000 7fc163a00700 1 -- start start 2024-04-26T23:51:07.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.488+0000 7fc163a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc15c1a0940 0x7fc15c1a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:07.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.488+0000 7fc163a00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c10f530 con 0x7fc15c1a0940 2024-04-26T23:51:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.488+0000 7fc161000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc15c1a0940 0x7fc15c1a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.488+0000 7fc161000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc15c1a0940 0x7fc15c1a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47912/0 (socket says 172.21.15.40:47912) 2024-04-26T23:51:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.488+0000 7fc161000700 1 -- 172.21.15.40:0/998314290 learned_addr learned my addr 172.21.15.40:0/998314290 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:51:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.489+0000 7fc161000700 1 -- 172.21.15.40:0/998314290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15c1a1260 con 0x7fc15c1a0940 2024-04-26T23:51:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.489+0000 7fc161000700 1 --2- 172.21.15.40:0/998314290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc15c1a0940 0x7fc15c1a0d20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc144005950 tx=0x7fc144005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.489+0000 7fc15a000700 1 -- 172.21.15.40:0/998314290 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc144026070 con 0x7fc15c1a0940 2024-04-26T23:51:07.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.489+0000 7fc15a000700 1 -- 172.21.15.40:0/998314290 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc14400b6f0 con 0x7fc15c1a0940 2024-04-26T23:51:07.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.489+0000 7fc163a00700 1 -- 172.21.15.40:0/998314290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc15c1a1550 con 0x7fc15c1a0940 2024-04-26T23:51:07.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.489+0000 7fc15a000700 1 -- 172.21.15.40:0/998314290 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1440037f0 con 0x7fc15c1a0940 2024-04-26T23:51:07.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.490+0000 7fc163a00700 1 -- 172.21.15.40:0/998314290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc15c19ad30 con 0x7fc15c1a0940 2024-04-26T23:51:07.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.491+0000 7fc163a00700 1 -- 172.21.15.40:0/998314290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc15c106790 con 0x7fc15c1a0940 2024-04-26T23:51:07.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.491+0000 7fc15a000700 1 -- 172.21.15.40:0/998314290 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc144029070 con 0x7fc15c1a0940 2024-04-26T23:51:07.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.492+0000 7fc15a000700 1 --2- 172.21.15.40:0/998314290 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc14c039650 0x7fc14c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:07.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.492+0000 7fc15a000700 1 -- 172.21.15.40:0/998314290 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc14401e070 con 0x7fc15c1a0940 2024-04-26T23:51:07.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.492+0000 7fc153e00700 1 --2- 172.21.15.40:0/998314290 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc14c039650 0x7fc14c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:07.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.493+0000 7fc153e00700 1 --2- 172.21.15.40:0/998314290 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc14c039650 0x7fc14c03bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc13c006fd0 tx=0x7fc13c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:07.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.498+0000 7fc15a000700 1 -- 172.21.15.40:0/998314290 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc15c1a1550 con 0x7fc15c1a0940 2024-04-26T23:51:07.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:07.735+0000 7fc163a00700 1 -- 172.21.15.40:0/998314290 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi080", "target": ["mon-mgr", ""]}) -- 0x7fc15c19b010 con 0x7fc14c039650 2024-04-26T23:51:08.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:07 smithi040 ceph-mon[28095]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:09.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:08 smithi040 ceph-mon[28095]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi080", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:51:10.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:09 smithi040 ceph-mon[28095]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:10.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:09 smithi040 ceph-mon[28095]: Deploying cephadm binary to smithi080 2024-04-26T23:51:12.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:11 smithi040 ceph-mon[28095]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:12.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:12.868+0000 7fc15a000700 1 -- 172.21.15.40:0/998314290 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fc15c19b010 con 0x7fc14c039650 2024-04-26T23:51:12.873 INFO:teuthology.orchestra.run.smithi040.stdout:Added host 'smithi080' with addr '172.21.15.80' 2024-04-26T23:51:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:12.871+0000 7fc163a00700 1 -- 172.21.15.40:0/998314290 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc14c039650 msgr2=0x7fc14c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:12.871+0000 7fc163a00700 1 --2- 172.21.15.40:0/998314290 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc14c039650 0x7fc14c03bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc13c006fd0 tx=0x7fc13c006e40 comp rx=0 tx=0).stop 2024-04-26T23:51:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:12.871+0000 7fc163a00700 1 -- 172.21.15.40:0/998314290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc15c1a0940 msgr2=0x7fc15c1a0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:12.871+0000 7fc163a00700 1 --2- 172.21.15.40:0/998314290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc15c1a0940 0x7fc15c1a0d20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc144005950 tx=0x7fc144005780 comp rx=0 tx=0).stop 2024-04-26T23:51:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:12.871+0000 7fc163a00700 1 -- 172.21.15.40:0/998314290 shutdown_connections 2024-04-26T23:51:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:12.871+0000 7fc163a00700 1 --2- 172.21.15.40:0/998314290 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc14c039650 0x7fc14c03bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:12.871+0000 7fc163a00700 1 --2- 172.21.15.40:0/998314290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc15c1a0940 0x7fc15c1a0d20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:12.871+0000 7fc163a00700 1 -- 172.21.15.40:0/998314290 >> 172.21.15.40:0/998314290 conn(0x7fc15c100b40 msgr2=0x7fc15c1027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:12.872+0000 7fc163a00700 1 -- 172.21.15.40:0/998314290 shutdown_connections 2024-04-26T23:51:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:12.872+0000 7fc163a00700 1 -- 172.21.15.40:0/998314290 wait complete. 2024-04-26T23:51:13.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T23:51:14.068 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:51:14.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:13 smithi040 ceph-mon[28095]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:14.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:13 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:14.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:13 smithi040 ceph-mon[28095]: Added host smithi080 2024-04-26T23:51:14.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:13 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:14.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:13 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:15.225 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:16.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.025+0000 7ff764400700 1 -- 172.21.15.40:0/423362356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff75c108ec0 msgr2=0x7ff75c1092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:16.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.025+0000 7ff764400700 1 --2- 172.21.15.40:0/423362356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff75c108ec0 0x7ff75c1092a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff750009910 tx=0x7ff750009c20 comp rx=0 tx=0).stop 2024-04-26T23:51:16.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.025+0000 7ff764400700 1 -- 172.21.15.40:0/423362356 shutdown_connections 2024-04-26T23:51:16.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.025+0000 7ff764400700 1 --2- 172.21.15.40:0/423362356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff75c108ec0 0x7ff75c1092a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:16.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.025+0000 7ff764400700 1 -- 172.21.15.40:0/423362356 >> 172.21.15.40:0/423362356 conn(0x7ff75c077880 msgr2=0x7ff75c077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:16.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.025+0000 7ff764400700 1 -- 172.21.15.40:0/423362356 shutdown_connections 2024-04-26T23:51:16.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.025+0000 7ff764400700 1 -- 172.21.15.40:0/423362356 wait complete. 2024-04-26T23:51:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.026+0000 7ff764400700 1 Processor -- start 2024-04-26T23:51:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.026+0000 7ff764400700 1 -- start start 2024-04-26T23:51:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.027+0000 7ff764400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff75c108ec0 0x7ff75c1a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.027+0000 7ff764400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff75c10d300 con 0x7ff75c108ec0 2024-04-26T23:51:16.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.027+0000 7ff761a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff75c108ec0 0x7ff75c1a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:16.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.027+0000 7ff761a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff75c108ec0 0x7ff75c1a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36060/0 (socket says 172.21.15.40:36060) 2024-04-26T23:51:16.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.027+0000 7ff761a00700 1 -- 172.21.15.40:0/1361126420 learned_addr learned my addr 172.21.15.40:0/1361126420 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:51:16.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.027+0000 7ff761a00700 1 -- 172.21.15.40:0/1361126420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75c1a0c30 con 0x7ff75c108ec0 2024-04-26T23:51:16.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.028+0000 7ff761a00700 1 --2- 172.21.15.40:0/1361126420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff75c108ec0 0x7ff75c1a06f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff7500056d0 tx=0x7ff7500057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:16.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.028+0000 7ff75a000700 1 -- 172.21.15.40:0/1361126420 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff750026070 con 0x7ff75c108ec0 2024-04-26T23:51:16.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.028+0000 7ff75a000700 1 -- 172.21.15.40:0/1361126420 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff75000bba0 con 0x7ff75c108ec0 2024-04-26T23:51:16.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.028+0000 7ff75a000700 1 -- 172.21.15.40:0/1361126420 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff750003920 con 0x7ff75c108ec0 2024-04-26T23:51:16.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.028+0000 7ff764400700 1 -- 172.21.15.40:0/1361126420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75c1a0f20 con 0x7ff75c108ec0 2024-04-26T23:51:16.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.028+0000 7ff764400700 1 -- 172.21.15.40:0/1361126420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff75c19a8b0 con 0x7ff75c108ec0 2024-04-26T23:51:16.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.029+0000 7ff764400700 1 -- 172.21.15.40:0/1361126420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff75c104570 con 0x7ff75c108ec0 2024-04-26T23:51:16.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.029+0000 7ff75a000700 1 -- 172.21.15.40:0/1361126420 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff750029070 con 0x7ff75c108ec0 2024-04-26T23:51:16.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.030+0000 7ff75a000700 1 --2- 172.21.15.40:0/1361126420 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff744039700 0x7ff74403bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:16.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.030+0000 7ff75a000700 1 -- 172.21.15.40:0/1361126420 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff75001e070 con 0x7ff75c108ec0 2024-04-26T23:51:16.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.030+0000 7ff761000700 1 --2- 172.21.15.40:0/1361126420 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff744039700 0x7ff74403bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:16.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.031+0000 7ff761000700 1 --2- 172.21.15.40:0/1361126420 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff744039700 0x7ff74403bbc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff748006fd0 tx=0x7ff748006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:16.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.036+0000 7ff75a000700 1 -- 172.21.15.40:0/1361126420 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff750013070 con 0x7ff75c108ec0 2024-04-26T23:51:16.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:15 smithi040 ceph-mon[28095]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:16.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:15 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:16.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.274+0000 7ff764400700 1 -- 172.21.15.40:0/1361126420 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff75c19bde0 con 0x7ff744039700 2024-04-26T23:51:16.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.276+0000 7ff75a000700 1 -- 172.21.15.40:0/1361126420 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7ff75c19bde0 con 0x7ff744039700 2024-04-26T23:51:16.281 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:51:16.281 INFO:teuthology.orchestra.run.smithi040.stdout:[{"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.80", "hostname": "smithi080", "labels": [], "status": ""}] 2024-04-26T23:51:16.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.279+0000 7ff764400700 1 -- 172.21.15.40:0/1361126420 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff744039700 msgr2=0x7ff74403bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:16.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.279+0000 7ff764400700 1 --2- 172.21.15.40:0/1361126420 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff744039700 0x7ff74403bbc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff748006fd0 tx=0x7ff748006e40 comp rx=0 tx=0).stop 2024-04-26T23:51:16.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.279+0000 7ff764400700 1 -- 172.21.15.40:0/1361126420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff75c108ec0 msgr2=0x7ff75c1a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:16.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.279+0000 7ff764400700 1 --2- 172.21.15.40:0/1361126420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff75c108ec0 0x7ff75c1a06f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff7500056d0 tx=0x7ff7500057b0 comp rx=0 tx=0).stop 2024-04-26T23:51:16.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.279+0000 7ff764400700 1 -- 172.21.15.40:0/1361126420 shutdown_connections 2024-04-26T23:51:16.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.279+0000 7ff764400700 1 --2- 172.21.15.40:0/1361126420 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff744039700 0x7ff74403bbc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:16.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.279+0000 7ff764400700 1 --2- 172.21.15.40:0/1361126420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff75c108ec0 0x7ff75c1a06f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:16.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.279+0000 7ff764400700 1 -- 172.21.15.40:0/1361126420 >> 172.21.15.40:0/1361126420 conn(0x7ff75c077880 msgr2=0x7ff75c0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:16.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.279+0000 7ff764400700 1 -- 172.21.15.40:0/1361126420 shutdown_connections 2024-04-26T23:51:16.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:16.280+0000 7ff764400700 1 -- 172.21.15.40:0/1361126420 wait complete. 2024-04-26T23:51:16.919 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi103 2024-04-26T23:51:16.919 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:51:16.920 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/etc/ceph/ceph.conf 2024-04-26T23:51:16.938 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:51:16.938 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:51:16.995 INFO:tasks.cephadm:Adding host smithi103 to orchestrator... 2024-04-26T23:51:16.996 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch host add smithi103 2024-04-26T23:51:17.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:16 smithi040 ceph-mon[28095]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:51:17.459 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:51:18.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:18 smithi040 ceph-mon[28095]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:19.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.527+0000 7fe84d400700 1 -- 172.21.15.40:0/22788382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe848103ff0 msgr2=0x7fe8481043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:19.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.527+0000 7fe84d400700 1 --2- 172.21.15.40:0/22788382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe848103ff0 0x7fe8481043d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe838009910 tx=0x7fe838009c20 comp rx=0 tx=0).stop 2024-04-26T23:51:19.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.527+0000 7fe84d400700 1 -- 172.21.15.40:0/22788382 shutdown_connections 2024-04-26T23:51:19.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.527+0000 7fe84d400700 1 --2- 172.21.15.40:0/22788382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe848103ff0 0x7fe8481043d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:19.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.527+0000 7fe84d400700 1 -- 172.21.15.40:0/22788382 >> 172.21.15.40:0/22788382 conn(0x7fe8480fd820 msgr2=0x7fe8480ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:19.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.527+0000 7fe84d400700 1 -- 172.21.15.40:0/22788382 shutdown_connections 2024-04-26T23:51:19.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.527+0000 7fe84d400700 1 -- 172.21.15.40:0/22788382 wait complete. 2024-04-26T23:51:19.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.528+0000 7fe84d400700 1 Processor -- start 2024-04-26T23:51:19.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.528+0000 7fe84d400700 1 -- start start 2024-04-26T23:51:19.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.529+0000 7fe84d400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8481a0a50 0x7fe8481a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:19.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.529+0000 7fe84d400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe84810d370 con 0x7fe8481a0a50 2024-04-26T23:51:19.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.529+0000 7fe846800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8481a0a50 0x7fe8481a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:19.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.529+0000 7fe846800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8481a0a50 0x7fe8481a0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36070/0 (socket says 172.21.15.40:36070) 2024-04-26T23:51:19.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.529+0000 7fe846800700 1 -- 172.21.15.40:0/2603185810 learned_addr learned my addr 172.21.15.40:0/2603185810 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:51:19.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.530+0000 7fe846800700 1 -- 172.21.15.40:0/2603185810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8481a1370 con 0x7fe8481a0a50 2024-04-26T23:51:19.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.530+0000 7fe846800700 1 --2- 172.21.15.40:0/2603185810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8481a0a50 0x7fe8481a0e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe838005950 tx=0x7fe838005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:19.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.530+0000 7fe83f400700 1 -- 172.21.15.40:0/2603185810 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe838026070 con 0x7fe8481a0a50 2024-04-26T23:51:19.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.530+0000 7fe83f400700 1 -- 172.21.15.40:0/2603185810 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe83800b750 con 0x7fe8481a0a50 2024-04-26T23:51:19.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.530+0000 7fe84d400700 1 -- 172.21.15.40:0/2603185810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84819ab20 con 0x7fe8481a0a50 2024-04-26T23:51:19.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.530+0000 7fe83f400700 1 -- 172.21.15.40:0/2603185810 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe838003810 con 0x7fe8481a0a50 2024-04-26T23:51:19.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.530+0000 7fe84d400700 1 -- 172.21.15.40:0/2603185810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84819af80 con 0x7fe8481a0a50 2024-04-26T23:51:19.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.532+0000 7fe84d400700 1 -- 172.21.15.40:0/2603185810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe84806bb10 con 0x7fe8481a0a50 2024-04-26T23:51:19.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.532+0000 7fe83f400700 1 -- 172.21.15.40:0/2603185810 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe838029070 con 0x7fe8481a0a50 2024-04-26T23:51:19.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.532+0000 7fe83f400700 1 --2- 172.21.15.40:0/2603185810 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe82c039650 0x7fe82c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:19.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.532+0000 7fe83f400700 1 -- 172.21.15.40:0/2603185810 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe83801e070 con 0x7fe8481a0a50 2024-04-26T23:51:19.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.533+0000 7fe83fe00700 1 --2- 172.21.15.40:0/2603185810 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe82c039650 0x7fe82c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:19.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.533+0000 7fe83fe00700 1 --2- 172.21.15.40:0/2603185810 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe82c039650 0x7fe82c03bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe830006fd0 tx=0x7fe830006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.538+0000 7fe83f400700 1 -- 172.21.15.40:0/2603185810 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe838013070 con 0x7fe8481a0a50 2024-04-26T23:51:19.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:19.781+0000 7fe84d400700 1 -- 172.21.15.40:0/2603185810 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}) -- 0x7fe84819bfe0 con 0x7fe82c039650 2024-04-26T23:51:20.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:19 smithi040 ceph-mon[28095]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:20.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:20.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:20.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:20.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:20.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi080", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:51:20.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:20.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:51:20.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:19 smithi040 ceph-mon[28095]: Updating smithi080:/etc/ceph/ceph.conf 2024-04-26T23:51:20.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:19 smithi040 ceph-mon[28095]: Updating smithi080:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:20.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:19 smithi040 ceph-mon[28095]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:51:21.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:20 smithi040 ceph-mon[28095]: Updating smithi080:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T23:51:21.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:21 smithi040 ceph-mon[28095]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:51:21.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:21 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:21.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:21 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:21.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:21 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:22.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:22 smithi040 ceph-mon[28095]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:22.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:22 smithi040 ceph-mon[28095]: Deploying cephadm binary to smithi103 2024-04-26T23:51:24.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:24 smithi040 ceph-mon[28095]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:24.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:24.802+0000 7fe83f400700 1 -- 172.21.15.40:0/2603185810 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fe84819bfe0 con 0x7fe82c039650 2024-04-26T23:51:24.808 INFO:teuthology.orchestra.run.smithi040.stdout:Added host 'smithi103' with addr '172.21.15.103' 2024-04-26T23:51:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:24.806+0000 7fe84d400700 1 -- 172.21.15.40:0/2603185810 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe82c039650 msgr2=0x7fe82c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:24.806+0000 7fe84d400700 1 --2- 172.21.15.40:0/2603185810 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe82c039650 0x7fe82c03bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe830006fd0 tx=0x7fe830006e40 comp rx=0 tx=0).stop 2024-04-26T23:51:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:24.806+0000 7fe84d400700 1 -- 172.21.15.40:0/2603185810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8481a0a50 msgr2=0x7fe8481a0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:24.806+0000 7fe84d400700 1 --2- 172.21.15.40:0/2603185810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8481a0a50 0x7fe8481a0e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe838005950 tx=0x7fe838005780 comp rx=0 tx=0).stop 2024-04-26T23:51:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:24.806+0000 7fe84d400700 1 -- 172.21.15.40:0/2603185810 shutdown_connections 2024-04-26T23:51:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:24.806+0000 7fe84d400700 1 --2- 172.21.15.40:0/2603185810 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe82c039650 0x7fe82c03bb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:24.806+0000 7fe84d400700 1 --2- 172.21.15.40:0/2603185810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8481a0a50 0x7fe8481a0e30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:24.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:24.806+0000 7fe84d400700 1 -- 172.21.15.40:0/2603185810 >> 172.21.15.40:0/2603185810 conn(0x7fe8480fd820 msgr2=0x7fe84810a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:24.807+0000 7fe84d400700 1 -- 172.21.15.40:0/2603185810 shutdown_connections 2024-04-26T23:51:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:24.807+0000 7fe84d400700 1 -- 172.21.15.40:0/2603185810 wait complete. 2024-04-26T23:51:25.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T23:51:25.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:25 smithi040 ceph-mon[28095]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:25.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:25 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:25.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:25 smithi040 ceph-mon[28095]: Added host smithi103 2024-04-26T23:51:25.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:25 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:25.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:25 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:26.091 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:51:27.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:26 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:27.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.993+0000 7f5a9a800700 1 -- 172.21.15.40:0/116120107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a941052b0 msgr2=0x7f5a94105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:27.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.993+0000 7f5a9a800700 1 --2- 172.21.15.40:0/116120107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a941052b0 0x7f5a94105690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a84009910 tx=0x7f5a84009c20 comp rx=0 tx=0).stop 2024-04-26T23:51:27.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.993+0000 7f5a9a800700 1 -- 172.21.15.40:0/116120107 shutdown_connections 2024-04-26T23:51:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.993+0000 7f5a9a800700 1 --2- 172.21.15.40:0/116120107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a941052b0 0x7f5a94105690 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.993+0000 7f5a9a800700 1 -- 172.21.15.40:0/116120107 >> 172.21.15.40:0/116120107 conn(0x7f5a94100b40 msgr2=0x7f5a94102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.993+0000 7f5a9a800700 1 -- 172.21.15.40:0/116120107 shutdown_connections 2024-04-26T23:51:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.994+0000 7f5a9a800700 1 -- 172.21.15.40:0/116120107 wait complete. 2024-04-26T23:51:28.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.995+0000 7f5a9a800700 1 Processor -- start 2024-04-26T23:51:28.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.995+0000 7f5a9a800700 1 -- start start 2024-04-26T23:51:28.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.996+0000 7f5a9a800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a941052b0 0x7f5a941a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:28.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.996+0000 7f5a9a800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9410f530 con 0x7f5a941052b0 2024-04-26T23:51:28.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.996+0000 7f5a93400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a941052b0 0x7f5a941a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:28.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.996+0000 7f5a93400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a941052b0 0x7f5a941a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52906/0 (socket says 172.21.15.40:52906) 2024-04-26T23:51:28.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.996+0000 7f5a93400700 1 -- 172.21.15.40:0/175295357 learned_addr learned my addr 172.21.15.40:0/175295357 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:51:28.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.996+0000 7f5a93400700 1 -- 172.21.15.40:0/175295357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a941a0c50 con 0x7f5a941052b0 2024-04-26T23:51:28.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.997+0000 7f5a93400700 1 --2- 172.21.15.40:0/175295357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a941052b0 0x7f5a941a0710 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5a84006010 tx=0x7f5a84005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:28.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.997+0000 7f5a8be00700 1 -- 172.21.15.40:0/175295357 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a84026070 con 0x7f5a941052b0 2024-04-26T23:51:28.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.997+0000 7f5a8be00700 1 -- 172.21.15.40:0/175295357 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a8400bba0 con 0x7f5a941052b0 2024-04-26T23:51:28.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.997+0000 7f5a8be00700 1 -- 172.21.15.40:0/175295357 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a84003910 con 0x7f5a941052b0 2024-04-26T23:51:28.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.997+0000 7f5a9a800700 1 -- 172.21.15.40:0/175295357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a941a0f40 con 0x7f5a941052b0 2024-04-26T23:51:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.997+0000 7f5a9a800700 1 -- 172.21.15.40:0/175295357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a9419a8d0 con 0x7f5a941052b0 2024-04-26T23:51:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.998+0000 7f5a9a800700 1 -- 172.21.15.40:0/175295357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a94106790 con 0x7f5a941052b0 2024-04-26T23:51:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.998+0000 7f5a8be00700 1 -- 172.21.15.40:0/175295357 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5a84029070 con 0x7f5a941052b0 2024-04-26T23:51:28.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.999+0000 7f5a8be00700 1 --2- 172.21.15.40:0/175295357 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5a780396b0 0x7f5a7803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:28.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.999+0000 7f5a8be00700 1 -- 172.21.15.40:0/175295357 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5a8401e070 con 0x7f5a941052b0 2024-04-26T23:51:28.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:27.999+0000 7f5a92a00700 1 --2- 172.21.15.40:0/175295357 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5a780396b0 0x7f5a7803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:28.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.000+0000 7f5a92a00700 1 --2- 172.21.15.40:0/175295357 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5a780396b0 0x7f5a7803bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5a80006fd0 tx=0x7f5a80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:28.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.005+0000 7f5a8be00700 1 -- 172.21.15.40:0/175295357 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5a84013070 con 0x7f5a941052b0 2024-04-26T23:51:28.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:27 smithi040 ceph-mon[28095]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:28.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:27 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.243+0000 7f5a9a800700 1 -- 172.21.15.40:0/175295357 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5a9419be70 con 0x7f5a780396b0 2024-04-26T23:51:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.247+0000 7f5a8be00700 1 -- 172.21.15.40:0/175295357 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f5a9419be70 con 0x7f5a780396b0 2024-04-26T23:51:28.251 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:51:28.251 INFO:teuthology.orchestra.run.smithi040.stdout:[{"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.80", "hostname": "smithi080", "labels": [], "status": ""}, {"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}] 2024-04-26T23:51:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.249+0000 7f5a9a800700 1 -- 172.21.15.40:0/175295357 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5a780396b0 msgr2=0x7f5a7803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.249+0000 7f5a9a800700 1 --2- 172.21.15.40:0/175295357 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5a780396b0 0x7f5a7803bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5a80006fd0 tx=0x7f5a80006e40 comp rx=0 tx=0).stop 2024-04-26T23:51:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.250+0000 7f5a9a800700 1 -- 172.21.15.40:0/175295357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a941052b0 msgr2=0x7f5a941a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.250+0000 7f5a9a800700 1 --2- 172.21.15.40:0/175295357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a941052b0 0x7f5a941a0710 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5a84006010 tx=0x7f5a84005780 comp rx=0 tx=0).stop 2024-04-26T23:51:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.250+0000 7f5a9a800700 1 -- 172.21.15.40:0/175295357 shutdown_connections 2024-04-26T23:51:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.250+0000 7f5a9a800700 1 --2- 172.21.15.40:0/175295357 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5a780396b0 0x7f5a7803bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.250+0000 7f5a9a800700 1 --2- 172.21.15.40:0/175295357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a941052b0 0x7f5a941a0710 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.250+0000 7f5a9a800700 1 -- 172.21.15.40:0/175295357 >> 172.21.15.40:0/175295357 conn(0x7f5a94100b40 msgr2=0x7f5a94108050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:28.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.250+0000 7f5a9a800700 1 -- 172.21.15.40:0/175295357 shutdown_connections 2024-04-26T23:51:28.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:28.250+0000 7f5a9a800700 1 -- 172.21.15.40:0/175295357 wait complete. 2024-04-26T23:51:28.922 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T23:51:28.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T23:51:29.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:28 smithi040 ceph-mon[28095]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:51:29.367 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:51:30.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:29 smithi040 ceph-mon[28095]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:31.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.332+0000 7f073a200700 1 -- 172.21.15.40:0/904357768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0734077950 msgr2=0x7f0734077d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:31.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.332+0000 7f073a200700 1 --2- 172.21.15.40:0/904357768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0734077950 0x7f0734077d30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0724009910 tx=0x7f0724009c20 comp rx=0 tx=0).stop 2024-04-26T23:51:31.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.333+0000 7f073a200700 1 -- 172.21.15.40:0/904357768 shutdown_connections 2024-04-26T23:51:31.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.333+0000 7f073a200700 1 --2- 172.21.15.40:0/904357768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0734077950 0x7f0734077d30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:31.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.333+0000 7f073a200700 1 -- 172.21.15.40:0/904357768 >> 172.21.15.40:0/904357768 conn(0x7f0734100b20 msgr2=0x7f0734102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:31.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.333+0000 7f073a200700 1 -- 172.21.15.40:0/904357768 shutdown_connections 2024-04-26T23:51:31.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.333+0000 7f073a200700 1 -- 172.21.15.40:0/904357768 wait complete. 2024-04-26T23:51:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.334+0000 7f073a200700 1 Processor -- start 2024-04-26T23:51:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.334+0000 7f073a200700 1 -- start start 2024-04-26T23:51:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.335+0000 7f073a200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0734077950 0x7f073419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.335+0000 7f073a200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0734113970 con 0x7f0734077950 2024-04-26T23:51:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.335+0000 7f0733400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0734077950 0x7f073419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.335+0000 7f0733400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0734077950 0x7f073419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45316/0 (socket says 172.21.15.40:45316) 2024-04-26T23:51:31.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.335+0000 7f0733400700 1 -- 172.21.15.40:0/4061309264 learned_addr learned my addr 172.21.15.40:0/4061309264 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:51:31.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.336+0000 7f0733400700 1 -- 172.21.15.40:0/4061309264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073419f3d0 con 0x7f0734077950 2024-04-26T23:51:31.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.336+0000 7f0733400700 1 --2- 172.21.15.40:0/4061309264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0734077950 0x7f073419ee90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f07240056d0 tx=0x7f07240057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:31.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.336+0000 7f072be00700 1 -- 172.21.15.40:0/4061309264 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0724026070 con 0x7f0734077950 2024-04-26T23:51:31.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.336+0000 7f072be00700 1 -- 172.21.15.40:0/4061309264 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f072400bc00 con 0x7f0734077950 2024-04-26T23:51:31.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.336+0000 7f072be00700 1 -- 172.21.15.40:0/4061309264 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0724003920 con 0x7f0734077950 2024-04-26T23:51:31.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.336+0000 7f073a200700 1 -- 172.21.15.40:0/4061309264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073419f660 con 0x7f0734077950 2024-04-26T23:51:31.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.337+0000 7f073a200700 1 -- 172.21.15.40:0/4061309264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f073419fac0 con 0x7f0734077950 2024-04-26T23:51:31.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.338+0000 7f073a200700 1 -- 172.21.15.40:0/4061309264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0734079090 con 0x7f0734077950 2024-04-26T23:51:31.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.338+0000 7f072be00700 1 -- 172.21.15.40:0/4061309264 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0724029070 con 0x7f0734077950 2024-04-26T23:51:31.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.339+0000 7f072be00700 1 --2- 172.21.15.40:0/4061309264 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0718039650 0x7f071803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:31.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.339+0000 7f072be00700 1 -- 172.21.15.40:0/4061309264 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f072401e070 con 0x7f0734077950 2024-04-26T23:51:31.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.339+0000 7f0732a00700 1 --2- 172.21.15.40:0/4061309264 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0718039650 0x7f071803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:31.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.340+0000 7f0732a00700 1 --2- 172.21.15.40:0/4061309264 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0718039650 0x7f071803bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0720006fd0 tx=0x7f0720006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:31.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.344+0000 7f072be00700 1 -- 172.21.15.40:0/4061309264 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0724013070 con 0x7f0734077950 2024-04-26T23:51:31.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:31.582+0000 7f073a200700 1 -- 172.21.15.40:0/4061309264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f07341a01f0 con 0x7f0734077950 2024-04-26T23:51:31.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:31.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:31.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:31.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:31 smithi040 ceph-mon[28095]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:31.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:31.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:51:31.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:31.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:51:31.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:31 smithi040 ceph-mon[28095]: Updating smithi103:/etc/ceph/ceph.conf 2024-04-26T23:51:31.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:31 smithi040 ceph-mon[28095]: Updating smithi103:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:31.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:31 smithi040 ceph-mon[28095]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:51:32.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:32.580+0000 7f072be00700 1 -- 172.21.15.40:0/4061309264 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f072400de00 con 0x7f0734077950 2024-04-26T23:51:32.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:32.585+0000 7f073a200700 1 -- 172.21.15.40:0/4061309264 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0718039650 msgr2=0x7f071803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:32.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:32.585+0000 7f073a200700 1 --2- 172.21.15.40:0/4061309264 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0718039650 0x7f071803bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0720006fd0 tx=0x7f0720006e40 comp rx=0 tx=0).stop 2024-04-26T23:51:32.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:32.585+0000 7f073a200700 1 -- 172.21.15.40:0/4061309264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0734077950 msgr2=0x7f073419ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:32.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:32.585+0000 7f073a200700 1 --2- 172.21.15.40:0/4061309264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0734077950 0x7f073419ee90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f07240056d0 tx=0x7f07240057b0 comp rx=0 tx=0).stop 2024-04-26T23:51:32.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:32.585+0000 7f073a200700 1 -- 172.21.15.40:0/4061309264 shutdown_connections 2024-04-26T23:51:32.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:32.585+0000 7f073a200700 1 --2- 172.21.15.40:0/4061309264 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0718039650 0x7f071803bb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:32.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:32.585+0000 7f073a200700 1 --2- 172.21.15.40:0/4061309264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0734077950 0x7f073419ee90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:32.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:32.585+0000 7f073a200700 1 -- 172.21.15.40:0/4061309264 >> 172.21.15.40:0/4061309264 conn(0x7f0734100b20 msgr2=0x7f07341023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:32.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:32.586+0000 7f073a200700 1 -- 172.21.15.40:0/4061309264 shutdown_connections 2024-04-26T23:51:32.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:32.586+0000 7f073a200700 1 -- 172.21.15.40:0/4061309264 wait complete. 2024-04-26T23:51:32.591 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2024-04-26T23:51:32.964 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:32 smithi040 ceph-mon[28095]: Updating smithi103:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T23:51:32.964 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:32 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/4061309264' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T23:51:32.964 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:32.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:32.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:33.327 INFO:tasks.cephadm:Adding mon.a on smithi040 2024-04-26T23:51:33.328 INFO:tasks.cephadm:Adding mon.b on smithi080 2024-04-26T23:51:33.328 INFO:tasks.cephadm:Adding mon.c on smithi103 2024-04-26T23:51:33.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi040:172.21.15.40=a;smithi080:172.21.15.80=b;smithi103:172.21.15.103=c' 2024-04-26T23:51:33.568 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:33.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:33 smithi040 ceph-mon[28095]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:33.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:33 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/4061309264' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T23:51:33.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:33 smithi040 ceph-mon[28095]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:51:34.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.907+0000 7f4d627da700 1 -- 172.21.15.103:0/1226432550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d5c101f90 msgr2=0x7f4d5c102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:34.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.907+0000 7f4d627da700 1 --2- 172.21.15.103:0/1226432550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d5c101f90 0x7f4d5c102370 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c009910 tx=0x7f4d4c009c20 comp rx=0 tx=0).stop 2024-04-26T23:51:34.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.907+0000 7f4d627da700 1 -- 172.21.15.103:0/1226432550 shutdown_connections 2024-04-26T23:51:34.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.907+0000 7f4d627da700 1 --2- 172.21.15.103:0/1226432550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d5c101f90 0x7f4d5c102370 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:34.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.907+0000 7f4d627da700 1 -- 172.21.15.103:0/1226432550 >> 172.21.15.103:0/1226432550 conn(0x7f4d5c0fd820 msgr2=0x7f4d5c0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:34.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.907+0000 7f4d627da700 1 -- 172.21.15.103:0/1226432550 shutdown_connections 2024-04-26T23:51:34.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.907+0000 7f4d627da700 1 -- 172.21.15.103:0/1226432550 wait complete. 2024-04-26T23:51:34.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.908+0000 7f4d627da700 1 Processor -- start 2024-04-26T23:51:34.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.908+0000 7f4d627da700 1 -- start start 2024-04-26T23:51:34.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.908+0000 7f4d627da700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d5c101f90 0x7f4d5c19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:34.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.908+0000 7f4d627da700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c10c210 con 0x7f4d5c101f90 2024-04-26T23:51:34.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.909+0000 7f4d5bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d5c101f90 0x7f4d5c19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:34.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.909+0000 7f4d5bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d5c101f90 0x7f4d5c19e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.103:48836/0 (socket says 172.21.15.103:48836) 2024-04-26T23:51:34.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.909+0000 7f4d5bfff700 1 -- 172.21.15.103:0/1249868889 learned_addr learned my addr 172.21.15.103:0/1249868889 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:51:34.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.910+0000 7f4d5bfff700 1 -- 172.21.15.103:0/1249868889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5c19eb20 con 0x7f4d5c101f90 2024-04-26T23:51:34.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.910+0000 7f4d5bfff700 1 --2- 172.21.15.103:0/1249868889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d5c101f90 0x7f4d5c19e5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c006010 tx=0x7f4d4c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:34.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.910+0000 7f4d597fa700 1 -- 172.21.15.103:0/1249868889 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d4c026070 con 0x7f4d5c101f90 2024-04-26T23:51:34.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.910+0000 7f4d627da700 1 -- 172.21.15.103:0/1249868889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5c19ee10 con 0x7f4d5c101f90 2024-04-26T23:51:34.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.910+0000 7f4d597fa700 1 -- 172.21.15.103:0/1249868889 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d4c00bba0 con 0x7f4d5c101f90 2024-04-26T23:51:34.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.910+0000 7f4d627da700 1 -- 172.21.15.103:0/1249868889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d5c198880 con 0x7f4d5c101f90 2024-04-26T23:51:34.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.910+0000 7f4d597fa700 1 -- 172.21.15.103:0/1249868889 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d4c0038b0 con 0x7f4d5c101f90 2024-04-26T23:51:34.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.911+0000 7f4d627da700 1 -- 172.21.15.103:0/1249868889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d5c103470 con 0x7f4d5c101f90 2024-04-26T23:51:34.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.912+0000 7f4d597fa700 1 -- 172.21.15.103:0/1249868889 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4d4c029070 con 0x7f4d5c101f90 2024-04-26T23:51:34.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.912+0000 7f4d597fa700 1 --2- 172.21.15.103:0/1249868889 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4d44039650 0x7f4d4403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:34.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.912+0000 7f4d597fa700 1 -- 172.21.15.103:0/1249868889 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4d4c01e070 con 0x7f4d5c101f90 2024-04-26T23:51:34.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.913+0000 7f4d5b7fe700 1 --2- 172.21.15.103:0/1249868889 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4d44039650 0x7f4d4403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:34.917 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.914+0000 7f4d5b7fe700 1 --2- 172.21.15.103:0/1249868889 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4d44039650 0x7f4d4403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4d50006fd0 tx=0x7f4d50006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:34.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:34.928+0000 7f4d597fa700 1 -- 172.21.15.103:0/1249868889 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4d4c013070 con 0x7f4d5c101f90 2024-04-26T23:51:35.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:35.137+0000 7f4d627da700 1 -- 172.21.15.103:0/1249868889 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi080:172.21.15.80=b;smithi103:172.21.15.103=c", "target": ["mon-mgr", ""]}) -- 0x7f4d5c0008d0 con 0x7f4d44039650 2024-04-26T23:51:35.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:35.152+0000 7f4d597fa700 1 -- 172.21.15.103:0/1249868889 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4d5c0008d0 con 0x7f4d44039650 2024-04-26T23:51:35.155 INFO:teuthology.orchestra.run.smithi103.stdout:Scheduled mon update... 2024-04-26T23:51:35.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:35.154+0000 7f4d627da700 1 -- 172.21.15.103:0/1249868889 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4d44039650 msgr2=0x7f4d4403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:35.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:35.154+0000 7f4d627da700 1 --2- 172.21.15.103:0/1249868889 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4d44039650 0x7f4d4403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4d50006fd0 tx=0x7f4d50006e40 comp rx=0 tx=0).stop 2024-04-26T23:51:35.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:35.154+0000 7f4d627da700 1 -- 172.21.15.103:0/1249868889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d5c101f90 msgr2=0x7f4d5c19e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:35.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:35.154+0000 7f4d627da700 1 --2- 172.21.15.103:0/1249868889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d5c101f90 0x7f4d5c19e5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c006010 tx=0x7f4d4c005780 comp rx=0 tx=0).stop 2024-04-26T23:51:35.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:35.154+0000 7f4d627da700 1 -- 172.21.15.103:0/1249868889 shutdown_connections 2024-04-26T23:51:35.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:35.154+0000 7f4d627da700 1 --2- 172.21.15.103:0/1249868889 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4d44039650 0x7f4d4403bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:35.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:35.154+0000 7f4d627da700 1 --2- 172.21.15.103:0/1249868889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d5c101f90 0x7f4d5c19e5e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:35.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:35.154+0000 7f4d627da700 1 -- 172.21.15.103:0/1249868889 >> 172.21.15.103:0/1249868889 conn(0x7f4d5c0fd820 msgr2=0x7f4d5c06b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:35.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:35.154+0000 7f4d627da700 1 -- 172.21.15.103:0/1249868889 shutdown_connections 2024-04-26T23:51:35.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:35.154+0000 7f4d627da700 1 -- 172.21.15.103:0/1249868889 wait complete. 2024-04-26T23:51:35.718 DEBUG:teuthology.orchestra.run.smithi080:mon.b> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T23:51:35.721 DEBUG:teuthology.orchestra.run.smithi103:mon.c> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T23:51:35.724 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T23:51:35.724 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T23:51:36.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:36 smithi040 ceph-mon[28095]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:36.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:36 smithi040 ceph-mon[28095]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi080:172.21.15.80=b;smithi103:172.21.15.103=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:51:36.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:36 smithi040 ceph-mon[28095]: Saving service mon spec with placement smithi040:172.21.15.40=a;smithi080:172.21.15.80=b;smithi103:172.21.15.103=c;count:3 2024-04-26T23:51:36.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:36.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:36.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:36.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:51:36.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:36.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:51:36.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:36.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:36 smithi040 ceph-mon[28095]: Deploying daemon mon.c on smithi103 2024-04-26T23:51:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:51:38.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:38 smithi040 ceph-mon[28095]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:38.745 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:38 smithi103 systemd[1]: Starting Ceph mon.c for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:51:39.567 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 podman[32906]: 2024-04-26 23:51:39.274550955 +0000 UTC m=+0.201499369 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:51:39.567 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 podman[32906]: 2024-04-26 23:51:39.386634238 +0000 UTC m=+0.313582649 container create 22caa4266198c439967a131dff07031dfbc911ca8b5a4855ad4f2635cfdeb5a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=) 2024-04-26T23:51:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.666+0000 7f45fda45700 1 -- 172.21.15.103:0/2017609601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f8107680 msgr2=0x7f45f8107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.666+0000 7f45fda45700 1 --2- 172.21.15.103:0/2017609601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f8107680 0x7f45f8107a60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f45ec00aa20 tx=0x7f45ec00ad30 comp rx=0 tx=0).stop 2024-04-26T23:51:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.666+0000 7f45fda45700 1 -- 172.21.15.103:0/2017609601 shutdown_connections 2024-04-26T23:51:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.666+0000 7f45fda45700 1 --2- 172.21.15.103:0/2017609601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f8107680 0x7f45f8107a60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:39.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.666+0000 7f45fda45700 1 -- 172.21.15.103:0/2017609601 >> 172.21.15.103:0/2017609601 conn(0x7f45f8078240 msgr2=0x7f45f8078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:39.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.666+0000 7f45fda45700 1 -- 172.21.15.103:0/2017609601 shutdown_connections 2024-04-26T23:51:39.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.666+0000 7f45fda45700 1 -- 172.21.15.103:0/2017609601 wait complete. 2024-04-26T23:51:39.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.666+0000 7f45fda45700 1 Processor -- start 2024-04-26T23:51:39.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.667+0000 7f45fda45700 1 -- start start 2024-04-26T23:51:39.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.667+0000 7f45fda45700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f8107680 0x7f45f819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:39.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.667+0000 7f45fda45700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f45f810d430 con 0x7f45f8107680 2024-04-26T23:51:39.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.667+0000 7f45f6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f8107680 0x7f45f819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:39.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.668+0000 7f45f6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f8107680 0x7f45f819e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.103:48858/0 (socket says 172.21.15.103:48858) 2024-04-26T23:51:39.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.668+0000 7f45f6ffd700 1 -- 172.21.15.103:0/678044104 learned_addr learned my addr 172.21.15.103:0/678044104 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:51:39.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.668+0000 7f45f6ffd700 1 -- 172.21.15.103:0/678044104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f819ebb0 con 0x7f45f8107680 2024-04-26T23:51:39.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.668+0000 7f45f6ffd700 1 --2- 172.21.15.103:0/678044104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f8107680 0x7f45f819e670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f45ec006010 tx=0x7f45ec005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:39.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.669+0000 7f45fca43700 1 -- 172.21.15.103:0/678044104 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45ec005400 con 0x7f45f8107680 2024-04-26T23:51:39.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.669+0000 7f45fda45700 1 -- 172.21.15.103:0/678044104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45f819ee40 con 0x7f45f8107680 2024-04-26T23:51:39.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.669+0000 7f45fca43700 1 -- 172.21.15.103:0/678044104 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45ec0055a0 con 0x7f45f8107680 2024-04-26T23:51:39.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.669+0000 7f45fda45700 1 -- 172.21.15.103:0/678044104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45f8198740 con 0x7f45f8107680 2024-04-26T23:51:39.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.669+0000 7f45fca43700 1 -- 172.21.15.103:0/678044104 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45ec00e760 con 0x7f45f8107680 2024-04-26T23:51:39.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.670+0000 7f45fda45700 1 -- 172.21.15.103:0/678044104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45f8102d20 con 0x7f45f8107680 2024-04-26T23:51:39.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.670+0000 7f45fca43700 1 -- 172.21.15.103:0/678044104 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f45ec016b40 con 0x7f45f8107680 2024-04-26T23:51:39.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.670+0000 7f45fca43700 1 --2- 172.21.15.103:0/678044104 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f45e0039650 0x7f45e003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:39.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.670+0000 7f45fca43700 1 -- 172.21.15.103:0/678044104 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f45ec01e070 con 0x7f45f8107680 2024-04-26T23:51:39.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.671+0000 7f45f67fc700 1 --2- 172.21.15.103:0/678044104 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f45e0039650 0x7f45e003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:39.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.672+0000 7f45f67fc700 1 --2- 172.21.15.103:0/678044104 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f45e0039650 0x7f45e003bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f45e8006fd0 tx=0x7f45e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:39.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.686+0000 7f45fca43700 1 -- 172.21.15.103:0/678044104 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f45ec014370 con 0x7f45f8107680 2024-04-26T23:51:39.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.799+0000 7f45fca43700 1 -- 172.21.15.103:0/678044104 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f45ec00d3e0 con 0x7f45f8107680 2024-04-26T23:51:39.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 podman[32906]: 2024-04-26 23:51:39.731773909 +0000 UTC m=+0.658722313 container init 22caa4266198c439967a131dff07031dfbc911ca8b5a4855ad4f2635cfdeb5a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-26T23:51:39.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 podman[32906]: 2024-04-26 23:51:39.735442469 +0000 UTC m=+0.662390875 container start 22caa4266198c439967a131dff07031dfbc911ca8b5a4855ad4f2635cfdeb5a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:51:39.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T23:51:39.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-26T23:51:39.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pidfile_write: ignore empty --pid-file 2024-04-26T23:51:39.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: load: jerasure load: lrc 2024-04-26T23:51:39.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: RocksDB version: 7.9.2 2024-04-26T23:51:39.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Git sha 0 2024-04-26T23:51:39.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-26T23:51:39.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: DB SUMMARY 2024-04-26T23:51:39.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: DB Session ID: B2AY1YX793VZF4016HII 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: CURRENT file: CURRENT 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.error_if_exists: 0 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.create_if_missing: 0 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.paranoid_checks: 1 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.env: 0x5612cf994cc0 2024-04-26T23:51:39.852 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T23:51:39.853 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.info_log: 0x5612d1b77d80 2024-04-26T23:51:39.853 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T23:51:39.854 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.statistics: (nil) 2024-04-26T23:51:39.854 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.use_fsync: 0 2024-04-26T23:51:39.854 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_log_file_size: 0 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.allow_fallocate: 1 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.use_direct_reads: 0 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.db_log_dir: 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.wal_dir: 2024-04-26T23:51:39.855 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.write_buffer_manager: 0x5612d1b490e0 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T23:51:39.856 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.unordered_write: 0 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.row_cache: None 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.wal_filter: None 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.two_write_queues: 0 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.wal_compression: 0 2024-04-26T23:51:39.857 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.atomic_flush: 0 2024-04-26T23:51:39.858 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T23:51:39.858 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T23:51:39.858 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T23:51:39.858 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.log_readahead_size: 0 2024-04-26T23:51:39.858 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T23:51:39.858 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T23:51:39.858 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T23:51:39.858 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T23:51:39.858 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T23:51:39.858 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_background_jobs: 2 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_background_compactions: -1 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_subcompactions: 1 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_open_files: -1 2024-04-26T23:51:39.859 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_background_flushes: -1 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Compression algorithms supported: 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: kZSTD supported: 0 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: kXpressCompression supported: 0 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: kZlibCompression supported: 1 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: kSnappyCompression supported: 1 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: kLZ4Compression supported: 1 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: kBZip2Compression supported: 0 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T23:51:39.860 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.merge_operator: 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_filter: None 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5612d1b77f40) 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout: cache_index_and_filter_blocks: 1 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T23:51:39.861 INFO:journalctl@ceph.mon.c.smithi103.stdout: pin_top_level_index_and_filter: 1 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: index_type: 0 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: data_block_index_type: 0 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: index_shortening: 1 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: checksum: 4 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: no_block_cache: 0 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: block_cache: 0x5612d1b7b1f0 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: block_cache_name: BinnedLRUCache 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: block_cache_options: 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: capacity : 536870912 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: num_shard_bits : 4 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: strict_capacity_limit : 0 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: high_pri_pool_ratio: 0.000 2024-04-26T23:51:39.862 INFO:journalctl@ceph.mon.c.smithi103.stdout: block_cache_compressed: (nil) 2024-04-26T23:51:39.863 INFO:journalctl@ceph.mon.c.smithi103.stdout: persistent_cache: (nil) 2024-04-26T23:51:39.863 INFO:journalctl@ceph.mon.c.smithi103.stdout: block_size: 4096 2024-04-26T23:51:39.863 INFO:journalctl@ceph.mon.c.smithi103.stdout: block_size_deviation: 10 2024-04-26T23:51:39.863 INFO:journalctl@ceph.mon.c.smithi103.stdout: block_restart_interval: 16 2024-04-26T23:51:39.863 INFO:journalctl@ceph.mon.c.smithi103.stdout: index_block_restart_interval: 1 2024-04-26T23:51:39.863 INFO:journalctl@ceph.mon.c.smithi103.stdout: metadata_block_size: 4096 2024-04-26T23:51:39.863 INFO:journalctl@ceph.mon.c.smithi103.stdout: partition_filters: 0 2024-04-26T23:51:39.863 INFO:journalctl@ceph.mon.c.smithi103.stdout: use_delta_encoding: 1 2024-04-26T23:51:39.863 INFO:journalctl@ceph.mon.c.smithi103.stdout: filter_policy: bloomfilter 2024-04-26T23:51:39.863 INFO:journalctl@ceph.mon.c.smithi103.stdout: whole_key_filtering: 1 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout: verify_compression: 0 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout: read_amp_bytes_per_bit: 0 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout: format_version: 5 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout: enable_index_compression: 1 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout: block_align: 0 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout: max_auto_readahead_size: 262144 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout: prepopulate_block_cache: 0 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout: initial_auto_readahead_size: 8192 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compression: NoCompression 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T23:51:39.864 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.num_levels: 7 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T23:51:39.865 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T23:51:39.866 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T23:51:39.867 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T23:51:39.868 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.inplace_update_support: 0 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.bloom_locality: 0 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.max_successive_merges: 0 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T23:51:39.869 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.ttl: 2592000 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.enable_blob_files: false 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.min_blob_size: 0 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T23:51:39.870 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T23:51:39.871 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T23:51:39.871 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T23:51:39.871 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-26T23:51:39.871 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-26T23:51:39.871 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b19f1ecb-ed82-46a9-aab1-6db47703a443 2024-04-26T23:51:39.871 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714175499765645, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-26T23:51:39.871 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-26T23:51:39.871 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714175499766214, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714175499, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b19f1ecb-ed82-46a9-aab1-6db47703a443", "db_session_id": "B2AY1YX793VZF4016HII", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-26T23:51:39.871 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714175499766317, "job": 1, "event": "recovery_finished"} 2024-04-26T23:51:39.871 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-26T23:51:39.871 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T23:51:39.871 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5612d1c98000 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: DB pointer 0x5612d1c84000 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout: ** DB Stats ** 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout: 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout: ** Compaction Stats [default] ** 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T23:51:39.872 INFO:journalctl@ceph.mon.c.smithi103.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T23:51:39.873 INFO:journalctl@ceph.mon.c.smithi103.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T23:51:39.873 INFO:journalctl@ceph.mon.c.smithi103.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T23:51:39.873 INFO:journalctl@ceph.mon.c.smithi103.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T23:51:39.873 INFO:journalctl@ceph.mon.c.smithi103.stdout: 2024-04-26T23:51:39.873 INFO:journalctl@ceph.mon.c.smithi103.stdout: ** Compaction Stats [default] ** 2024-04-26T23:51:39.873 INFO:journalctl@ceph.mon.c.smithi103.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T23:51:39.873 INFO:journalctl@ceph.mon.c.smithi103.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T23:51:39.873 INFO:journalctl@ceph.mon.c.smithi103.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T23:51:39.873 INFO:journalctl@ceph.mon.c.smithi103.stdout: 2024-04-26T23:51:39.873 INFO:journalctl@ceph.mon.c.smithi103.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T23:51:39.873 INFO:journalctl@ceph.mon.c.smithi103.stdout: 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: Cumulative compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: Interval compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: Block cache BinnedLRUCache@0x5612d1b7b1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T23:51:39.874 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: using public_addr v2:172.21.15.103:0/0 -> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: starting mon.c rank -1 at public addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] at bind addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(???) e0 preinit fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).mds e1 new map 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout: e1 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout: legacy client fscid: -1 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout: 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout: No filesystems configured 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T23:51:39.875 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Added host smithi080 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.876 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi080", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Updating smithi080:/etc/ceph/ceph.conf 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Updating smithi080:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Updating smithi080:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T23:51:39.877 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Deploying cephadm binary to smithi103 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Added host smithi103 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.878 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.879 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:51:39.879 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.879 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.879 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.879 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Updating smithi103:/etc/ceph/ceph.conf 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Updating smithi103:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Updating smithi103:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/4061309264' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.880 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/4061309264' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi080:172.21.15.80=b;smithi103:172.21.15.103=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Saving service mon spec with placement smithi040:172.21.15.40=a;smithi080:172.21.15.80=b;smithi103:172.21.15.103=c;count:3 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: Deploying daemon mon.c on smithi103 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 ceph-mon[32952]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T23:51:39.881 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 bash[32906]: 22caa4266198c439967a131dff07031dfbc911ca8b5a4855ad4f2635cfdeb5a3 2024-04-26T23:51:39.882 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:39 smithi103 systemd[1]: Started Ceph mon.c for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:51:39.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:39.976+0000 7f45fda45700 1 -- 172.21.15.103:0/678044104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f45f8199c70 con 0x7f45f8107680 2024-04-26T23:51:44.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:44.824+0000 7f45fca43700 1 -- 172.21.15.103:0/678044104 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f45ec00ee20 con 0x7f45f8107680 2024-04-26T23:51:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:51:44.828 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":2,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","modified":"2024-04-26T23:51:39.798478Z","created":"2024-04-26T23:49:22.376995Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T23:51:44.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:44.827+0000 7f45fda45700 1 -- 172.21.15.103:0/678044104 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f45e0039650 msgr2=0x7f45e003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:44.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:44.827+0000 7f45fda45700 1 --2- 172.21.15.103:0/678044104 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f45e0039650 0x7f45e003bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f45e8006fd0 tx=0x7f45e8006e40 comp rx=0 tx=0).stop 2024-04-26T23:51:44.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:44.827+0000 7f45fda45700 1 -- 172.21.15.103:0/678044104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f8107680 msgr2=0x7f45f819e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:44.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:44.827+0000 7f45fda45700 1 --2- 172.21.15.103:0/678044104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f8107680 0x7f45f819e670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f45ec006010 tx=0x7f45ec005050 comp rx=0 tx=0).stop 2024-04-26T23:51:44.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:44.827+0000 7f45fda45700 1 -- 172.21.15.103:0/678044104 shutdown_connections 2024-04-26T23:51:44.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:44.827+0000 7f45fda45700 1 --2- 172.21.15.103:0/678044104 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f45e0039650 0x7f45e003bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:44.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:44.827+0000 7f45fda45700 1 --2- 172.21.15.103:0/678044104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45f8107680 0x7f45f819e670 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:44.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:44.827+0000 7f45fda45700 1 -- 172.21.15.103:0/678044104 >> 172.21.15.103:0/678044104 conn(0x7f45f8078240 msgr2=0x7f45f80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:44.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:44.828+0000 7f45fda45700 1 -- 172.21.15.103:0/678044104 shutdown_connections 2024-04-26T23:51:44.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:44.828+0000 7f45fda45700 1 -- 172.21.15.103:0/678044104 wait complete. 2024-04-26T23:51:44.832 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 2 2024-04-26T23:51:45.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:51:45.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: mon.a calling monitor election 2024-04-26T23:51:45.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:45.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='client.? 172.21.15.103:0/678044104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T23:51:45.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:45.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:45.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:45.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: mon.c calling monitor election 2024-04-26T23:51:45.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:45.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:45.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:45.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:45.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: monmap epoch 2 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: last_changed 2024-04-26T23:51:39.798478+0000 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: created 2024-04-26T23:49:22.376995+0000 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: min_mon_release 19 (squid) 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: election_strategy: 1 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: 1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.c 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: fsmap 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: mgrmap e13: x(active, since 60s) 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: overall HEALTH_OK 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:45.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:45.103 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:45.103 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:51:45.103 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:45.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:51:45.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: mon.a calling monitor election 2024-04-26T23:51:45.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:45.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='client.? 172.21.15.103:0/678044104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T23:51:45.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: mon.c calling monitor election 2024-04-26T23:51:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T23:51:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: monmap epoch 2 2024-04-26T23:51:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:51:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: last_changed 2024-04-26T23:51:39.798478+0000 2024-04-26T23:51:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: created 2024-04-26T23:49:22.376995+0000 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: min_mon_release 19 (squid) 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: election_strategy: 1 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: 1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.c 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: fsmap 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: mgrmap e13: x(active, since 60s) 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: overall HEALTH_OK 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:51:45.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:46.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:45 smithi040 ceph-mon[28095]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:46.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:45 smithi040 ceph-mon[28095]: Deploying daemon mon.b on smithi080 2024-04-26T23:51:46.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:46.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:45 smithi103 ceph-mon[32952]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:46.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:45 smithi103 ceph-mon[32952]: Deploying daemon mon.b on smithi080 2024-04-26T23:51:46.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:46.402 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T23:51:46.402 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T23:51:46.645 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:51:48.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.107+0000 7f4913996700 1 -- 172.21.15.103:0/2399055573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c104130 msgr2=0x7f490c104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:48.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.107+0000 7f4913996700 1 --2- 172.21.15.103:0/2399055573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c104130 0x7f490c104510 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009910 tx=0x7f48fc009c20 comp rx=0 tx=0).stop 2024-04-26T23:51:48.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.107+0000 7f4913996700 1 -- 172.21.15.103:0/2399055573 shutdown_connections 2024-04-26T23:51:48.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.107+0000 7f4913996700 1 --2- 172.21.15.103:0/2399055573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c104130 0x7f490c104510 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:48.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.107+0000 7f4913996700 1 -- 172.21.15.103:0/2399055573 >> 172.21.15.103:0/2399055573 conn(0x7f490c0fd900 msgr2=0x7f490c0ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:48.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.107+0000 7f4913996700 1 -- 172.21.15.103:0/2399055573 shutdown_connections 2024-04-26T23:51:48.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.107+0000 7f4913996700 1 -- 172.21.15.103:0/2399055573 wait complete. 2024-04-26T23:51:48.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.107+0000 7f4913996700 1 Processor -- start 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4913996700 1 -- start start 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4913996700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490c104130 0x7f490c101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4913996700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c1022d0 0x7f490c102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4913996700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f490c10d460 con 0x7f490c1022d0 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4913996700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f490c10d2e0 con 0x7f490c104130 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4911732700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490c104130 0x7f490c101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4911732700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490c104130 0x7f490c101d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35236/0 (socket says 172.21.15.103:35236) 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4911732700 1 -- 172.21.15.103:0/3579475402 learned_addr learned my addr 172.21.15.103:0/3579475402 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4911732700 1 -- 172.21.15.103:0/3579475402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490c104130 msgr2=0x7f490c101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4911732700 1 -- 172.21.15.103:0/3579475402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490c104130 msgr2=0x7f490c101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4911732700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490c104130 0x7f490c101d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4911732700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490c104130 0x7f490c101d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:51:48.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.108+0000 7f4910f31700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c1022d0 0x7f490c102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:48.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.109+0000 7f4910f31700 1 -- 172.21.15.103:0/3579475402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490c104130 msgr2=0x7f490c101d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:51:48.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.109+0000 7f4910f31700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490c104130 0x7f490c101d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:48.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.109+0000 7f4910f31700 1 -- 172.21.15.103:0/3579475402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490c103cb0 con 0x7f490c1022d0 2024-04-26T23:51:48.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.109+0000 7f4910f31700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c1022d0 0x7f490c102750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f490800b4d0 tx=0x7f490800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:48.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.110+0000 7f49027fc700 1 -- 172.21.15.103:0/3579475402 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f4908011630 con 0x7f490c1022d0 2024-04-26T23:51:48.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.110+0000 7f4913996700 1 -- 172.21.15.103:0/3579475402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f490c1a4e80 con 0x7f490c1022d0 2024-04-26T23:51:48.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.110+0000 7f4913996700 1 -- 172.21.15.103:0/3579475402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f490c1a5390 con 0x7f490c1022d0 2024-04-26T23:51:48.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.110+0000 7f49027fc700 1 -- 172.21.15.103:0/3579475402 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4908011c80 con 0x7f490c1022d0 2024-04-26T23:51:48.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.110+0000 7f49027fc700 1 -- 172.21.15.103:0/3579475402 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f490801adb0 con 0x7f490c1022d0 2024-04-26T23:51:48.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.111+0000 7f4913996700 1 -- 172.21.15.103:0/3579475402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48f0005330 con 0x7f490c1022d0 2024-04-26T23:51:48.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.111+0000 7f49027fc700 1 -- 172.21.15.103:0/3579475402 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4908018020 con 0x7f490c1022d0 2024-04-26T23:51:48.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.111+0000 7f49027fc700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f48f80393f0 0x7f48f803b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:48.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.111+0000 7f49027fc700 1 -- 172.21.15.103:0/3579475402 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f490804c220 con 0x7f490c1022d0 2024-04-26T23:51:48.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.112+0000 7f4911732700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f48f80393f0 0x7f48f803b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:48.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.113+0000 7f4911732700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f48f80393f0 0x7f48f803b8b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009750 tx=0x7f48fc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:48.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.129+0000 7f49027fc700 1 -- 172.21.15.103:0/3579475402 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f490801f020 con 0x7f490c1022d0 2024-04-26T23:51:48.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:47 smithi040 ceph-mon[28095]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:48.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:47 smithi103 ceph-mon[32952]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:48.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.406+0000 7f4913996700 1 -- 172.21.15.103:0/3579475402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f48f00058f0 con 0x7f490c1022d0 2024-04-26T23:51:48.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.407+0000 7f49027fc700 1 -- 172.21.15.103:0/3579475402 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f4908020d20 con 0x7f490c1022d0 2024-04-26T23:51:48.410 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:51:48.411 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":2,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","modified":"2024-04-26T23:51:39.798478Z","created":"2024-04-26T23:49:22.376995Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T23:51:48.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.409+0000 7f4913996700 1 -- 172.21.15.103:0/3579475402 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f48f80393f0 msgr2=0x7f48f803b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:48.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.409+0000 7f4913996700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f48f80393f0 0x7f48f803b8b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009750 tx=0x7f48fc01a040 comp rx=0 tx=0).stop 2024-04-26T23:51:48.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.409+0000 7f4913996700 1 -- 172.21.15.103:0/3579475402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c1022d0 msgr2=0x7f490c102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:48.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.409+0000 7f4913996700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c1022d0 0x7f490c102750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f490800b4d0 tx=0x7f490800b890 comp rx=0 tx=0).stop 2024-04-26T23:51:48.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.409+0000 7f4913996700 1 -- 172.21.15.103:0/3579475402 shutdown_connections 2024-04-26T23:51:48.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.409+0000 7f4913996700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f48f80393f0 0x7f48f803b8b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:48.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.409+0000 7f4913996700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f490c104130 0x7f490c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:48.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.409+0000 7f4913996700 1 --2- 172.21.15.103:0/3579475402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f490c1022d0 0x7f490c102750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:48.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.409+0000 7f4913996700 1 -- 172.21.15.103:0/3579475402 >> 172.21.15.103:0/3579475402 conn(0x7f490c0fd900 msgr2=0x7f490c106ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:48.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.409+0000 7f4913996700 1 -- 172.21.15.103:0/3579475402 shutdown_connections 2024-04-26T23:51:48.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:48.409+0000 7f4913996700 1 -- 172.21.15.103:0/3579475402 wait complete. 2024-04-26T23:51:48.415 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 2 2024-04-26T23:51:49.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:48 smithi080 systemd[1]: Started Ceph mon.b for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:51:49.963 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T23:51:49.963 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T23:51:50.207 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:51:54.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:54.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: mon.c calling monitor election 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: mon.a calling monitor election 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: monmap epoch 3 2024-04-26T23:51:54.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: last_changed 2024-04-26T23:51:48.752128+0000 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: created 2024-04-26T23:49:22.376995+0000 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: min_mon_release 19 (squid) 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: election_strategy: 1 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: 1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.c 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: 2: [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon.b 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: fsmap 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: mgrmap e13: x(active, since 69s) 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: overall HEALTH_OK 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:54.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:54.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:54.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: mon.c calling monitor election 2024-04-26T23:51:54.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:51:54.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: mon.a calling monitor election 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: monmap epoch 3 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: last_changed 2024-04-26T23:51:48.752128+0000 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: created 2024-04-26T23:49:22.376995+0000 2024-04-26T23:51:54.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: min_mon_release 19 (squid) 2024-04-26T23:51:54.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: election_strategy: 1 2024-04-26T23:51:54.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-26T23:51:54.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: 1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.c 2024-04-26T23:51:54.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: 2: [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon.b 2024-04-26T23:51:54.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: fsmap 2024-04-26T23:51:54.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:51:54.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: mgrmap e13: x(active, since 69s) 2024-04-26T23:51:54.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: overall HEALTH_OK 2024-04-26T23:51:54.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:54.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:54.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.605+0000 7f19bbfe5700 1 -- 172.21.15.103:0/1699204402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a00035c0 msgr2=0x7f19a0005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:54.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.605+0000 7f19bbfe5700 1 --2- 172.21.15.103:0/1699204402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a00035c0 0x7f19a0005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f19b406b6b0 tx=0x7f19b000feb0 comp rx=0 tx=0).stop 2024-04-26T23:51:54.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.605+0000 7f19bbfe5700 1 -- 172.21.15.103:0/1699204402 shutdown_connections 2024-04-26T23:51:54.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.605+0000 7f19bbfe5700 1 --2- 172.21.15.103:0/1699204402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a00035c0 0x7f19a0005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:54.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.605+0000 7f19bbfe5700 1 -- 172.21.15.103:0/1699204402 >> 172.21.15.103:0/1699204402 conn(0x7f19b40fd960 msgr2=0x7f19b40ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:54.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.605+0000 7f19bbfe5700 1 -- 172.21.15.103:0/1699204402 shutdown_connections 2024-04-26T23:51:54.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.605+0000 7f19bbfe5700 1 -- 172.21.15.103:0/1699204402 wait complete. 2024-04-26T23:51:54.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.606+0000 7f19bbfe5700 1 Processor -- start 2024-04-26T23:51:54.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.606+0000 7f19bbfe5700 1 -- start start 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.606+0000 7f19bbfe5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a00035c0 0x7f19b419adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.606+0000 7f19bbfe5700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f19b419b2f0 0x7f19b419f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.606+0000 7f19bbfe5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b419fbc0 0x7f19b41a0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.606+0000 7f19bbfe5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f19b410ae20 con 0x7f19a00035c0 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.606+0000 7f19bbfe5700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f19b410aca0 con 0x7f19b419b2f0 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.606+0000 7f19bbfe5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f19b410afa0 con 0x7f19b419fbc0 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19ba582700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b419fbc0 0x7f19b41a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19ba582700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b419fbc0 0x7f19b41a0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37840/0 (socket says 172.21.15.103:37840) 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19ba582700 1 -- 172.21.15.103:0/2870692185 learned_addr learned my addr 172.21.15.103:0/2870692185 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19b9580700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f19b419b2f0 0x7f19b419f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19ba582700 1 -- 172.21.15.103:0/2870692185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b419fbc0 msgr2=0x7f19b41a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19ba582700 1 -- 172.21.15.103:0/2870692185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b419fbc0 msgr2=0x7f19b41a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T23:51:54.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19ba582700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b419fbc0 0x7f19b41a0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:51:54.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19ba582700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b419fbc0 0x7f19b41a0070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:51:54.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19b9580700 1 -- 172.21.15.103:0/2870692185 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f19b419b2f0 msgr2=0x7f19b419f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T23:51:54.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19b9580700 1 -- 172.21.15.103:0/2870692185 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f19b419b2f0 msgr2=0x7f19b419f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T23:51:54.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19b9580700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f19b419b2f0 0x7f19b419f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:51:54.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19b9580700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f19b419b2f0 0x7f19b419f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:51:54.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19b9d81700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a00035c0 0x7f19b419adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:54.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19b9d81700 1 -- 172.21.15.103:0/2870692185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b419fbc0 msgr2=0x7f19b41a0070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:51:54.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19b9d81700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b419fbc0 0x7f19b41a0070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:54.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19b9d81700 1 -- 172.21.15.103:0/2870692185 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f19b419b2f0 msgr2=0x7f19b419f680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:51:54.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.607+0000 7f19b9d81700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f19b419b2f0 0x7f19b419f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:54.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.608+0000 7f19b9d81700 1 -- 172.21.15.103:0/2870692185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19b41a7930 con 0x7f19a00035c0 2024-04-26T23:51:54.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.608+0000 7f19b9d81700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a00035c0 0x7f19b419adb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f19b0005ed0 tx=0x7f19b0013960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:54.615 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.608+0000 7f19aaffd700 1 -- 172.21.15.103:0/2870692185 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b0029030 con 0x7f19a00035c0 2024-04-26T23:51:54.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.608+0000 7f19bbfe5700 1 -- 172.21.15.103:0/2870692185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19b41a7b00 con 0x7f19a00035c0 2024-04-26T23:51:54.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.608+0000 7f19aaffd700 1 -- 172.21.15.103:0/2870692185 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b00044e0 con 0x7f19a00035c0 2024-04-26T23:51:54.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.608+0000 7f19bbfe5700 1 -- 172.21.15.103:0/2870692185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19b41a8090 con 0x7f19a00035c0 2024-04-26T23:51:54.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.608+0000 7f19aaffd700 1 -- 172.21.15.103:0/2870692185 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b00038a0 con 0x7f19a00035c0 2024-04-26T23:51:54.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.609+0000 7f19bbfe5700 1 -- 172.21.15.103:0/2870692185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19b4064a80 con 0x7f19a00035c0 2024-04-26T23:51:54.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.610+0000 7f19aaffd700 1 -- 172.21.15.103:0/2870692185 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f19b002c070 con 0x7f19a00035c0 2024-04-26T23:51:54.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.610+0000 7f19aaffd700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1998039ae0 0x7f199803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:54.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.610+0000 7f19aaffd700 1 -- 172.21.15.103:0/2870692185 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f19b004ef10 con 0x7f19a00035c0 2024-04-26T23:51:54.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.610+0000 7f19b9580700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1998039ae0 0x7f199803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:54.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.611+0000 7f19b9580700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1998039ae0 0x7f199803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f19a4009850 tx=0x7f19a4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:54.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.625+0000 7f19aaffd700 1 -- 172.21.15.103:0/2870692185 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f19b00342a0 con 0x7f19a00035c0 2024-04-26T23:51:54.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.926+0000 7f19bbfe5700 1 -- 172.21.15.103:0/2870692185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f19b402e490 con 0x7f19a00035c0 2024-04-26T23:51:54.931 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:54.931 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:54.931 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:54.931 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:54.931 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:54.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.927+0000 7f19aaffd700 1 -- 172.21.15.103:0/2870692185 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f19b002ecf0 con 0x7f19a00035c0 2024-04-26T23:51:54.932 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-26T23:51:54.932 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":3,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","modified":"2024-04-26T23:51:48.752128Z","created":"2024-04-26T23:49:22.376995Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T23:51:54.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.929+0000 7f19bbfe5700 1 -- 172.21.15.103:0/2870692185 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1998039ae0 msgr2=0x7f199803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:54.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.929+0000 7f19bbfe5700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1998039ae0 0x7f199803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f19a4009850 tx=0x7f19a4008040 comp rx=0 tx=0).stop 2024-04-26T23:51:54.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.929+0000 7f19bbfe5700 1 -- 172.21.15.103:0/2870692185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a00035c0 msgr2=0x7f19b419adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:54.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.929+0000 7f19bbfe5700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a00035c0 0x7f19b419adb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f19b0005ed0 tx=0x7f19b0013960 comp rx=0 tx=0).stop 2024-04-26T23:51:54.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.929+0000 7f19bbfe5700 1 -- 172.21.15.103:0/2870692185 shutdown_connections 2024-04-26T23:51:54.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.929+0000 7f19bbfe5700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1998039ae0 0x7f199803bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:54.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.929+0000 7f19bbfe5700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19b419fbc0 0x7f19b41a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:54.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.929+0000 7f19bbfe5700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a00035c0 0x7f19b419adb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:54.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.929+0000 7f19bbfe5700 1 --2- 172.21.15.103:0/2870692185 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f19b419b2f0 0x7f19b419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:54.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.929+0000 7f19bbfe5700 1 -- 172.21.15.103:0/2870692185 >> 172.21.15.103:0/2870692185 conn(0x7f19b40fd960 msgr2=0x7f19b41036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:54.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.929+0000 7f19bbfe5700 1 -- 172.21.15.103:0/2870692185 shutdown_connections 2024-04-26T23:51:54.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:51:54.929+0000 7f19bbfe5700 1 -- 172.21.15.103:0/2870692185 wait complete. 2024-04-26T23:51:54.934 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 3 2024-04-26T23:51:55.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:55.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:55.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:55.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:55.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:56.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:56.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: mon.c calling monitor election 2024-04-26T23:51:56.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:51:56.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: mon.a calling monitor election 2024-04-26T23:51:56.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:56.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:51:56.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:56.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:56.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:56.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:56.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:56.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:56.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:56.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T23:51:56.055 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: monmap epoch 3 2024-04-26T23:51:56.055 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:51:56.055 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: last_changed 2024-04-26T23:51:48.752128+0000 2024-04-26T23:51:56.055 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: created 2024-04-26T23:49:22.376995+0000 2024-04-26T23:51:56.055 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: min_mon_release 19 (squid) 2024-04-26T23:51:56.055 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: election_strategy: 1 2024-04-26T23:51:56.055 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-26T23:51:56.056 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: 1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.c 2024-04-26T23:51:56.056 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: 2: [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon.b 2024-04-26T23:51:56.056 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: fsmap 2024-04-26T23:51:56.056 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:51:56.056 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: mgrmap e13: x(active, since 69s) 2024-04-26T23:51:56.056 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: overall HEALTH_OK 2024-04-26T23:51:56.057 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:56.057 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:56.057 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:56.057 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:56.057 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:56.057 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:56.057 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:56.121 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T23:51:56.121 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T23:51:56.546 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:51:56.930 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: mon.b calling monitor election 2024-04-26T23:51:56.930 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-26T23:51:56.930 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: Updating smithi080:/etc/ceph/ceph.conf 2024-04-26T23:51:56.930 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: Updating smithi103:/etc/ceph/ceph.conf 2024-04-26T23:51:56.930 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: mon.b calling monitor election 2024-04-26T23:51:56.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: mon.c calling monitor election 2024-04-26T23:51:56.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: mon.a calling monitor election 2024-04-26T23:51:56.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T23:51:56.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: monmap epoch 3 2024-04-26T23:51:56.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:51:56.931 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: last_changed 2024-04-26T23:51:48.752128+0000 2024-04-26T23:51:56.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: created 2024-04-26T23:49:22.376995+0000 2024-04-26T23:51:56.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: min_mon_release 19 (squid) 2024-04-26T23:51:56.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: election_strategy: 1 2024-04-26T23:51:56.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-26T23:51:56.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: 1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.c 2024-04-26T23:51:56.932 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: 2: [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon.b 2024-04-26T23:51:56.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: fsmap 2024-04-26T23:51:56.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:51:56.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: mgrmap e13: x(active, since 71s) 2024-04-26T23:51:56.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: overall HEALTH_OK 2024-04-26T23:51:56.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: Updating smithi080:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:56.933 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: Updating smithi103:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:56.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:56.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:56.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:56.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:56.934 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:56 smithi040 ceph-mon[28095]: Updating smithi040:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:57.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: mon.b calling monitor election 2024-04-26T23:51:57.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-26T23:51:57.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: Updating smithi080:/etc/ceph/ceph.conf 2024-04-26T23:51:57.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: Updating smithi103:/etc/ceph/ceph.conf 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: mon.b calling monitor election 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: mon.c calling monitor election 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: mon.a calling monitor election 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: monmap epoch 3 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: last_changed 2024-04-26T23:51:48.752128+0000 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: created 2024-04-26T23:49:22.376995+0000 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: min_mon_release 19 (squid) 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: election_strategy: 1 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: 1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.c 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: 2: [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon.b 2024-04-26T23:51:57.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: fsmap 2024-04-26T23:51:57.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:51:57.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: mgrmap e13: x(active, since 71s) 2024-04-26T23:51:57.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: overall HEALTH_OK 2024-04-26T23:51:57.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: Updating smithi080:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:57.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: Updating smithi103:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:57.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:56 smithi080 ceph-mon[32362]: Updating smithi040:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:57.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: mon.b calling monitor election 2024-04-26T23:51:57.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: Updating smithi040:/etc/ceph/ceph.conf 2024-04-26T23:51:57.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: Updating smithi080:/etc/ceph/ceph.conf 2024-04-26T23:51:57.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: Updating smithi103:/etc/ceph/ceph.conf 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: mon.b calling monitor election 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: mon.c calling monitor election 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: mon.a calling monitor election 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: monmap epoch 3 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: last_changed 2024-04-26T23:51:48.752128+0000 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: created 2024-04-26T23:49:22.376995+0000 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: min_mon_release 19 (squid) 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: election_strategy: 1 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: 1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.c 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: 2: [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon.b 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: fsmap 2024-04-26T23:51:57.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:51:57.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: mgrmap e13: x(active, since 71s) 2024-04-26T23:51:57.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: overall HEALTH_OK 2024-04-26T23:51:57.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: Updating smithi080:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:57.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: Updating smithi103:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:57.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:56 smithi103 ceph-mon[32952]: Updating smithi040:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:51:57.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:57.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:57.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T23:51:57.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:51:57.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:51:57.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:57.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:57 smithi103 ceph-mon[32952]: Reconfiguring daemon mon.a on smithi040 2024-04-26T23:51:57.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:57.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:57.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:57.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T23:51:57.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:51:57.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:51:57.900 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:57.900 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-mon[28095]: Reconfiguring daemon mon.a on smithi040 2024-04-26T23:51:57.900 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:51:57 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:51:57.752+0000 7f352c600700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-26T23:51:58.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:58.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:51:58.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:58.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:58.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:58.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:58.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:58.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:58.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:51:58.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T23:51:58.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:51:58.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:51:58.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:58.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:57 smithi080 ceph-mon[32362]: Reconfiguring daemon mon.a on smithi040 2024-04-26T23:51:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.715+0000 7fbb09800700 1 -- 172.21.15.40:0/2515379263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb04075740 msgr2=0x7fbb04075b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.715+0000 7fbb09800700 1 --2- 172.21.15.40:0/2515379263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb04075740 0x7fbb04075b20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4009910 tx=0x7fbaf4009c20 comp rx=0 tx=0).stop 2024-04-26T23:51:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.716+0000 7fbb09800700 1 -- 172.21.15.40:0/2515379263 shutdown_connections 2024-04-26T23:51:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.716+0000 7fbb09800700 1 --2- 172.21.15.40:0/2515379263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb04075740 0x7fbb04075b20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.716+0000 7fbb09800700 1 -- 172.21.15.40:0/2515379263 >> 172.21.15.40:0/2515379263 conn(0x7fbb040fe960 msgr2=0x7fbb04100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.716+0000 7fbb09800700 1 -- 172.21.15.40:0/2515379263 shutdown_connections 2024-04-26T23:51:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.716+0000 7fbb09800700 1 -- 172.21.15.40:0/2515379263 wait complete. 2024-04-26T23:51:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.717+0000 7fbb09800700 1 Processor -- start 2024-04-26T23:51:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.717+0000 7fbb09800700 1 -- start start 2024-04-26T23:51:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.718+0000 7fbb09800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb0419afc0 0x7fbb0419b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.718+0000 7fbb09800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb0419b8e0 0x7fbb0419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.718+0000 7fbb09800700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbb0419fdd0 0x7fbb041a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.718+0000 7fbb09800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0410f570 con 0x7fbb0419afc0 2024-04-26T23:51:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.718+0000 7fbb09800700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0410f3f0 con 0x7fbb0419fdd0 2024-04-26T23:51:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.718+0000 7fbb09800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0410f6f0 con 0x7fbb0419b8e0 2024-04-26T23:51:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.718+0000 7fbb02a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb0419afc0 0x7fbb0419b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.718+0000 7fbb02a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb0419afc0 0x7fbb0419b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44770/0 (socket says 172.21.15.40:44770) 2024-04-26T23:51:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.718+0000 7fbb02a00700 1 -- 172.21.15.40:0/4044454130 learned_addr learned my addr 172.21.15.40:0/4044454130 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:51:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.718+0000 7fbb03400700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbb0419fdd0 0x7fbb041a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.718+0000 7fbb02000700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb0419b8e0 0x7fbb0419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.718+0000 7fbb02a00700 1 -- 172.21.15.40:0/4044454130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb0419b8e0 msgr2=0x7fbb0419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.719+0000 7fbb02a00700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb0419b8e0 0x7fbb0419f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.719+0000 7fbb02a00700 1 -- 172.21.15.40:0/4044454130 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbb0419fdd0 msgr2=0x7fbb041a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.719+0000 7fbb02a00700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbb0419fdd0 0x7fbb041a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.719+0000 7fbb02a00700 1 -- 172.21.15.40:0/4044454130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb041a7e10 con 0x7fbb0419afc0 2024-04-26T23:51:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.719+0000 7fbb02a00700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb0419afc0 0x7fbb0419b3a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4005950 tx=0x7fbaf4004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.719+0000 7fbafb400700 1 -- 172.21.15.40:0/4044454130 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4004240 con 0x7fbb0419afc0 2024-04-26T23:51:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.719+0000 7fbafb400700 1 -- 172.21.15.40:0/4044454130 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaf400bc00 con 0x7fbb0419afc0 2024-04-26T23:51:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.719+0000 7fbb09800700 1 -- 172.21.15.40:0/4044454130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb041a80a0 con 0x7fbb0419afc0 2024-04-26T23:51:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.719+0000 7fbafb400700 1 -- 172.21.15.40:0/4044454130 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4003970 con 0x7fbb0419afc0 2024-04-26T23:51:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.719+0000 7fbb09800700 1 -- 172.21.15.40:0/4044454130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb041a8580 con 0x7fbb0419afc0 2024-04-26T23:51:58.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.720+0000 7fbb09800700 1 -- 172.21.15.40:0/4044454130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb04064a80 con 0x7fbb0419afc0 2024-04-26T23:51:58.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.721+0000 7fbafb400700 1 -- 172.21.15.40:0/4044454130 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fbaf400b750 con 0x7fbb0419afc0 2024-04-26T23:51:58.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.721+0000 7fbafb400700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbae8039ae0 0x7fbae803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:51:58.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.721+0000 7fbafb400700 1 -- 172.21.15.40:0/4044454130 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbaf404d8b0 con 0x7fbb0419afc0 2024-04-26T23:51:58.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.721+0000 7fbb02000700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbae8039ae0 0x7fbae803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:51:58.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.722+0000 7fbb02000700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbae8039ae0 0x7fbae803bfa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009ad0 tx=0x7fbaf0009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:51:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.727+0000 7fbafb400700 1 -- 172.21.15.40:0/4044454130 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbaf4013070 con 0x7fbb0419afc0 2024-04-26T23:51:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.969+0000 7fbb09800700 1 -- 172.21.15.40:0/4044454130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fbb041a88a0 con 0x7fbb0419afc0 2024-04-26T23:51:58.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.970+0000 7fbafb400700 1 -- 172.21.15.40:0/4044454130 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7fbaf4020370 con 0x7fbb0419afc0 2024-04-26T23:51:58.974 INFO:teuthology.orchestra.run.smithi040.stdout:# minimal ceph.conf for 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:51:58.974 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2024-04-26T23:51:58.974 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-26T23:51:58.974 INFO:teuthology.orchestra.run.smithi040.stdout: mon_host = [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] 2024-04-26T23:51:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.973+0000 7fbb09800700 1 -- 172.21.15.40:0/4044454130 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbae8039ae0 msgr2=0x7fbae803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.973+0000 7fbb09800700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbae8039ae0 0x7fbae803bfa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009ad0 tx=0x7fbaf0009210 comp rx=0 tx=0).stop 2024-04-26T23:51:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.973+0000 7fbb09800700 1 -- 172.21.15.40:0/4044454130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb0419afc0 msgr2=0x7fbb0419b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:51:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.973+0000 7fbb09800700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb0419afc0 0x7fbb0419b3a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4005950 tx=0x7fbaf4004f80 comp rx=0 tx=0).stop 2024-04-26T23:51:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.975+0000 7fbb09800700 1 -- 172.21.15.40:0/4044454130 shutdown_connections 2024-04-26T23:51:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.975+0000 7fbb09800700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbae8039ae0 0x7fbae803bfa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.975+0000 7fbb09800700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbb0419fdd0 0x7fbb041a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.975+0000 7fbb09800700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb0419afc0 0x7fbb0419b3a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.975+0000 7fbb09800700 1 --2- 172.21.15.40:0/4044454130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb0419b8e0 0x7fbb0419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:51:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.975+0000 7fbb09800700 1 -- 172.21.15.40:0/4044454130 >> 172.21.15.40:0/4044454130 conn(0x7fbb040fe960 msgr2=0x7fbb0410af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:51:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.975+0000 7fbb09800700 1 -- 172.21.15.40:0/4044454130 shutdown_connections 2024-04-26T23:51:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:51:58.976+0000 7fbb09800700 1 -- 172.21.15.40:0/4044454130 wait complete. 2024-04-26T23:51:59.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:59 smithi040 ceph-mon[28095]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:51:59.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:51:59 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/4044454130' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:00.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:59 smithi080 ceph-mon[32362]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:00.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:51:59 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/4044454130' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:00.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:59 smithi103 ceph-mon[32952]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:00.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:51:59 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/4044454130' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:00.391 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T23:52:00.392 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:52:00.392 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T23:52:00.481 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:52:00.481 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:52:00.599 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-26T23:52:00.599 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T23:52:00.625 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-26T23:52:00.625 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:52:00.688 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:52:00.688 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T23:52:00.721 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:52:00.721 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:52:00.788 INFO:tasks.cephadm:Adding mgr.x on smithi040 2024-04-26T23:52:00.789 INFO:tasks.cephadm:Adding mgr.y on smithi080 2024-04-26T23:52:00.789 INFO:tasks.cephadm:Adding mgr.z on smithi103 2024-04-26T23:52:00.789 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi040=x;smithi080=y;smithi103=z' 2024-04-26T23:52:00.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:00.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:00.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:00 smithi040 ceph-mon[28095]: Reconfiguring mon.b (monmap changed)... 2024-04-26T23:52:00.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:52:00.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:52:00.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:00.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:00 smithi040 ceph-mon[28095]: Reconfiguring daemon mon.b on smithi080 2024-04-26T23:52:01.061 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:52:01.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:01.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:01.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:00 smithi103 ceph-mon[32952]: Reconfiguring mon.b (monmap changed)... 2024-04-26T23:52:01.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:52:01.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:52:01.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:01.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:00 smithi103 ceph-mon[32952]: Reconfiguring daemon mon.b on smithi080 2024-04-26T23:52:01.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:01.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:01.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:00 smithi080 ceph-mon[32362]: Reconfiguring mon.b (monmap changed)... 2024-04-26T23:52:01.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:52:01.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:52:01.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:01.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:00 smithi080 ceph-mon[32362]: Reconfiguring daemon mon.b on smithi080 2024-04-26T23:52:02.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:01 smithi080 ceph-mon[32362]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:02.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:02.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:02.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:52:02.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:52:02.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:02.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:01 smithi103 ceph-mon[32952]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:02.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:02.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:02.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:52:02.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:52:02.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:02.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:01 smithi040 ceph-mon[28095]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:02.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:02.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:02.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:52:02.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:52:02.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:02.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.511+0000 7fa19f8b1700 1 -- 172.21.15.103:0/2882091060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102fb0 msgr2=0x7fa1981033d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:02.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.511+0000 7fa19f8b1700 1 --2- 172.21.15.103:0/2882091060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102fb0 0x7fa1981033d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa19000b0a0 tx=0x7fa19000b3b0 comp rx=0 tx=0).stop 2024-04-26T23:52:02.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.511+0000 7fa19f8b1700 1 -- 172.21.15.103:0/2882091060 shutdown_connections 2024-04-26T23:52:02.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.511+0000 7fa19f8b1700 1 --2- 172.21.15.103:0/2882091060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102fb0 0x7fa1981033d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.511+0000 7fa19f8b1700 1 -- 172.21.15.103:0/2882091060 >> 172.21.15.103:0/2882091060 conn(0x7fa1980fe550 msgr2=0x7fa198100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.511+0000 7fa19f8b1700 1 -- 172.21.15.103:0/2882091060 shutdown_connections 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.511+0000 7fa19f8b1700 1 -- 172.21.15.103:0/2882091060 wait complete. 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.511+0000 7fa19f8b1700 1 Processor -- start 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.511+0000 7fa19f8b1700 1 -- start start 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19f8b1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102fb0 0x7fa19819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19f8b1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa19819ae80 0x7fa19819fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19f8b1700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa19819b300 0x7fa1981a2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19f8b1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa198108ea0 con 0x7fa198102fb0 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19f8b1700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa198108d20 con 0x7fa19819b300 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19f8b1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa198109020 con 0x7fa19819ae80 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19de4e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa19819b300 0x7fa1981a2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:52:02.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19de4e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa19819b300 0x7fa1981a2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.103:57866/0 (socket says 172.21.15.103:57866) 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19de4e700 1 -- 172.21.15.103:0/1401544483 learned_addr learned my addr 172.21.15.103:0/1401544483 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19ce4c700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa19819ae80 0x7fa19819fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19d64d700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102fb0 0x7fa19819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19de4e700 1 -- 172.21.15.103:0/1401544483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa19819ae80 msgr2=0x7fa19819fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19de4e700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa19819ae80 0x7fa19819fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19de4e700 1 -- 172.21.15.103:0/1401544483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102fb0 msgr2=0x7fa19819a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19de4e700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102fb0 0x7fa19819a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.512+0000 7fa19de4e700 1 -- 172.21.15.103:0/1401544483 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1981a2d50 con 0x7fa19819b300 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.513+0000 7fa19de4e700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa19819b300 0x7fa1981a2480 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa18800b550 tx=0x7fa18800b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.513+0000 7fa1967fc700 1 -- 172.21.15.103:0/1401544483 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188015020 con 0x7fa19819b300 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.513+0000 7fa1967fc700 1 -- 172.21.15.103:0/1401544483 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa188011ce0 con 0x7fa19819b300 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.513+0000 7fa19f8b1700 1 -- 172.21.15.103:0/1401544483 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1981a2fe0 con 0x7fa19819b300 2024-04-26T23:52:02.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.513+0000 7fa1967fc700 1 -- 172.21.15.103:0/1401544483 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188010db0 con 0x7fa19819b300 2024-04-26T23:52:02.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.513+0000 7fa19f8b1700 1 -- 172.21.15.103:0/1401544483 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1981a3570 con 0x7fa19819b300 2024-04-26T23:52:02.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.514+0000 7fa1967fc700 1 -- 172.21.15.103:0/1401544483 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fa188018020 con 0x7fa19819b300 2024-04-26T23:52:02.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.514+0000 7fa19f8b1700 1 -- 172.21.15.103:0/1401544483 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa198064a80 con 0x7fa19819b300 2024-04-26T23:52:02.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.514+0000 7fa1967fc700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa180039ae0 0x7fa18003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:52:02.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.514+0000 7fa1967fc700 1 -- 172.21.15.103:0/1401544483 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa18804c360 con 0x7fa19819b300 2024-04-26T23:52:02.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.514+0000 7fa19d64d700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa180039ae0 0x7fa18003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:52:02.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.515+0000 7fa19d64d700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa180039ae0 0x7fa18003bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa190003f40 tx=0x7fa19000b7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:52:02.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.520+0000 7fa1967fc700 1 -- 172.21.15.103:0/1401544483 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa18801f020 con 0x7fa19819b300 2024-04-26T23:52:02.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.739+0000 7fa19f8b1700 1 -- 172.21.15.103:0/1401544483 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi080=y;smithi103=z", "target": ["mon-mgr", ""]}) -- 0x7fa1980638f0 con 0x7fa180039ae0 2024-04-26T23:52:02.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.753+0000 7fa1967fc700 1 -- 172.21.15.103:0/1401544483 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa1980638f0 con 0x7fa180039ae0 2024-04-26T23:52:02.758 INFO:teuthology.orchestra.run.smithi103.stdout:Scheduled mgr update... 2024-04-26T23:52:02.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.755+0000 7fa19f8b1700 1 -- 172.21.15.103:0/1401544483 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa180039ae0 msgr2=0x7fa18003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:02.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.755+0000 7fa19f8b1700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa180039ae0 0x7fa18003bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa190003f40 tx=0x7fa19000b7f0 comp rx=0 tx=0).stop 2024-04-26T23:52:02.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.755+0000 7fa19f8b1700 1 -- 172.21.15.103:0/1401544483 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa19819b300 msgr2=0x7fa1981a2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:02.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.755+0000 7fa19f8b1700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa19819b300 0x7fa1981a2480 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa18800b550 tx=0x7fa18800b860 comp rx=0 tx=0).stop 2024-04-26T23:52:02.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.756+0000 7fa19f8b1700 1 -- 172.21.15.103:0/1401544483 shutdown_connections 2024-04-26T23:52:02.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.756+0000 7fa19f8b1700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa180039ae0 0x7fa18003bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:02.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.756+0000 7fa19f8b1700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa19819b300 0x7fa1981a2480 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:02.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.756+0000 7fa19f8b1700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa198102fb0 0x7fa19819a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:02.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.756+0000 7fa19f8b1700 1 --2- 172.21.15.103:0/1401544483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa19819ae80 0x7fa19819fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:02.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.756+0000 7fa19f8b1700 1 -- 172.21.15.103:0/1401544483 >> 172.21.15.103:0/1401544483 conn(0x7fa1980fe550 msgr2=0x7fa198100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:52:02.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.756+0000 7fa19f8b1700 1 -- 172.21.15.103:0/1401544483 shutdown_connections 2024-04-26T23:52:02.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:52:02.756+0000 7fa19f8b1700 1 -- 172.21.15.103:0/1401544483 wait complete. 2024-04-26T23:52:03.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:02 smithi103 ceph-mon[32952]: Reconfiguring mon.c (monmap changed)... 2024-04-26T23:52:03.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:02 smithi103 ceph-mon[32952]: Reconfiguring daemon mon.c on smithi103 2024-04-26T23:52:03.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:02 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:03.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:02 smithi040 ceph-mon[28095]: Reconfiguring mon.c (monmap changed)... 2024-04-26T23:52:03.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:02 smithi040 ceph-mon[28095]: Reconfiguring daemon mon.c on smithi103 2024-04-26T23:52:03.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:02 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:03.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:02 smithi080 ceph-mon[32362]: Reconfiguring mon.c (monmap changed)... 2024-04-26T23:52:03.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:02 smithi080 ceph-mon[32362]: Reconfiguring daemon mon.c on smithi103 2024-04-26T23:52:03.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:02 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:04.000 DEBUG:teuthology.orchestra.run.smithi080:mgr.y> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T23:52:04.003 DEBUG:teuthology.orchestra.run.smithi103:mgr.z> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.z.service 2024-04-26T23:52:04.006 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T23:52:04.010 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:52:04.010 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T23:52:04.072 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T23:52:04.072 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-04-26T23:52:04.127 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T23:52:04.128 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:52:04.128 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-04-26T23:52:04.128 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:52:04.128 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:52:04.128 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-26 23:50:58.024031862 +0000 2024-04-26T23:52:04.128 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-26 23:45:38.302397044 +0000 2024-04-26T23:52:04.128 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-26 23:45:38.302397044 +0000 2024-04-26T23:52:04.128 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-26 23:45:38.302397044 +0000 2024-04-26T23:52:04.128 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T23:52:04.215 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-26T23:52:04.215 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-26T23:52:04.215 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000516682 s, 991 kB/s 2024-04-26T23:52:04.220 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T23:52:04.281 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-04-26T23:52:04.334 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T23:52:04.334 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:52:04.334 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-04-26T23:52:04.334 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:52:04.334 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:52:04.334 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-26 23:50:58.117030682 +0000 2024-04-26T23:52:04.334 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-26 23:45:38.922389202 +0000 2024-04-26T23:52:04.335 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-26 23:45:38.922389202 +0000 2024-04-26T23:52:04.335 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-26 23:45:38.922389202 +0000 2024-04-26T23:52:04.335 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T23:52:04.424 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-26T23:52:04.424 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-26T23:52:04.424 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000391601 s, 1.3 MB/s 2024-04-26T23:52:04.428 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T23:52:04.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:04.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi080=y;smithi103=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:52:04.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: Saving service mgr spec with placement smithi040=x;smithi080=y;smithi103=z;count:3 2024-04-26T23:52:04.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:04.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:04.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:04.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:04.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:04.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:04.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:52:04.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T23:52:04.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:52:04.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:04.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:04 smithi040 ceph-mon[28095]: Deploying daemon mgr.z on smithi103 2024-04-26T23:52:04.487 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-04-26T23:52:04.584 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T23:52:04.584 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:52:04.584 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-04-26T23:52:04.584 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:52:04.584 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:52:04.584 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-26 23:50:58.221029362 +0000 2024-04-26T23:52:04.584 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-26 23:45:39.554381209 +0000 2024-04-26T23:52:04.584 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-26 23:45:39.554381209 +0000 2024-04-26T23:52:04.585 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-26 23:45:39.554381209 +0000 2024-04-26T23:52:04.585 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T23:52:04.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:04.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi080=y;smithi103=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:52:04.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: Saving service mgr spec with placement smithi040=x;smithi080=y;smithi103=z;count:3 2024-04-26T23:52:04.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:04.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:04.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:04.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:04.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:04.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:04.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:52:04.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T23:52:04.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:52:04.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:04.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:04 smithi103 ceph-mon[32952]: Deploying daemon mgr.z on smithi103 2024-04-26T23:52:04.672 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-26T23:52:04.672 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-26T23:52:04.672 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000396141 s, 1.3 MB/s 2024-04-26T23:52:04.676 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T23:52:04.734 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-04-26T23:52:04.787 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T23:52:04.787 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:52:04.787 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-04-26T23:52:04.787 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:52:04.787 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:52:04.787 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-26 23:50:58.315028169 +0000 2024-04-26T23:52:04.787 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-26 23:45:40.182373266 +0000 2024-04-26T23:52:04.787 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-26 23:45:40.182373266 +0000 2024-04-26T23:52:04.788 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-26 23:45:40.182373266 +0000 2024-04-26T23:52:04.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T23:52:04.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:04.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi080=y;smithi103=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:52:04.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: Saving service mgr spec with placement smithi040=x;smithi080=y;smithi103=z;count:3 2024-04-26T23:52:04.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:04.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:04.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:04.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:04.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:04.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:04.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:52:04.803 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T23:52:04.803 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:52:04.803 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:04.803 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:04 smithi080 ceph-mon[32362]: Deploying daemon mgr.z on smithi103 2024-04-26T23:52:04.871 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-04-26T23:52:04.871 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-04-26T23:52:04.872 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.0003616 s, 1.4 MB/s 2024-04-26T23:52:04.875 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T23:52:04.935 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-26T23:52:04.935 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T23:52:04.953 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T23:52:04.953 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_1 2024-04-26T23:52:05.011 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T23:52:05.011 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:52:05.011 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-26T23:52:05.011 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:52:05.011 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:52:05.012 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-04-26 23:51:48.186030110 +0000 2024-04-26T23:52:05.012 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-04-26 23:44:33.525599196 +0000 2024-04-26T23:52:05.012 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-04-26 23:44:33.525599196 +0000 2024-04-26T23:52:05.012 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-04-26 23:44:33.525599196 +0000 2024-04-26T23:52:05.012 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T23:52:05.077 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-04-26T23:52:05.078 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-04-26T23:52:05.078 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000295749 s, 1.7 MB/s 2024-04-26T23:52:05.079 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T23:52:05.138 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_2 2024-04-26T23:52:05.196 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T23:52:05.196 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:52:05.196 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-26T23:52:05.196 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:52:05.196 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:52:05.196 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-04-26 23:51:48.187030087 +0000 2024-04-26T23:52:05.196 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-04-26 23:44:33.884590530 +0000 2024-04-26T23:52:05.196 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-04-26 23:44:33.884590530 +0000 2024-04-26T23:52:05.196 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-04-26 23:44:33.884590530 +0000 2024-04-26T23:52:05.197 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T23:52:05.262 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-04-26T23:52:05.263 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-04-26T23:52:05.263 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000215464 s, 2.4 MB/s 2024-04-26T23:52:05.264 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T23:52:05.324 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_3 2024-04-26T23:52:05.382 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T23:52:05.383 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:52:05.383 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-26T23:52:05.383 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:52:05.383 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:52:05.383 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-04-26 23:51:48.187030087 +0000 2024-04-26T23:52:05.383 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-04-26 23:44:34.215582540 +0000 2024-04-26T23:52:05.383 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-04-26 23:44:34.215582540 +0000 2024-04-26T23:52:05.383 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-04-26 23:44:34.215582540 +0000 2024-04-26T23:52:05.383 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T23:52:05.448 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-04-26T23:52:05.449 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-04-26T23:52:05.449 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000207763 s, 2.5 MB/s 2024-04-26T23:52:05.450 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T23:52:05.508 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_4 2024-04-26T23:52:05.531 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:05 smithi103 systemd[1]: Starting Ceph mgr.z for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:52:05.531 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:05 smithi103 podman[34586]: 2024-04-26 23:52:05.527683596 +0000 UTC m=+0.099473974 container create 3a8618c5cbded6abddf27877ecada445365418f0b723065ce8a25f125a19a2ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T23:52:05.564 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T23:52:05.564 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:52:05.564 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-26T23:52:05.564 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:52:05.564 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:52:05.564 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-04-26 23:51:48.187030087 +0000 2024-04-26T23:52:05.564 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-04-26 23:44:34.553574381 +0000 2024-04-26T23:52:05.565 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-04-26 23:44:34.553574381 +0000 2024-04-26T23:52:05.565 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-04-26 23:44:34.553574381 +0000 2024-04-26T23:52:05.565 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T23:52:05.628 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-04-26T23:52:05.628 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-04-26T23:52:05.629 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000208436 s, 2.5 MB/s 2024-04-26T23:52:05.630 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T23:52:05.687 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-26T23:52:05.687 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T23:52:05.703 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T23:52:05.703 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-04-26T23:52:05.758 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T23:52:05.758 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:52:05.758 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-26T23:52:05.759 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:52:05.759 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:52:05.759 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-26 23:52:05.263733513 +0000 2024-04-26T23:52:05.759 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-26 23:44:25.043877997 +0000 2024-04-26T23:52:05.759 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-26 23:44:25.043877997 +0000 2024-04-26T23:52:05.759 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-26 23:44:25.043877997 +0000 2024-04-26T23:52:05.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T23:52:05.783 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:05 smithi103 podman[34586]: 2024-04-26 23:52:05.446051314 +0000 UTC m=+0.017841706 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:52:05.783 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:05 smithi103 podman[34586]: 2024-04-26 23:52:05.713289458 +0000 UTC m=+0.285079836 container init 3a8618c5cbded6abddf27877ecada445365418f0b723065ce8a25f125a19a2ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T23:52:05.783 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:05 smithi103 podman[34586]: 2024-04-26 23:52:05.715877514 +0000 UTC m=+0.287667894 container start 3a8618c5cbded6abddf27877ecada445365418f0b723065ce8a25f125a19a2ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:52:05.783 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:05 smithi103 ceph-mgr[34620]: -- 172.21.15.103:0/978364064 <== mon.1 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561c0862a5a0 con 0x561c085fec00 2024-04-26T23:52:05.808 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-26T23:52:05.809 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-26T23:52:05.809 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000195693 s, 2.6 MB/s 2024-04-26T23:52:05.810 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T23:52:05.869 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-04-26T23:52:05.923 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T23:52:05.924 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:52:05.924 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-26T23:52:05.924 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:52:05.924 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:52:05.924 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-26 23:52:05.263733513 +0000 2024-04-26T23:52:05.924 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-26 23:44:25.382871530 +0000 2024-04-26T23:52:05.924 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-26 23:44:25.383871510 +0000 2024-04-26T23:52:05.924 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-26 23:44:25.382871530 +0000 2024-04-26T23:52:05.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T23:52:05.987 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-26T23:52:05.987 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-26T23:52:05.987 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000192405 s, 2.7 MB/s 2024-04-26T23:52:05.988 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T23:52:06.043 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-04-26T23:52:06.068 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:05 smithi103 bash[34586]: 3a8618c5cbded6abddf27877ecada445365418f0b723065ce8a25f125a19a2ae 2024-04-26T23:52:06.069 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:05 smithi103 systemd[1]: Started Ceph mgr.z for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:52:06.069 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:05 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:05.873+0000 7fc731902200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T23:52:06.069 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:06 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:06.064+0000 7fc731902200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:52:06.084 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T23:52:06.084 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:52:06.084 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-26T23:52:06.085 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:52:06.085 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:52:06.085 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-26 23:52:05.264733495 +0000 2024-04-26T23:52:06.085 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-26 23:44:25.709865291 +0000 2024-04-26T23:52:06.085 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-26 23:44:25.709865291 +0000 2024-04-26T23:52:06.085 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-26 23:44:25.709865291 +0000 2024-04-26T23:52:06.085 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T23:52:06.148 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-26T23:52:06.148 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-26T23:52:06.148 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000201946 s, 2.5 MB/s 2024-04-26T23:52:06.149 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T23:52:06.206 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-04-26T23:52:06.261 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T23:52:06.262 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:52:06.262 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-26T23:52:06.262 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:52:06.262 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:52:06.262 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-26 23:52:05.264733495 +0000 2024-04-26T23:52:06.262 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-26 23:44:26.038859014 +0000 2024-04-26T23:52:06.262 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-26 23:44:26.038859014 +0000 2024-04-26T23:52:06.262 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-04-26 23:44:26.038859014 +0000 2024-04-26T23:52:06.262 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T23:52:06.322 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:06 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:06.172+0000 7fc731902200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T23:52:06.324 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-26T23:52:06.324 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-26T23:52:06.324 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000204828 s, 2.5 MB/s 2024-04-26T23:52:06.325 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T23:52:06.381 INFO:tasks.cephadm:Deploying osd.0 on smithi040 with /dev/vg_nvme/lv_4... 2024-04-26T23:52:06.382 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T23:52:06.581 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:06 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:06.493+0000 7fc731902200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T23:52:06.581 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:06 smithi103 ceph-mon[32952]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:06.581 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:06 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:06.581 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:06 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:06.581 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:06 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:06.581 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:06 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:52:06.581 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:06 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T23:52:06.581 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:06 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:52:06.582 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:06 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:06.704 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:06 smithi080 ceph-mon[32362]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:06.704 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:06 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:06.704 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:06 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:06.704 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:06 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:06.704 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:06 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:52:06.704 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:06 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T23:52:06.705 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:06 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:52:06.705 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:06 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:06.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:06 smithi040 ceph-mon[28095]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:06.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:06.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:06.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:06.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:52:06.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T23:52:06.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:52:06.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:06.834 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:52:06.850 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:06 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:06.576+0000 7fc731902200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T23:52:07.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:07 smithi040 ceph-mon[28095]: Deploying daemon mgr.y on smithi080 2024-04-26T23:52:07.774 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:07 smithi080 ceph-mon[32362]: Deploying daemon mgr.y on smithi080 2024-04-26T23:52:07.789 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:07 smithi103 ceph-mon[32952]: Deploying daemon mgr.y on smithi080 2024-04-26T23:52:08.052 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:07 smithi080 systemd[1]: Starting Ceph mgr.y for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:52:08.100 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:07 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:07.785+0000 7fc731902200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T23:52:08.100 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:07 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:07.952+0000 7fc731902200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T23:52:08.407 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:08 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:08.120+0000 7fc731902200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T23:52:08.422 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:08 smithi080 podman[33740]: 2024-04-26 23:52:08.064986315 +0000 UTC m=+0.018583923 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:52:08.422 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:08 smithi080 podman[33740]: 2024-04-26 23:52:08.16362603 +0000 UTC m=+0.117223611 container create c49e7b5b507ae5d4f82f40755deac3e62e8d016f32794fef2a0913612f5b9d58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T23:52:08.422 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:08 smithi080 podman[33740]: 2024-04-26 23:52:08.383782669 +0000 UTC m=+0.337380251 container init c49e7b5b507ae5d4f82f40755deac3e62e8d016f32794fef2a0913612f5b9d58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:52:08.422 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:08 smithi080 podman[33740]: 2024-04-26 23:52:08.386346288 +0000 UTC m=+0.339943867 container start c49e7b5b507ae5d4f82f40755deac3e62e8d016f32794fef2a0913612f5b9d58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T23:52:08.422 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:08 smithi080 ceph-mgr[33759]: -- 172.21.15.80:0/3156543249 <== mon.2 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562be4fd45a0 con 0x562be4fa8800 2024-04-26T23:52:08.604 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:08 smithi040 ceph-mon[28095]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:08.703 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:08 smithi103 ceph-mon[32952]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:08.703 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:08 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:08.403+0000 7fc731902200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T23:52:08.703 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:08 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:08.496+0000 7fc731902200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T23:52:08.703 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:08 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:08.699+0000 7fc731902200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T23:52:08.728 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:08 smithi080 ceph-mon[32362]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:08.729 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:08 smithi080 bash[33740]: c49e7b5b507ae5d4f82f40755deac3e62e8d016f32794fef2a0913612f5b9d58 2024-04-26T23:52:08.729 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:08 smithi080 systemd[1]: Started Ceph mgr.y for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:52:08.729 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:08 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:08.539+0000 7f5c582c4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T23:52:08.729 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:08 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:08.725+0000 7f5c582c4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:52:09.052 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:08 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:08.828+0000 7f5c582c4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T23:52:09.100 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:08 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:08.894+0000 7fc731902200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:52:09.536 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:09 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:09.154+0000 7f5c582c4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T23:52:09.536 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:09 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:09.242+0000 7f5c582c4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T23:52:09.582 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T23:52:09.583 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T23:52:09.583 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-26T23:52:09.583 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-26T23:52:09.583 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0337466 s, 311 MB/s 2024-04-26T23:52:09.583 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-26T23:52:09.793 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:09 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:09.794 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:09 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:09.794 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:09 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:09.794 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:09 smithi080 ceph-mon[32362]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:09.794 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:09 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:09.794 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:09 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:09.850 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:09 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:09.528+0000 7fc731902200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T23:52:09.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:09 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:09.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:09 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:09.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:09 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:09.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:09 smithi103 ceph-mon[32952]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:09.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:09 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:09.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:09 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:09.938 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:09 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:09.938 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:09 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:09.938 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:09 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:09.938 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:09 smithi040 ceph-mon[28095]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:09.938 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:09 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:09.938 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:09 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:10.268 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:09 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:09.959+0000 7fc731902200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T23:52:10.269 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:10.059+0000 7fc731902200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T23:52:10.269 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:10.264+0000 7fc731902200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T23:52:10.330 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_4 2024-04-26T23:52:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:52:10.801 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:10.496+0000 7f5c582c4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T23:52:10.802 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:10.664+0000 7f5c582c4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T23:52:10.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:10.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:10.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:10.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:10.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:10.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:10.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:10.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:10.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:52:10.803 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:52:10.803 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:10.939 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:10.939 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:10.939 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:10.939 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:10.939 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:10.939 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:10.939 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:10.939 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:10.939 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:52:10.939 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:52:10.940 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:11.100 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:10.786+0000 7fc731902200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T23:52:11.100 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:10.876+0000 7fc731902200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T23:52:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:11.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:11.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:11.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:11.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:11.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:52:11.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:52:11.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:11.128 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:10 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:10.841+0000 7f5c582c4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T23:52:11.129 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:11 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:11.125+0000 7f5c582c4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T23:52:11.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:11 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:11.217+0000 7f5c582c4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T23:52:11.421 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:11 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:11.417+0000 7f5c582c4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T23:52:11.458 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:11 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:11.116+0000 7fc731902200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T23:52:11.458 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:11 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:11.246+0000 7fc731902200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T23:52:11.459 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:11 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:11.340+0000 7fc731902200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T23:52:11.459 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:11 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:11.453+0000 7fc731902200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T23:52:11.802 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:11 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:11.607+0000 7f5c582c4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:52:12.100 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:11 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:11.753+0000 7fc731902200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T23:52:12.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:12 smithi040 ceph-mon[28095]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T23:52:12.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:12 smithi040 ceph-mon[28095]: Reconfiguring daemon mgr.x on smithi040 2024-04-26T23:52:12.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:12 smithi040 ceph-mon[28095]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:12.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:12 smithi080 ceph-mon[32362]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T23:52:12.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:12 smithi080 ceph-mon[32362]: Reconfiguring daemon mgr.x on smithi040 2024-04-26T23:52:12.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:12 smithi080 ceph-mon[32362]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:12.552 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:12.201+0000 7f5c582c4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T23:52:12.570 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:12 smithi103 ceph-mon[32952]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T23:52:12.570 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:12 smithi103 ceph-mon[32952]: Reconfiguring daemon mgr.x on smithi040 2024-04-26T23:52:12.570 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:12 smithi103 ceph-mon[32952]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:12.571 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:12.351+0000 7fc731902200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T23:52:12.571 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 26 23:52:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z[34600]: 2024-04-26T23:52:12.565+0000 7fc731902200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T23:52:12.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.903+0000 7f7a78000700 1 -- 172.21.15.40:0/3659965317 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a70077dd0 msgr2=0x7f7a700781f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:12.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.903+0000 7f7a78000700 1 --2- 172.21.15.40:0/3659965317 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a70077dd0 0x7f7a700781f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a640098f0 tx=0x7f7a64009c00 comp rx=0 tx=0).stop 2024-04-26T23:52:12.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.903+0000 7f7a78000700 1 -- 172.21.15.40:0/3659965317 shutdown_connections 2024-04-26T23:52:12.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.903+0000 7f7a78000700 1 --2- 172.21.15.40:0/3659965317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a700799e0 0x7f7a7007a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:12.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.903+0000 7f7a78000700 1 --2- 172.21.15.40:0/3659965317 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a70077dd0 0x7f7a700781f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:12.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.903+0000 7f7a78000700 1 --2- 172.21.15.40:0/3659965317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a70079020 0x7f7a700794a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:12.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.903+0000 7f7a78000700 1 -- 172.21.15.40:0/3659965317 >> 172.21.15.40:0/3659965317 conn(0x7f7a70100c70 msgr2=0x7f7a701030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:52:12.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.903+0000 7f7a78000700 1 -- 172.21.15.40:0/3659965317 shutdown_connections 2024-04-26T23:52:12.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.903+0000 7f7a78000700 1 -- 172.21.15.40:0/3659965317 wait complete. 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.904+0000 7f7a78000700 1 Processor -- start 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.905+0000 7f7a78000700 1 -- start start 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.905+0000 7f7a78000700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a70077dd0 0x7f7a7019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.905+0000 7f7a78000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a70079020 0x7f7a701a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.905+0000 7f7a78000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a700799e0 0x7f7a701a71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.905+0000 7f7a78000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7a701155a0 con 0x7f7a700799e0 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.905+0000 7f7a78000700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f7a70115420 con 0x7f7a70077dd0 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.905+0000 7f7a78000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7a70115720 con 0x7f7a70079020 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.905+0000 7f7a74c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a70079020 0x7f7a701a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.905+0000 7f7a76000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a700799e0 0x7f7a701a71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.905+0000 7f7a74c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a70079020 0x7f7a701a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60796/0 (socket says 172.21.15.40:60796) 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.905+0000 7f7a74c00700 1 -- 172.21.15.40:0/2186951605 learned_addr learned my addr 172.21.15.40:0/2186951605 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:52:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.905+0000 7f7a75600700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a70077dd0 0x7f7a7019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:52:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.906+0000 7f7a75600700 1 -- 172.21.15.40:0/2186951605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a70079020 msgr2=0x7f7a701a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.906+0000 7f7a75600700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a70079020 0x7f7a701a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.906+0000 7f7a75600700 1 -- 172.21.15.40:0/2186951605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a700799e0 msgr2=0x7f7a701a71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.906+0000 7f7a75600700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a700799e0 0x7f7a701a71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.906+0000 7f7a75600700 1 -- 172.21.15.40:0/2186951605 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a701a7870 con 0x7f7a70077dd0 2024-04-26T23:52:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.906+0000 7f7a74c00700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a70079020 0x7f7a701a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:52:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.906+0000 7f7a76000700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a700799e0 0x7f7a701a71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:52:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.906+0000 7f7a75600700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a70077dd0 0x7f7a7019fc20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7a64009230 tx=0x7f7a64005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:52:12.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.906+0000 7f7a6e000700 1 -- 172.21.15.40:0/2186951605 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a64005460 con 0x7f7a70077dd0 2024-04-26T23:52:12.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.907+0000 7f7a6e000700 1 -- 172.21.15.40:0/2186951605 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a64005600 con 0x7f7a70077dd0 2024-04-26T23:52:12.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.907+0000 7f7a78000700 1 -- 172.21.15.40:0/2186951605 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a701a7b00 con 0x7f7a70077dd0 2024-04-26T23:52:12.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.907+0000 7f7a6e000700 1 -- 172.21.15.40:0/2186951605 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a64003a70 con 0x7f7a70077dd0 2024-04-26T23:52:12.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.907+0000 7f7a78000700 1 -- 172.21.15.40:0/2186951605 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a701a7fb0 con 0x7f7a70077dd0 2024-04-26T23:52:12.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.908+0000 7f7a6e000700 1 -- 172.21.15.40:0/2186951605 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f7a64003c70 con 0x7f7a70077dd0 2024-04-26T23:52:12.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.908+0000 7f7a78000700 1 -- 172.21.15.40:0/2186951605 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a7006a110 con 0x7f7a70077dd0 2024-04-26T23:52:12.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.908+0000 7f7a6e000700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7a58039890 0x7f7a5803bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:52:12.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.908+0000 7f7a6e000700 1 -- 172.21.15.40:0/2186951605 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7a6404d910 con 0x7f7a70077dd0 2024-04-26T23:52:12.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.909+0000 7f7a74c00700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7a58039890 0x7f7a5803bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:52:12.912 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:12.614+0000 7f5c582c4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T23:52:12.912 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:12.709+0000 7f5c582c4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T23:52:12.913 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:12.908+0000 7f5c582c4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T23:52:12.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.909+0000 7f7a74c00700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7a58039890 0x7f7a5803bd50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c011630 tx=0x7f7a5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:52:12.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:12.914+0000 7f7a6e000700 1 -- 172.21.15.40:0/2186951605 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7a64013030 con 0x7f7a70077dd0 2024-04-26T23:52:13.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:13.164+0000 7f7a78000700 1 -- 172.21.15.40:0/2186951605 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7a70063860 con 0x7f7a58039890 2024-04-26T23:52:13.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:13.195+0000 7f7a6e000700 1 -- 172.21.15.40:0/2186951605 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f7a6402c440 con 0x7f7a70077dd0 2024-04-26T23:52:13.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:13 smithi040 ceph-mon[28095]: from='mgr.? 172.21.15.103:0/2248752081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T23:52:13.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:13 smithi040 ceph-mon[28095]: from='mgr.? 172.21.15.103:0/2248752081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T23:52:13.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:13 smithi040 ceph-mon[28095]: Standby manager daemon z started 2024-04-26T23:52:13.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:13 smithi040 ceph-mon[28095]: from='mgr.? 172.21.15.103:0/2248752081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T23:52:13.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:13 smithi040 ceph-mon[28095]: from='mgr.? 172.21.15.103:0/2248752081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T23:52:13.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:13 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:52:13.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:13 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:52:13.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:13 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:13.518 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:13.425+0000 7f5c582c4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T23:52:13.518 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-mon[32362]: from='mgr.? 172.21.15.103:0/2248752081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T23:52:13.518 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-mon[32362]: from='mgr.? 172.21.15.103:0/2248752081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T23:52:13.518 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-mon[32362]: Standby manager daemon z started 2024-04-26T23:52:13.518 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-mon[32362]: from='mgr.? 172.21.15.103:0/2248752081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T23:52:13.518 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-mon[32362]: from='mgr.? 172.21.15.103:0/2248752081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T23:52:13.519 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:52:13.519 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:52:13.519 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:13.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:13 smithi103 ceph-mon[32952]: from='mgr.? 172.21.15.103:0/2248752081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T23:52:13.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:13 smithi103 ceph-mon[32952]: from='mgr.? 172.21.15.103:0/2248752081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T23:52:13.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:13 smithi103 ceph-mon[32952]: Standby manager daemon z started 2024-04-26T23:52:13.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:13 smithi103 ceph-mon[32952]: from='mgr.? 172.21.15.103:0/2248752081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T23:52:13.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:13 smithi103 ceph-mon[32952]: from='mgr.? 172.21.15.103:0/2248752081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T23:52:13.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:13 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:52:13.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:13 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:52:13.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:13 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:13.802 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:13.514+0000 7f5c582c4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T23:52:13.802 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:13.755+0000 7f5c582c4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T23:52:14.198 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:13.889+0000 7f5c582c4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T23:52:14.198 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:13 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:13.984+0000 7f5c582c4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T23:52:14.198 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:14.097+0000 7f5c582c4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T23:52:14.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:14 smithi040 ceph-mon[28095]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:14.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:14 smithi040 ceph-mon[28095]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:52:14.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:14 smithi040 ceph-mon[28095]: mgrmap e14: x(active, since 88s), standbys: z 2024-04-26T23:52:14.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T23:52:14.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:14.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:14.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:14.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:14.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:14.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:14.552 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:14.393+0000 7f5c582c4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T23:52:14.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-mon[32362]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:14.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-mon[32362]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:52:14.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-mon[32362]: mgrmap e14: x(active, since 88s), standbys: z 2024-04-26T23:52:14.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T23:52:14.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:14.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:14.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:14.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:14.553 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:14.553 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:14.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:14 smithi103 ceph-mon[32952]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:14.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:14 smithi103 ceph-mon[32952]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:52:14.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:14 smithi103 ceph-mon[32952]: mgrmap e14: x(active, since 88s), standbys: z 2024-04-26T23:52:14.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T23:52:14.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:14.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:14.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:14.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:14.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:14.602 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:15.302 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:14 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:14.984+0000 7f5c582c4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T23:52:15.302 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 26 23:52:15 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y[33755]: 2024-04-26T23:52:15.196+0000 7f5c582c4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T23:52:16.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:16.224+0000 7f7a6e000700 1 -- 172.21.15.40:0/2186951605 <== mon.2 v2:172.21.15.80:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f7a6400b8e0 con 0x7f7a70077dd0 2024-04-26T23:52:16.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:16 smithi040 ceph-mon[28095]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:16.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:16 smithi040 ceph-mon[28095]: from='mgr.? 172.21.15.80:0/2516832265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T23:52:16.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:16 smithi040 ceph-mon[28095]: from='mgr.? 172.21.15.80:0/2516832265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T23:52:16.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:16 smithi040 ceph-mon[28095]: from='mgr.? 172.21.15.80:0/2516832265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T23:52:16.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:16 smithi040 ceph-mon[28095]: from='mgr.? 172.21.15.80:0/2516832265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T23:52:16.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:16 smithi040 ceph-mon[28095]: Standby manager daemon y started 2024-04-26T23:52:16.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:16 smithi080 ceph-mon[32362]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:16.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:16 smithi080 ceph-mon[32362]: from='mgr.? 172.21.15.80:0/2516832265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T23:52:16.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:16 smithi080 ceph-mon[32362]: from='mgr.? 172.21.15.80:0/2516832265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T23:52:16.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:16 smithi080 ceph-mon[32362]: from='mgr.? 172.21.15.80:0/2516832265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T23:52:16.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:16 smithi080 ceph-mon[32362]: from='mgr.? 172.21.15.80:0/2516832265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T23:52:16.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:16 smithi080 ceph-mon[32362]: Standby manager daemon y started 2024-04-26T23:52:16.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:16 smithi103 ceph-mon[32952]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:16.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:16 smithi103 ceph-mon[32952]: from='mgr.? 172.21.15.80:0/2516832265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T23:52:16.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:16 smithi103 ceph-mon[32952]: from='mgr.? 172.21.15.80:0/2516832265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T23:52:16.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:16 smithi103 ceph-mon[32952]: from='mgr.? 172.21.15.80:0/2516832265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T23:52:16.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:16 smithi103 ceph-mon[32952]: from='mgr.? 172.21.15.80:0/2516832265' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T23:52:16.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:16 smithi103 ceph-mon[32952]: Standby manager daemon y started 2024-04-26T23:52:17.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:17 smithi040 ceph-mon[28095]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-04-26T23:52:17.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:17 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T23:52:17.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:17 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/599526623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54e9c732-e7a5-45f0-be16-dee18cc70647"}]: dispatch 2024-04-26T23:52:17.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:17 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54e9c732-e7a5-45f0-be16-dee18cc70647"}]: dispatch 2024-04-26T23:52:17.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:17 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54e9c732-e7a5-45f0-be16-dee18cc70647"}]': finished 2024-04-26T23:52:17.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:17 smithi040 ceph-mon[28095]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T23:52:17.555 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:17 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:17.556 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:17 smithi080 ceph-mon[32362]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-04-26T23:52:17.556 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:17 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T23:52:17.556 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:17 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/599526623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54e9c732-e7a5-45f0-be16-dee18cc70647"}]: dispatch 2024-04-26T23:52:17.556 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:17 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54e9c732-e7a5-45f0-be16-dee18cc70647"}]: dispatch 2024-04-26T23:52:17.557 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:17 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54e9c732-e7a5-45f0-be16-dee18cc70647"}]': finished 2024-04-26T23:52:17.557 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:17 smithi080 ceph-mon[32362]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T23:52:17.557 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:17 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:17.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:17 smithi103 ceph-mon[32952]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-04-26T23:52:17.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:17 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T23:52:17.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:17 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/599526623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54e9c732-e7a5-45f0-be16-dee18cc70647"}]: dispatch 2024-04-26T23:52:17.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:17 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54e9c732-e7a5-45f0-be16-dee18cc70647"}]: dispatch 2024-04-26T23:52:17.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:17 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54e9c732-e7a5-45f0-be16-dee18cc70647"}]': finished 2024-04-26T23:52:17.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:17 smithi103 ceph-mon[32952]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T23:52:17.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:17 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:18.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:18 smithi080 ceph-mon[32362]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:18.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:18 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3655435735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:52:18.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:18 smithi103 ceph-mon[32952]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:18.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:18 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3655435735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:52:18.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:18 smithi040 ceph-mon[28095]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:18.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:18 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3655435735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:52:20.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:20 smithi080 ceph-mon[32362]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:20.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:20 smithi103 ceph-mon[32952]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:20.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:20 smithi040 ceph-mon[28095]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:22.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:22 smithi040 ceph-mon[28095]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:22.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:22 smithi080 ceph-mon[32362]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:22.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:22 smithi103 ceph-mon[32952]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:24.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:24 smithi040 ceph-mon[28095]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:24.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:24 smithi080 ceph-mon[32362]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:24.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:24 smithi103 ceph-mon[32952]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:26.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:26 smithi040 ceph-mon[28095]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:26.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:26 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T23:52:26.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:26 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:26.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:26 smithi080 ceph-mon[32362]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:26.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:26 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T23:52:26.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:26 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:26.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:26 smithi103 ceph-mon[32952]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:26.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:26 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T23:52:26.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:26 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:27.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:27 smithi040 ceph-mon[28095]: Deploying daemon osd.0 on smithi040 2024-04-26T23:52:27.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:27 smithi080 ceph-mon[32362]: Deploying daemon osd.0 on smithi040 2024-04-26T23:52:27.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:27 smithi103 ceph-mon[32952]: Deploying daemon osd.0 on smithi040 2024-04-26T23:52:28.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:28 smithi040 ceph-mon[28095]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:28 smithi080 ceph-mon[32362]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:28.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:28 smithi103 ceph-mon[32952]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:30.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:30 smithi040 ceph-mon[28095]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:30.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:30 smithi080 ceph-mon[32362]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:30.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:30 smithi103 ceph-mon[32952]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:32.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:32 smithi080 ceph-mon[32362]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:32.601 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:32 smithi103 ceph-mon[32952]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:32.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:32 smithi040 ceph-mon[28095]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:34.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:34 smithi040 ceph-mon[28095]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:34.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:34 smithi080 ceph-mon[32362]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:34.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:34 smithi103 ceph-mon[32952]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:35.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:35.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:35.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:35.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:35.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:35.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:35.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:35.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:35.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:36.703 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:36 smithi080 ceph-mon[32362]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:36.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:36 smithi040 ceph-mon[28095]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:36.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:36 smithi103 ceph-mon[32952]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:38.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:38 smithi040 ceph-mon[28095]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:38.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:38.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:38.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:38.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:38.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:38.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:38 smithi080 ceph-mon[32362]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:38.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:38.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:38.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:38.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:38.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:38.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:38 smithi103 ceph-mon[32952]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:38.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:38.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:38.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:38.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:38.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:39.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.314+0000 7f7a6e000700 1 -- 172.21.15.40:0/2186951605 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7a70063860 con 0x7f7a58039890 2024-04-26T23:52:39.319 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 0 on host 'smithi040' 2024-04-26T23:52:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.318+0000 7f7a78000700 1 -- 172.21.15.40:0/2186951605 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7a58039890 msgr2=0x7f7a5803bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.318+0000 7f7a78000700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7a58039890 0x7f7a5803bd50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c011630 tx=0x7f7a5c009260 comp rx=0 tx=0).stop 2024-04-26T23:52:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.318+0000 7f7a78000700 1 -- 172.21.15.40:0/2186951605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a70077dd0 msgr2=0x7f7a7019fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:39.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.318+0000 7f7a78000700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a70077dd0 0x7f7a7019fc20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7a64009230 tx=0x7f7a64005270 comp rx=0 tx=0).stop 2024-04-26T23:52:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.319+0000 7f7a78000700 1 -- 172.21.15.40:0/2186951605 shutdown_connections 2024-04-26T23:52:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.319+0000 7f7a78000700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7a58039890 0x7f7a5803bd50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.319+0000 7f7a78000700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a700799e0 0x7f7a701a71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.319+0000 7f7a78000700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7a70077dd0 0x7f7a7019fc20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.319+0000 7f7a78000700 1 --2- 172.21.15.40:0/2186951605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a70079020 0x7f7a701a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.319+0000 7f7a78000700 1 -- 172.21.15.40:0/2186951605 >> 172.21.15.40:0/2186951605 conn(0x7f7a70100c70 msgr2=0x7f7a70102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:52:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.319+0000 7f7a78000700 1 -- 172.21.15.40:0/2186951605 shutdown_connections 2024-04-26T23:52:39.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:52:39.319+0000 7f7a78000700 1 -- 172.21.15.40:0/2186951605 wait complete. 2024-04-26T23:52:39.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:39 smithi040 ceph-mon[28095]: from='osd.0 [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T23:52:39.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:39 smithi040 ceph-mon[28095]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T23:52:39.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:39 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:39.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:39 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:39.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:39 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:39.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:39 smithi080 ceph-mon[32362]: from='osd.0 [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T23:52:39.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:39 smithi080 ceph-mon[32362]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T23:52:39.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:39 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:39.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:39 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:39.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:39 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:39.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:39 smithi103 ceph-mon[32952]: from='osd.0 [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T23:52:39.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:39 smithi103 ceph-mon[32952]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T23:52:39.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:39.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:39.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:40.140 DEBUG:teuthology.orchestra.run.smithi040:osd.0> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T23:52:40.145 INFO:tasks.cephadm:Deploying osd.1 on smithi080 with /dev/vg_nvme/lv_4... 2024-04-26T23:52:40.145 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T23:52:40.386 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:52:40.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:40 smithi040 ceph-mon[28095]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:40.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:40 smithi040 ceph-mon[28095]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T23:52:40.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:40 smithi040 ceph-mon[28095]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T23:52:40.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:40 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:40.667 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:40 smithi040 ceph-mon[28095]: from='osd.0 [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:52:40.667 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:40 smithi040 ceph-mon[28095]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:52:40.791 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:40 smithi080 ceph-mon[32362]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:40.791 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:40 smithi080 ceph-mon[32362]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T23:52:40.791 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:40 smithi080 ceph-mon[32362]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T23:52:40.791 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:40 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:40.791 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:40 smithi080 ceph-mon[32362]: from='osd.0 [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:52:40.792 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:40 smithi080 ceph-mon[32362]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:52:40.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:40 smithi103 ceph-mon[32952]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:40.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:40 smithi103 ceph-mon[32952]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T23:52:40.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:40 smithi103 ceph-mon[32952]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T23:52:40.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:40 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:40.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:40 smithi103 ceph-mon[32952]: from='osd.0 [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:52:40.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:40 smithi103 ceph-mon[32952]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:52:41.469 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 26 23:52:41 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0[40450]: 2024-04-26T23:52:41.311+0000 7f7529c00700 -1 osd.0 0 waiting for initial osdmap 2024-04-26T23:52:41.470 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 26 23:52:41 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0[40450]: 2024-04-26T23:52:41.340+0000 7f751fa00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:52:41.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:41 smithi080 ceph-mon[32362]: purged_snaps scrub starts 2024-04-26T23:52:41.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:41 smithi080 ceph-mon[32362]: purged_snaps scrub ok 2024-04-26T23:52:41.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:41 smithi080 ceph-mon[32362]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-26T23:52:41.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:41 smithi080 ceph-mon[32362]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T23:52:41.715 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:41 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:41.715 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:41 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:41.715 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:41 smithi080 ceph-mon[32362]: from='osd.0 ' entity='osd.0' 2024-04-26T23:52:41.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:41 smithi103 ceph-mon[32952]: purged_snaps scrub starts 2024-04-26T23:52:41.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:41 smithi103 ceph-mon[32952]: purged_snaps scrub ok 2024-04-26T23:52:41.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:41 smithi103 ceph-mon[32952]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-26T23:52:41.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:41 smithi103 ceph-mon[32952]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T23:52:41.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:41 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:41.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:41 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:41.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:41 smithi103 ceph-mon[32952]: from='osd.0 ' entity='osd.0' 2024-04-26T23:52:41.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:41 smithi040 ceph-mon[28095]: purged_snaps scrub starts 2024-04-26T23:52:41.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:41 smithi040 ceph-mon[28095]: purged_snaps scrub ok 2024-04-26T23:52:41.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:41 smithi040 ceph-mon[28095]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-26T23:52:41.856 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:41 smithi040 ceph-mon[28095]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T23:52:41.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:41 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:41.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:41 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:41.857 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:41 smithi040 ceph-mon[28095]: from='osd.0 ' entity='osd.0' 2024-04-26T23:52:42.261 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T23:52:42.261 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T23:52:42.261 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-04-26T23:52:42.261 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-04-26T23:52:42.261 INFO:teuthology.orchestra.run.smithi080.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113534 s, 924 MB/s 2024-04-26T23:52:42.261 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-04-26T23:52:42.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:42 smithi080 ceph-mon[32362]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:42.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:42 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:42.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:42 smithi080 ceph-mon[32362]: osd.0 [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598] boot 2024-04-26T23:52:42.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:42 smithi080 ceph-mon[32362]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T23:52:42.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:42 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:42.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:42 smithi103 ceph-mon[32952]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:42.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:42 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:42.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:42 smithi103 ceph-mon[32952]: osd.0 [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598] boot 2024-04-26T23:52:42.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:42 smithi103 ceph-mon[32952]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T23:52:42.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:42 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:42.886 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi080:vg_nvme/lv_4 2024-04-26T23:52:42.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:42 smithi040 ceph-mon[28095]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:52:42.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:42 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:42.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:42 smithi040 ceph-mon[28095]: osd.0 [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598] boot 2024-04-26T23:52:42.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:42 smithi040 ceph-mon[28095]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T23:52:42.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:42 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:52:43.117 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:52:44.530 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.524+0000 7f2c511f8700 1 -- 172.21.15.80:0/2005587111 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c4c1036b0 msgr2=0x7f2c4c109f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:44.530 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.524+0000 7f2c511f8700 1 --2- 172.21.15.80:0/2005587111 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c4c1036b0 0x7f2c4c109f60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2c4400b0d0 tx=0x7f2c4400b3e0 comp rx=0 tx=0).stop 2024-04-26T23:52:44.530 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.524+0000 7f2c511f8700 1 -- 172.21.15.80:0/2005587111 shutdown_connections 2024-04-26T23:52:44.530 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.524+0000 7f2c511f8700 1 --2- 172.21.15.80:0/2005587111 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c4c1036b0 0x7f2c4c109f60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:44.530 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.524+0000 7f2c511f8700 1 --2- 172.21.15.80:0/2005587111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c4c101b10 0x7f2c4c101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:44.530 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.524+0000 7f2c511f8700 1 --2- 172.21.15.80:0/2005587111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c4c102d10 0x7f2c4c103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:44.530 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.524+0000 7f2c511f8700 1 -- 172.21.15.80:0/2005587111 >> 172.21.15.80:0/2005587111 conn(0x7f2c4c0fd090 msgr2=0x7f2c4c0ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:52:44.531 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.524+0000 7f2c511f8700 1 -- 172.21.15.80:0/2005587111 shutdown_connections 2024-04-26T23:52:44.531 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.524+0000 7f2c511f8700 1 -- 172.21.15.80:0/2005587111 wait complete. 2024-04-26T23:52:44.531 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.525+0000 7f2c511f8700 1 Processor -- start 2024-04-26T23:52:44.531 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.525+0000 7f2c511f8700 1 -- start start 2024-04-26T23:52:44.531 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.525+0000 7f2c511f8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c4c101b10 0x7f2c4c10b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:52:44.531 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.525+0000 7f2c511f8700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c4c102d10 0x7f2c4c19e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:52:44.531 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c511f8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c4c1036b0 0x7f2c4c1a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:52:44.531 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c511f8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c10df20 con 0x7f2c4c101b10 2024-04-26T23:52:44.531 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c511f8700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c10dda0 con 0x7f2c4c102d10 2024-04-26T23:52:44.531 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c511f8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c10e0a0 con 0x7f2c4c1036b0 2024-04-26T23:52:44.531 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c4a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c4c102d10 0x7f2c4c19e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:52:44.531 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c4a59c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c4c102d10 0x7f2c4c19e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:58966/0 (socket says 172.21.15.80:58966) 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c4a59c700 1 -- 172.21.15.80:0/2553897535 learned_addr learned my addr 172.21.15.80:0/2553897535 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c4b59e700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c4c1036b0 0x7f2c4c1a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c4a59c700 1 -- 172.21.15.80:0/2553897535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c4c1036b0 msgr2=0x7f2c4c1a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c4a59c700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c4c1036b0 0x7f2c4c1a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c4a59c700 1 -- 172.21.15.80:0/2553897535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c4c101b10 msgr2=0x7f2c4c10b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c4a59c700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c4c101b10 0x7f2c4c10b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c4a59c700 1 -- 172.21.15.80:0/2553897535 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4c1a12d0 con 0x7f2c4c102d10 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c4b59e700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c4c1036b0 0x7f2c4c1a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c4a59c700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c4c102d10 0x7f2c4c19e3c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000b530 tx=0x7f2c4000b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c33fff700 1 -- 172.21.15.80:0/2553897535 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40015020 con 0x7f2c4c102d10 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c511f8700 1 -- 172.21.15.80:0/2553897535 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c4c1a15c0 con 0x7f2c4c102d10 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c33fff700 1 -- 172.21.15.80:0/2553897535 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c40011ce0 con 0x7f2c4c102d10 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c33fff700 1 -- 172.21.15.80:0/2553897535 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40010d20 con 0x7f2c4c102d10 2024-04-26T23:52:44.532 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.526+0000 7f2c511f8700 1 -- 172.21.15.80:0/2553897535 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c4c1a1b80 con 0x7f2c4c102d10 2024-04-26T23:52:44.533 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.527+0000 7f2c511f8700 1 -- 172.21.15.80:0/2553897535 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c4c06a110 con 0x7f2c4c102d10 2024-04-26T23:52:44.533 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.528+0000 7f2c33fff700 1 -- 172.21.15.80:0/2553897535 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f2c40018020 con 0x7f2c4c102d10 2024-04-26T23:52:44.533 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.529+0000 7f2c33fff700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2c340a3bb0 0x7f2c340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:52:44.534 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.529+0000 7f2c33fff700 1 -- 172.21.15.80:0/2553897535 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f2c400ce700 con 0x7f2c4c102d10 2024-04-26T23:52:44.534 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.529+0000 7f2c4ad9d700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2c340a3bb0 0x7f2c340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:52:44.534 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.530+0000 7f2c4ad9d700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2c340a3bb0 0x7f2c340a6070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c102b70 tx=0x7f2c3c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:52:44.536 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.533+0000 7f2c33fff700 1 -- 172.21.15.80:0/2553897535 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2c4001f020 con 0x7f2c4c102d10 2024-04-26T23:52:44.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:44 smithi103 ceph-mon[32952]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:44.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:44 smithi103 ceph-mon[32952]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T23:52:44.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:44 smithi040 ceph-mon[28095]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:44.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:44 smithi040 ceph-mon[28095]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T23:52:44.740 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:52:44.737+0000 7f2c511f8700 1 -- 172.21.15.80:0/2553897535 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2c4c063860 con 0x7f2c340a3bb0 2024-04-26T23:52:44.740 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:44 smithi080 ceph-mon[32362]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:44.740 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:44 smithi080 ceph-mon[32362]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T23:52:45.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:52:45.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:52:45.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:45.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:45.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:45.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:45.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:45.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:45.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:45.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:52:45.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:52:45.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:45.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:45.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:45.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:45.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:45.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:45.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:45.653 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:52:45.653 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:52:45.654 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:45.654 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:45.654 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:45.654 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:45.654 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:45.654 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:45.654 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:46.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:46 smithi080 ceph-mon[32362]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:46.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:46 smithi080 ceph-mon[32362]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:52:46.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:46 smithi080 ceph-mon[32362]: Detected new or changed devices on smithi040 2024-04-26T23:52:46.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:46 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi040 to 17094M 2024-04-26T23:52:46.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:46.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:46 smithi103 ceph-mon[32952]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:46.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:46 smithi103 ceph-mon[32952]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:52:46.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:46 smithi103 ceph-mon[32952]: Detected new or changed devices on smithi040 2024-04-26T23:52:46.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:46 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi040 to 17094M 2024-04-26T23:52:46.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:46.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:46 smithi040 ceph-mon[28095]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:46.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:46 smithi040 ceph-mon[28095]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:52:46.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:46 smithi040 ceph-mon[28095]: Detected new or changed devices on smithi040 2024-04-26T23:52:46.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:46 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi040 to 17094M 2024-04-26T23:52:46.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:52:48.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:48 smithi040 ceph-mon[28095]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:48.514 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:48 smithi040 ceph-mon[28095]: from='client.? 172.21.15.80:0/761644368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee040674-8f99-4406-a7bf-a989815dce57"}]: dispatch 2024-04-26T23:52:48.514 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:48 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee040674-8f99-4406-a7bf-a989815dce57"}]: dispatch 2024-04-26T23:52:48.514 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:48 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee040674-8f99-4406-a7bf-a989815dce57"}]': finished 2024-04-26T23:52:48.514 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:48 smithi040 ceph-mon[28095]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T23:52:48.514 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:52:48.514 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:48 smithi040 ceph-mon[28095]: from='client.? 172.21.15.80:0/1487962301' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:52:48.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:48 smithi103 ceph-mon[32952]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:48.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:48 smithi103 ceph-mon[32952]: from='client.? 172.21.15.80:0/761644368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee040674-8f99-4406-a7bf-a989815dce57"}]: dispatch 2024-04-26T23:52:48.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:48 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee040674-8f99-4406-a7bf-a989815dce57"}]: dispatch 2024-04-26T23:52:48.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:48 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee040674-8f99-4406-a7bf-a989815dce57"}]': finished 2024-04-26T23:52:48.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:48 smithi103 ceph-mon[32952]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T23:52:48.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:52:48.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:48 smithi103 ceph-mon[32952]: from='client.? 172.21.15.80:0/1487962301' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:52:48.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:48 smithi080 ceph-mon[32362]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:48.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:48 smithi080 ceph-mon[32362]: from='client.? 172.21.15.80:0/761644368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee040674-8f99-4406-a7bf-a989815dce57"}]: dispatch 2024-04-26T23:52:48.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:48 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee040674-8f99-4406-a7bf-a989815dce57"}]: dispatch 2024-04-26T23:52:48.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:48 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee040674-8f99-4406-a7bf-a989815dce57"}]': finished 2024-04-26T23:52:48.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:48 smithi080 ceph-mon[32362]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T23:52:48.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:52:48.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:48 smithi080 ceph-mon[32362]: from='client.? 172.21.15.80:0/1487962301' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:52:50.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:50 smithi103 ceph-mon[32952]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:50.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:50 smithi040 ceph-mon[28095]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:50.733 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:50 smithi080 ceph-mon[32362]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:52.675 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:52 smithi080 ceph-mon[32362]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:52.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:52 smithi040 ceph-mon[28095]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:52.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:52 smithi103 ceph-mon[32952]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:54.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:54 smithi040 ceph-mon[28095]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:54.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:54 smithi080 ceph-mon[32362]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:54.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:54 smithi103 ceph-mon[32952]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:55.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T23:52:55.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:55.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T23:52:55.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:55.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T23:52:55.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:56.496 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:56 smithi080 ceph-mon[32362]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:56.496 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:56 smithi080 ceph-mon[32362]: Deploying daemon osd.1 on smithi080 2024-04-26T23:52:56.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:56 smithi040 ceph-mon[28095]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:56.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:56 smithi040 ceph-mon[28095]: Deploying daemon osd.1 on smithi080 2024-04-26T23:52:56.751 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:56 smithi103 ceph-mon[32952]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:56.751 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:56 smithi103 ceph-mon[32952]: Deploying daemon osd.1 on smithi080 2024-04-26T23:52:58.451 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:52:58 smithi080 ceph-mon[32362]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:58.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:52:58 smithi040 ceph-mon[28095]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:52:58.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:52:58 smithi103 ceph-mon[32952]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:00.716 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:00 smithi080 ceph-mon[32362]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:00.716 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:00.716 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:00.717 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:00.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:00 smithi040 ceph-mon[28095]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:00.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:00.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:00.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:00.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:00 smithi103 ceph-mon[32952]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:00.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:00.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:00.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:02.652 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:02 smithi080 ceph-mon[32362]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:02.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:02 smithi040 ceph-mon[28095]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:02.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:02 smithi103 ceph-mon[32952]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:03.657 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:03.657 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:03.658 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:03.658 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:03.658 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:03.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:03.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:03.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:03.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:03.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:03.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.740+0000 7f2c33fff700 1 -- 172.21.15.80:0/2553897535 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2c4c063860 con 0x7f2c340a3bb0 2024-04-26T23:53:03.743 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 1 on host 'smithi080' 2024-04-26T23:53:03.745 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.742+0000 7f2c511f8700 1 -- 172.21.15.80:0/2553897535 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2c340a3bb0 msgr2=0x7f2c340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:53:03.745 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.742+0000 7f2c511f8700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2c340a3bb0 0x7f2c340a6070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c102b70 tx=0x7f2c3c006d20 comp rx=0 tx=0).stop 2024-04-26T23:53:03.745 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.742+0000 7f2c511f8700 1 -- 172.21.15.80:0/2553897535 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c4c102d10 msgr2=0x7f2c4c19e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:53:03.746 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.742+0000 7f2c511f8700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c4c102d10 0x7f2c4c19e3c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000b530 tx=0x7f2c4000b8f0 comp rx=0 tx=0).stop 2024-04-26T23:53:03.746 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.742+0000 7f2c511f8700 1 -- 172.21.15.80:0/2553897535 shutdown_connections 2024-04-26T23:53:03.746 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.742+0000 7f2c511f8700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2c340a3bb0 0x7f2c340a6070 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:03.746 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.742+0000 7f2c511f8700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c4c1036b0 0x7f2c4c1a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:03.746 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.742+0000 7f2c511f8700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c4c101b10 0x7f2c4c10b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:03.746 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.742+0000 7f2c511f8700 1 --2- 172.21.15.80:0/2553897535 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c4c102d10 0x7f2c4c19e3c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:03.746 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.742+0000 7f2c511f8700 1 -- 172.21.15.80:0/2553897535 >> 172.21.15.80:0/2553897535 conn(0x7f2c4c0fd090 msgr2=0x7f2c4c106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:53:03.746 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.742+0000 7f2c511f8700 1 -- 172.21.15.80:0/2553897535 shutdown_connections 2024-04-26T23:53:03.746 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:53:03.742+0000 7f2c511f8700 1 -- 172.21.15.80:0/2553897535 wait complete. 2024-04-26T23:53:03.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:03.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:03.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:03.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:03.851 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:04.428 DEBUG:teuthology.orchestra.run.smithi080:osd.1> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T23:53:04.431 INFO:tasks.cephadm:Deploying osd.2 on smithi103 with /dev/vg_nvme/lv_4... 2024-04-26T23:53:04.431 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T23:53:04.661 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:04 smithi080 ceph-mon[32362]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:04.661 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:04.661 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:04.661 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:04.661 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:04 smithi080 ceph-mon[32362]: from='osd.1 [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T23:53:04.661 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:04 smithi080 ceph-mon[32362]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T23:53:04.672 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:53:04.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:04 smithi040 ceph-mon[28095]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:04.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:04.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:04.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:04.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:04 smithi040 ceph-mon[28095]: from='osd.1 [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T23:53:04.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:04 smithi040 ceph-mon[28095]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T23:53:04.796 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:04 smithi103 ceph-mon[32952]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:04.796 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:04.796 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:04.796 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:04.796 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:04 smithi103 ceph-mon[32952]: from='osd.1 [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T23:53:04.796 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:04 smithi103 ceph-mon[32952]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T23:53:05.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:05 smithi103 ceph-mon[32952]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:05.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:05 smithi103 ceph-mon[32952]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T23:53:05.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:05 smithi103 ceph-mon[32952]: from='osd.1 [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:53:05.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:05 smithi103 ceph-mon[32952]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T23:53:05.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:05 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:53:05.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:05 smithi103 ceph-mon[32952]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:53:05.945 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:05 smithi080 ceph-mon[32362]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:05.945 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:05 smithi080 ceph-mon[32362]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T23:53:05.946 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:05 smithi080 ceph-mon[32362]: from='osd.1 [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:53:05.946 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:05 smithi080 ceph-mon[32362]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T23:53:05.946 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:05 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:53:05.946 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:05 smithi080 ceph-mon[32362]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:53:05.946 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 26 23:53:05 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1[36955]: 2024-04-26T23:53:05.713+0000 7f00e6a13700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T23:53:05.946 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 26 23:53:05 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1[36955]: 2024-04-26T23:53:05.716+0000 7f00e2806700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:53:06.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:05 smithi040 ceph-mon[28095]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:06.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:05 smithi040 ceph-mon[28095]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T23:53:06.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:05 smithi040 ceph-mon[28095]: from='osd.1 [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:53:06.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:05 smithi040 ceph-mon[28095]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T23:53:06.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:05 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:53:06.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:05 smithi040 ceph-mon[28095]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:53:06.579 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T23:53:06.579 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T23:53:06.579 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-04-26T23:53:06.579 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-04-26T23:53:06.579 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114019 s, 920 MB/s 2024-04-26T23:53:06.580 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-04-26T23:53:06.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:06 smithi103 ceph-mon[32952]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-04-26T23:53:06.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:06 smithi103 ceph-mon[32952]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T23:53:06.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:06 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:53:06.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:06 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:53:07.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:06 smithi080 ceph-mon[32362]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-04-26T23:53:07.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:06 smithi080 ceph-mon[32362]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T23:53:07.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:06 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:53:07.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:06 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:53:07.124 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_4 2024-04-26T23:53:07.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:06 smithi040 ceph-mon[28095]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-04-26T23:53:07.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:06 smithi040 ceph-mon[28095]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T23:53:07.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:53:07.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:53:07.376 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:53:08.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: purged_snaps scrub starts 2024-04-26T23:53:08.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: purged_snaps scrub ok 2024-04-26T23:53:08.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: pgmap v73: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:08.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: osd.1 [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093] boot 2024-04-26T23:53:08.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T23:53:08.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:53:08.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: Detected new or changed devices on smithi080 2024-04-26T23:53:08.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:08.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:08.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:08.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:08.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:08.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:08.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:07 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:08.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: purged_snaps scrub starts 2024-04-26T23:53:08.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: purged_snaps scrub ok 2024-04-26T23:53:08.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: pgmap v73: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:08.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: osd.1 [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093] boot 2024-04-26T23:53:08.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T23:53:08.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:53:08.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: Detected new or changed devices on smithi080 2024-04-26T23:53:08.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:08.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:08.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:08.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:08.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:08.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:08.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:07 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:08.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: purged_snaps scrub starts 2024-04-26T23:53:08.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: purged_snaps scrub ok 2024-04-26T23:53:08.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: pgmap v73: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:53:08.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: osd.1 [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093] boot 2024-04-26T23:53:08.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T23:53:08.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:53:08.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: Detected new or changed devices on smithi080 2024-04-26T23:53:08.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:08.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:08.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:08.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:08.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:08.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:08.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:07 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:08.811 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:08 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi080 to 17088M 2024-04-26T23:53:08.812 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:08 smithi103 ceph-mon[32952]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T23:53:08.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.808+0000 7fef3b09e700 1 -- 172.21.15.103:0/2042881440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef34101730 msgr2=0x7fef34101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:53:08.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.808+0000 7fef3b09e700 1 --2- 172.21.15.103:0/2042881440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef34101730 0x7fef34101b50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00b0a0 tx=0x7fef2c00b3b0 comp rx=0 tx=0).stop 2024-04-26T23:53:08.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.808+0000 7fef3b09e700 1 -- 172.21.15.103:0/2042881440 shutdown_connections 2024-04-26T23:53:08.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.808+0000 7fef3b09e700 1 --2- 172.21.15.103:0/2042881440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef341050b0 0x7fef3410b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:08.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.808+0000 7fef3b09e700 1 --2- 172.21.15.103:0/2042881440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef34101730 0x7fef34101b50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:08.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.808+0000 7fef3b09e700 1 --2- 172.21.15.103:0/2042881440 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef34104710 0x7fef34104b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:08.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.808+0000 7fef3b09e700 1 -- 172.21.15.103:0/2042881440 >> 172.21.15.103:0/2042881440 conn(0x7fef340fd0b0 msgr2=0x7fef340ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:53:08.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.809+0000 7fef3b09e700 1 -- 172.21.15.103:0/2042881440 shutdown_connections 2024-04-26T23:53:08.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.809+0000 7fef3b09e700 1 -- 172.21.15.103:0/2042881440 wait complete. 2024-04-26T23:53:08.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.809+0000 7fef3b09e700 1 Processor -- start 2024-04-26T23:53:08.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.809+0000 7fef3b09e700 1 -- start start 2024-04-26T23:53:08.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.809+0000 7fef3b09e700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef34101730 0x7fef34199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:53:08.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.809+0000 7fef3b09e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef34104710 0x7fef34199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:53:08.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.809+0000 7fef3b09e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef341050b0 0x7fef341a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:53:08.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.809+0000 7fef3b09e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef3410ef80 con 0x7fef34104710 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.809+0000 7fef3b09e700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fef3410ee00 con 0x7fef34101730 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.809+0000 7fef3b09e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef3410f100 con 0x7fef341050b0 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef3963b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef341050b0 0x7fef341a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef3963b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef341050b0 0x7fef341a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52578/0 (socket says 172.21.15.103:52578) 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef3963b700 1 -- 172.21.15.103:0/2408188641 learned_addr learned my addr 172.21.15.103:0/2408188641 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef38e3a700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef34101730 0x7fef34199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef3963b700 1 -- 172.21.15.103:0/2408188641 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef34101730 msgr2=0x7fef34199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef3963b700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef34101730 0x7fef34199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef3963b700 1 -- 172.21.15.103:0/2408188641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef34104710 msgr2=0x7fef34199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef3963b700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef34104710 0x7fef34199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef3963b700 1 -- 172.21.15.103:0/2408188641 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef341a1290 con 0x7fef341050b0 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef3963b700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef341050b0 0x7fef341a0bf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00b580 tx=0x7fef1c00b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef31ffb700 1 -- 172.21.15.103:0/2408188641 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c015020 con 0x7fef341050b0 2024-04-26T23:53:08.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef3b09e700 1 -- 172.21.15.103:0/2408188641 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef341a1580 con 0x7fef341050b0 2024-04-26T23:53:08.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef31ffb700 1 -- 172.21.15.103:0/2408188641 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef1c011ce0 con 0x7fef341050b0 2024-04-26T23:53:08.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.810+0000 7fef31ffb700 1 -- 172.21.15.103:0/2408188641 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c010dd0 con 0x7fef341050b0 2024-04-26T23:53:08.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.811+0000 7fef3b09e700 1 -- 172.21.15.103:0/2408188641 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef341a1af0 con 0x7fef341050b0 2024-04-26T23:53:08.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.811+0000 7fef3b09e700 1 -- 172.21.15.103:0/2408188641 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef3406a110 con 0x7fef341050b0 2024-04-26T23:53:08.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.813+0000 7fef31ffb700 1 -- 172.21.15.103:0/2408188641 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fef1c018020 con 0x7fef341050b0 2024-04-26T23:53:08.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.814+0000 7fef31ffb700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fef200a3bb0 0x7fef200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:53:08.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.814+0000 7fef31ffb700 1 -- 172.21.15.103:0/2408188641 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fef1c0ce2c0 con 0x7fef341050b0 2024-04-26T23:53:08.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.814+0000 7fef38e3a700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fef200a3bb0 0x7fef200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:53:08.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.815+0000 7fef38e3a700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fef200a3bb0 0x7fef200a6070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fef2c009fd0 tx=0x7fef2c00b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:53:08.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:08.819+0000 7fef31ffb700 1 -- 172.21.15.103:0/2408188641 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fef1c01f020 con 0x7fef341050b0 2024-04-26T23:53:09.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:09.025+0000 7fef3b09e700 1 -- 172.21.15.103:0/2408188641 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fef34063860 con 0x7fef200a3bb0 2024-04-26T23:53:09.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:08 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi080 to 17088M 2024-04-26T23:53:09.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:08 smithi080 ceph-mon[32362]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T23:53:09.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:08 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi080 to 17088M 2024-04-26T23:53:09.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:08 smithi040 ceph-mon[28095]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T23:53:09.935 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:09 smithi103 ceph-mon[32952]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:09.935 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:09 smithi103 ceph-mon[32952]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:53:09.935 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:09 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:53:09.935 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:09 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:53:09.935 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:09 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:10.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:09 smithi080 ceph-mon[32362]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:10.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:09 smithi080 ceph-mon[32362]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:53:10.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:09 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:53:10.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:09 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:53:10.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:09 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:10.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:09 smithi040 ceph-mon[28095]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:10.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:09 smithi040 ceph-mon[28095]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:53:10.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:09 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:53:10.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:09 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:53:10.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:09 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:12.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:11 smithi080 ceph-mon[32362]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:12.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:11 smithi080 ceph-mon[32362]: from='client.? 172.21.15.103:0/872863709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219"}]: dispatch 2024-04-26T23:53:12.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:11 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219"}]: dispatch 2024-04-26T23:53:12.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:11 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219"}]': finished 2024-04-26T23:53:12.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:11 smithi080 ceph-mon[32362]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T23:53:12.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:11 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:12.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:11 smithi103 ceph-mon[32952]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:12.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:11 smithi103 ceph-mon[32952]: from='client.? 172.21.15.103:0/872863709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219"}]: dispatch 2024-04-26T23:53:12.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:11 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219"}]: dispatch 2024-04-26T23:53:12.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:11 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219"}]': finished 2024-04-26T23:53:12.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:11 smithi103 ceph-mon[32952]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T23:53:12.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:11 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:12.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:11 smithi040 ceph-mon[28095]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:12.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:11 smithi040 ceph-mon[28095]: from='client.? 172.21.15.103:0/872863709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219"}]: dispatch 2024-04-26T23:53:12.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:11 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219"}]: dispatch 2024-04-26T23:53:12.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:11 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219"}]': finished 2024-04-26T23:53:12.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:11 smithi040 ceph-mon[28095]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T23:53:12.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:11 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:13.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:12 smithi080 ceph-mon[32362]: from='client.? 172.21.15.103:0/4277697279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:53:13.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:12 smithi103 ceph-mon[32952]: from='client.? 172.21.15.103:0/4277697279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:53:13.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:12 smithi040 ceph-mon[28095]: from='client.? 172.21.15.103:0/4277697279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:53:14.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:13 smithi080 ceph-mon[32362]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:14.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:13 smithi103 ceph-mon[32952]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:14.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:13 smithi040 ceph-mon[28095]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:15.822 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:15 smithi103 ceph-mon[32952]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:16.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:15 smithi080 ceph-mon[32362]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:16.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:15 smithi040 ceph-mon[28095]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:17.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:17 smithi103 ceph-mon[32952]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:18.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:17 smithi080 ceph-mon[32362]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:18.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:17 smithi040 ceph-mon[28095]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:19.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:18 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T23:53:19.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:18 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:19.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:18 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T23:53:19.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:18 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:19.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:18 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T23:53:19.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:18 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:19.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:19 smithi103 ceph-mon[32952]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:19.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:19 smithi103 ceph-mon[32952]: Deploying daemon osd.2 on smithi103 2024-04-26T23:53:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:19 smithi040 ceph-mon[28095]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:19 smithi040 ceph-mon[28095]: Deploying daemon osd.2 on smithi103 2024-04-26T23:53:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:19 smithi080 ceph-mon[32362]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:19 smithi080 ceph-mon[32362]: Deploying daemon osd.2 on smithi103 2024-04-26T23:53:22.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:21 smithi103 ceph-mon[32952]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:22.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:21 smithi040 ceph-mon[28095]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:22.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:21 smithi080 ceph-mon[32362]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:23 smithi103 ceph-mon[32952]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:23 smithi040 ceph-mon[28095]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:23 smithi080 ceph-mon[32362]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:25.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:25.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:25.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:25.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:25.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:25.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:25.352 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:25.352 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:25.352 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:26.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:25 smithi103 ceph-mon[32952]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:26.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:25 smithi080 ceph-mon[32362]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:26.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:25 smithi040 ceph-mon[28095]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:27.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.259+0000 7fef31ffb700 1 -- 172.21.15.103:0/2408188641 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fef34063860 con 0x7fef200a3bb0 2024-04-26T23:53:27.262 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 2 on host 'smithi103' 2024-04-26T23:53:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.261+0000 7fef3b09e700 1 -- 172.21.15.103:0/2408188641 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fef200a3bb0 msgr2=0x7fef200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:53:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.261+0000 7fef3b09e700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fef200a3bb0 0x7fef200a6070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fef2c009fd0 tx=0x7fef2c00b7b0 comp rx=0 tx=0).stop 2024-04-26T23:53:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.261+0000 7fef3b09e700 1 -- 172.21.15.103:0/2408188641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef341050b0 msgr2=0x7fef341a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:53:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.261+0000 7fef3b09e700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef341050b0 0x7fef341a0bf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00b580 tx=0x7fef1c00b890 comp rx=0 tx=0).stop 2024-04-26T23:53:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.261+0000 7fef3b09e700 1 -- 172.21.15.103:0/2408188641 shutdown_connections 2024-04-26T23:53:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.261+0000 7fef3b09e700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fef200a3bb0 0x7fef200a6070 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.261+0000 7fef3b09e700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef341050b0 0x7fef341a0bf0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.261+0000 7fef3b09e700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef34101730 0x7fef34199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.261+0000 7fef3b09e700 1 --2- 172.21.15.103:0/2408188641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef34104710 0x7fef34199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.261+0000 7fef3b09e700 1 -- 172.21.15.103:0/2408188641 >> 172.21.15.103:0/2408188641 conn(0x7fef340fd0b0 msgr2=0x7fef34107b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:53:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.261+0000 7fef3b09e700 1 -- 172.21.15.103:0/2408188641 shutdown_connections 2024-04-26T23:53:27.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:53:27.261+0000 7fef3b09e700 1 -- 172.21.15.103:0/2408188641 wait complete. 2024-04-26T23:53:27.889 DEBUG:teuthology.orchestra.run.smithi103:osd.2> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T23:53:27.891 INFO:tasks.cephadm:Deploying osd.3 on smithi040 with /dev/vg_nvme/lv_3... 2024-04-26T23:53:27.891 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T23:53:28.352 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:53:28.504 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:28 smithi040 ceph-mon[28095]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:28.504 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:28.504 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:28.504 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:28 smithi040 ceph-mon[28095]: from='osd.2 [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T23:53:28.504 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:28 smithi040 ceph-mon[28095]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T23:53:28.504 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:28.504 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:28 smithi080 ceph-mon[32362]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:28.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:28.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:28 smithi080 ceph-mon[32362]: from='osd.2 [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T23:53:28.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:28 smithi080 ceph-mon[32362]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T23:53:28.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:28.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:28.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:28 smithi103 ceph-mon[32952]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:28.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:28.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:28.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:28 smithi103 ceph-mon[32952]: from='osd.2 [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T23:53:28.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:28 smithi103 ceph-mon[32952]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T23:53:28.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:28.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:29.392 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:29 smithi040 ceph-mon[28095]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T23:53:29.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:29 smithi040 ceph-mon[28095]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T23:53:29.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:29 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:29.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:29 smithi040 ceph-mon[28095]: from='osd.2 [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:53:29.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:29 smithi040 ceph-mon[28095]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:53:29.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:29 smithi080 ceph-mon[32362]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T23:53:29.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:29 smithi080 ceph-mon[32362]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T23:53:29.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:29 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:29.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:29 smithi080 ceph-mon[32362]: from='osd.2 [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:53:29.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:29 smithi080 ceph-mon[32362]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:53:29.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:29 smithi103 ceph-mon[32952]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T23:53:29.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:29 smithi103 ceph-mon[32952]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T23:53:29.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:29 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:29.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:29 smithi103 ceph-mon[32952]: from='osd.2 [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:53:29.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:29 smithi103 ceph-mon[32952]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:53:30.270 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:30 smithi103 ceph-mon[32952]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:30.270 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:30 smithi103 ceph-mon[32952]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T23:53:30.270 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:30 smithi103 ceph-mon[32952]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T23:53:30.270 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:30 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:30.271 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:30 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:30.271 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:30 smithi103 ceph-mon[32952]: from='osd.2 ' entity='osd.2' 2024-04-26T23:53:30.271 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 26 23:53:29 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2[38087]: 2024-04-26T23:53:29.928+0000 7f58f066e700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T23:53:30.271 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 26 23:53:29 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2[38087]: 2024-04-26T23:53:29.932+0000 7f58e8459700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:53:30.456 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:30 smithi040 ceph-mon[28095]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:30.456 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:30 smithi040 ceph-mon[28095]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T23:53:30.456 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:30 smithi040 ceph-mon[28095]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T23:53:30.457 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:30 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:30.457 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:30 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:30.457 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:30 smithi040 ceph-mon[28095]: from='osd.2 ' entity='osd.2' 2024-04-26T23:53:30.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:30 smithi080 ceph-mon[32362]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:30.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:30 smithi080 ceph-mon[32362]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T23:53:30.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:30 smithi080 ceph-mon[32362]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T23:53:30.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:30 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:30.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:30 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:30.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:30 smithi080 ceph-mon[32362]: from='osd.2 ' entity='osd.2' 2024-04-26T23:53:31.445 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T23:53:31.445 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T23:53:31.445 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-26T23:53:31.446 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-26T23:53:31.446 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0544119 s, 193 MB/s 2024-04-26T23:53:31.446 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 2024-04-26T23:53:31.446 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-26T23:53:31.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:31 smithi080 ceph-mon[32362]: purged_snaps scrub starts 2024-04-26T23:53:31.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:31 smithi080 ceph-mon[32362]: purged_snaps scrub ok 2024-04-26T23:53:31.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:31.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:31 smithi080 ceph-mon[32362]: osd.2 [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187] boot 2024-04-26T23:53:31.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:31 smithi080 ceph-mon[32362]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T23:53:31.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:31.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:31 smithi103 ceph-mon[32952]: purged_snaps scrub starts 2024-04-26T23:53:31.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:31 smithi103 ceph-mon[32952]: purged_snaps scrub ok 2024-04-26T23:53:31.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:31.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:31 smithi103 ceph-mon[32952]: osd.2 [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187] boot 2024-04-26T23:53:31.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:31 smithi103 ceph-mon[32952]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T23:53:31.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:31.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:31 smithi040 ceph-mon[28095]: purged_snaps scrub starts 2024-04-26T23:53:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:31 smithi040 ceph-mon[28095]: purged_snaps scrub ok 2024-04-26T23:53:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:31 smithi040 ceph-mon[28095]: osd.2 [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187] boot 2024-04-26T23:53:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:31 smithi040 ceph-mon[28095]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T23:53:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:53:32.233 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_3 2024-04-26T23:53:32.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:32.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: Detected new or changed devices on smithi103 2024-04-26T23:53:32.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:32.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi103 to 17088M 2024-04-26T23:53:32.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:32.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:32.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:32.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:32.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:32.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: Detected new or changed devices on smithi103 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi103 to 17088M 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:32.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.749 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:53:32.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: Detected new or changed devices on smithi103 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi103 to 17088M 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:32.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:33.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:33 smithi040 ceph-mon[28095]: pgmap v92: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:33.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:33 smithi040 ceph-mon[28095]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T23:53:33.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:33 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:53:34.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:33 smithi080 ceph-mon[32362]: pgmap v92: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:34.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:33 smithi080 ceph-mon[32362]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T23:53:34.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:33 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:53:34.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:33 smithi103 ceph-mon[32952]: pgmap v92: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:34.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:33 smithi103 ceph-mon[32952]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T23:53:34.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:33 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:53:34.918 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T23:53:34.918 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:34 smithi040 ceph-mon[28095]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T23:53:34.918 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:53:34.919 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 26 23:53:34 smithi040 sudo[45103]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T23:53:34.919 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 26 23:53:34 smithi040 sudo[45103]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:53:34.919 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 26 23:53:34 smithi040 sudo[45103]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:53:34.919 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 26 23:53:34 smithi040 sudo[45103]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:53:34.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.932+0000 7f7055200700 1 -- 172.21.15.40:0/1352435928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7050104680 msgr2=0x7f7050104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:53:34.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.932+0000 7f7055200700 1 --2- 172.21.15.40:0/1352435928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7050104680 0x7f7050104ae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f703c009990 tx=0x7f703c009ca0 comp rx=0 tx=0).stop 2024-04-26T23:53:34.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.932+0000 7f7055200700 1 -- 172.21.15.40:0/1352435928 shutdown_connections 2024-04-26T23:53:34.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.932+0000 7f7055200700 1 --2- 172.21.15.40:0/1352435928 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7050105020 0x7f705010b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:34.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.932+0000 7f7055200700 1 --2- 172.21.15.40:0/1352435928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7050103480 0x7f70501038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:34.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.932+0000 7f7055200700 1 --2- 172.21.15.40:0/1352435928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7050104680 0x7f7050104ae0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:34.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.932+0000 7f7055200700 1 -- 172.21.15.40:0/1352435928 >> 172.21.15.40:0/1352435928 conn(0x7f70500fea20 msgr2=0x7f7050100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:53:34.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.932+0000 7f7055200700 1 -- 172.21.15.40:0/1352435928 shutdown_connections 2024-04-26T23:53:34.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.932+0000 7f7055200700 1 -- 172.21.15.40:0/1352435928 wait complete. 2024-04-26T23:53:34.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.933+0000 7f7055200700 1 Processor -- start 2024-04-26T23:53:34.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.933+0000 7f7055200700 1 -- start start 2024-04-26T23:53:34.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.934+0000 7f7055200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7050103480 0x7f705019b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:53:34.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.934+0000 7f7055200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7050104680 0x7f705019bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:53:34.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.934+0000 7f7055200700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7050105020 0x7f70501a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:53:34.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.934+0000 7f7055200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f705010ef70 con 0x7f7050104680 2024-04-26T23:53:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.934+0000 7f7055200700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f705010edf0 con 0x7f7050105020 2024-04-26T23:53:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.934+0000 7f7055200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f705010f0f0 con 0x7f7050103480 2024-04-26T23:53:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.934+0000 7f704de00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7050103480 0x7f705019b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:53:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.934+0000 7f704d400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7050104680 0x7f705019bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:53:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.934+0000 7f704de00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7050103480 0x7f705019b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:51072/0 (socket says 172.21.15.40:51072) 2024-04-26T23:53:34.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.934+0000 7f704de00700 1 -- 172.21.15.40:0/1236655054 learned_addr learned my addr 172.21.15.40:0/1236655054 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:53:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.934+0000 7f704e800700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7050105020 0x7f70501a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:53:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.935+0000 7f704de00700 1 -- 172.21.15.40:0/1236655054 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7050105020 msgr2=0x7f70501a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:53:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.935+0000 7f704de00700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7050105020 0x7f70501a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.935+0000 7f704de00700 1 -- 172.21.15.40:0/1236655054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7050104680 msgr2=0x7f705019bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:53:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.935+0000 7f704de00700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7050104680 0x7f705019bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:53:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.935+0000 7f704de00700 1 -- 172.21.15.40:0/1236655054 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70501a3460 con 0x7f7050103480 2024-04-26T23:53:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.935+0000 7f704de00700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7050103480 0x7f705019b800 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f704000e9a0 tx=0x7f704000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:53:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.935+0000 7f7046a00700 1 -- 172.21.15.40:0/1236655054 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704000cbd0 con 0x7f7050103480 2024-04-26T23:53:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.935+0000 7f7046a00700 1 -- 172.21.15.40:0/1236655054 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70400044d0 con 0x7f7050103480 2024-04-26T23:53:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.935+0000 7f7055200700 1 -- 172.21.15.40:0/1236655054 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70501a3750 con 0x7f7050103480 2024-04-26T23:53:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.935+0000 7f7046a00700 1 -- 172.21.15.40:0/1236655054 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040003d80 con 0x7f7050103480 2024-04-26T23:53:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.935+0000 7f7055200700 1 -- 172.21.15.40:0/1236655054 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70501a3ce0 con 0x7f7050103480 2024-04-26T23:53:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.937+0000 7f7055200700 1 -- 172.21.15.40:0/1236655054 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f705006a110 con 0x7f7050103480 2024-04-26T23:53:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.939+0000 7f7046a00700 1 -- 172.21.15.40:0/1236655054 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f7040003f60 con 0x7f7050103480 2024-04-26T23:53:34.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.940+0000 7f7046a00700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f70340a3c30 0x7f70340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:53:34.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.940+0000 7f7046a00700 1 -- 172.21.15.40:0/1236655054 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f70400cf270 con 0x7f7050103480 2024-04-26T23:53:34.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.940+0000 7f704d400700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f70340a3c30 0x7f70340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:53:34.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.941+0000 7f704d400700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f70340a3c30 0x7f70340a60f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f703c00b720 tx=0x7f703c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:53:34.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:34.944+0000 7f7046a00700 1 -- 172.21.15.40:0/1236655054 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7040016330 con 0x7f7050103480 2024-04-26T23:53:35.013 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T23:53:35.013 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:34 smithi080 ceph-mon[32362]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T23:53:35.014 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:53:35.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T23:53:35.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:34 smithi103 ceph-mon[32952]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T23:53:35.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:53:35.202 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 26 23:53:34 smithi040 sudo[45117]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T23:53:35.203 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 26 23:53:34 smithi040 sudo[45117]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:53:35.203 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 26 23:53:34 smithi040 sudo[45117]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:53:35.203 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 26 23:53:34 smithi040 sudo[45117]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:53:35.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:35.204+0000 7f7055200700 1 -- 172.21.15.40:0/1236655054 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f705010b540 con 0x7f70340a3c30 2024-04-26T23:53:35.301 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41059]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T23:53:35.301 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41059]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:53:35.301 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41059]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:53:35.301 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41059]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:53:35.302 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41062]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T23:53:35.302 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41062]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:53:35.302 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41062]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:53:35.302 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41062]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:53:35.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 sudo[45129]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T23:53:35.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 sudo[45129]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:53:35.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 sudo[45129]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:53:35.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 sudo[45129]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:53:35.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 sudo[45133]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T23:53:35.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 sudo[45133]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:53:35.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 sudo[45133]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:53:35.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 sudo[45133]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:53:35.602 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[42004]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T23:53:35.603 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[42004]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:53:35.605 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[42004]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:53:35.606 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[42004]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:53:35.606 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[42007]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T23:53:35.606 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[42007]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:53:35.606 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[42007]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:53:35.606 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[42007]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:53:35.606 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[41998]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T23:53:35.606 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[41998]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:53:35.606 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[41998]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:53:35.606 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[41998]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:53:35.606 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[42001]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T23:53:35.607 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[42001]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:53:35.607 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[42001]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:53:35.607 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 26 23:53:35 smithi103 sudo[42001]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:53:35.672 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41065]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T23:53:35.672 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41065]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:53:35.672 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41065]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:53:35.672 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41065]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:53:35.673 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41068]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T23:53:35.673 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41068]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:53:35.673 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41068]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:53:35.673 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 sudo[41068]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:53:35.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:35.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T23:53:35.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T23:53:35.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:53:35.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:53:35.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:53:35.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:35.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:53:35.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:53:35.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:53:35.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:53:35.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:53:35.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:35 smithi040 ceph-mon[28095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:53:36.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:36.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T23:53:36.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T23:53:36.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T23:53:36.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:36.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:36.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:36.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:53:36.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:53:36.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:53:36.053 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:53:36.054 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:35 smithi080 ceph-mon[32362]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:53:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T23:53:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T23:53:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T23:53:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:53:36.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:53:36.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:53:36.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:53:36.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:53:36.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:35 smithi103 ceph-mon[32952]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:53:36.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:53:36.689+0000 7f7046a00700 1 -- 172.21.15.40:0/1236655054 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7040098fd0 con 0x7f7050103480 2024-04-26T23:53:36.953 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:36 smithi080 ceph-mon[32362]: osdmap e22: 3 total, 3 up, 3 in 2024-04-26T23:53:36.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:36 smithi040 ceph-mon[28095]: osdmap e22: 3 total, 3 up, 3 in 2024-04-26T23:53:37.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:36 smithi103 ceph-mon[32952]: osdmap e22: 3 total, 3 up, 3 in 2024-04-26T23:53:37.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:37 smithi040 ceph-mon[28095]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:37.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:37 smithi040 ceph-mon[28095]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T23:53:37.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:37 smithi040 ceph-mon[28095]: Cluster is now healthy 2024-04-26T23:53:37.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:37 smithi040 ceph-mon[28095]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T23:53:38.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:37 smithi080 ceph-mon[32362]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:38.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:37 smithi080 ceph-mon[32362]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T23:53:38.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:37 smithi080 ceph-mon[32362]: Cluster is now healthy 2024-04-26T23:53:38.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:37 smithi080 ceph-mon[32362]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T23:53:38.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:37 smithi103 ceph-mon[32952]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:38.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:37 smithi103 ceph-mon[32952]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T23:53:38.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:37 smithi103 ceph-mon[32952]: Cluster is now healthy 2024-04-26T23:53:38.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:37 smithi103 ceph-mon[32952]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T23:53:39.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:39 smithi040 ceph-mon[28095]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:39.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:39 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/200796523' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c08a594-7548-44d3-b6cc-62b25ee078eb"}]: dispatch 2024-04-26T23:53:39.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:39 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c08a594-7548-44d3-b6cc-62b25ee078eb"}]: dispatch 2024-04-26T23:53:39.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:39 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c08a594-7548-44d3-b6cc-62b25ee078eb"}]': finished 2024-04-26T23:53:39.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:39 smithi040 ceph-mon[28095]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T23:53:39.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:39 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:53:40.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:39 smithi080 ceph-mon[32362]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:40.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:39 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/200796523' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c08a594-7548-44d3-b6cc-62b25ee078eb"}]: dispatch 2024-04-26T23:53:40.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:39 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c08a594-7548-44d3-b6cc-62b25ee078eb"}]: dispatch 2024-04-26T23:53:40.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:39 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c08a594-7548-44d3-b6cc-62b25ee078eb"}]': finished 2024-04-26T23:53:40.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:39 smithi080 ceph-mon[32362]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T23:53:40.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:39 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:53:40.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:39 smithi103 ceph-mon[32952]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:40.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:39 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/200796523' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c08a594-7548-44d3-b6cc-62b25ee078eb"}]: dispatch 2024-04-26T23:53:40.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:39 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c08a594-7548-44d3-b6cc-62b25ee078eb"}]: dispatch 2024-04-26T23:53:40.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:39 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c08a594-7548-44d3-b6cc-62b25ee078eb"}]': finished 2024-04-26T23:53:40.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:39 smithi103 ceph-mon[32952]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T23:53:40.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:53:40.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:40 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/340253287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:53:41.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:40 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/340253287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:53:41.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:40 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/340253287' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:53:41.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:41 smithi040 ceph-mon[28095]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:42.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:41 smithi080 ceph-mon[32362]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:42.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:41 smithi103 ceph-mon[32952]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:43.962 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:43 smithi040 ceph-mon[28095]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:44.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:43 smithi080 ceph-mon[32362]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:44.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:43 smithi103 ceph-mon[32952]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:45.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:45 smithi040 ceph-mon[28095]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:46.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:45 smithi080 ceph-mon[32362]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:46.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:45 smithi103 ceph-mon[32952]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:47.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:47 smithi040 ceph-mon[28095]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:48.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:47 smithi080 ceph-mon[32362]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:48.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:47 smithi103 ceph-mon[32952]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:49.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T23:53:49.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:49.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T23:53:49.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:49.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T23:53:49.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:49.797 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:49 smithi040 ceph-mon[28095]: Deploying daemon osd.3 on smithi040 2024-04-26T23:53:49.797 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:49 smithi040 ceph-mon[28095]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:50.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:49 smithi080 ceph-mon[32362]: Deploying daemon osd.3 on smithi040 2024-04-26T23:53:50.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:49 smithi080 ceph-mon[32362]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:50.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:49 smithi103 ceph-mon[32952]: Deploying daemon osd.3 on smithi040 2024-04-26T23:53:50.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:49 smithi103 ceph-mon[32952]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:51.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:51 smithi040 ceph-mon[28095]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:52.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:51 smithi080 ceph-mon[32362]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:52.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:51 smithi103 ceph-mon[32952]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:53.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:53 smithi040 ceph-mon[28095]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:54.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:53 smithi080 ceph-mon[32362]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:54.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:53 smithi103 ceph-mon[32952]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:56.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:55 smithi080 ceph-mon[32362]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:56.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:55 smithi103 ceph-mon[32952]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:56.155 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:55 smithi040 ceph-mon[28095]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:58.429 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:58 smithi040 ceph-mon[28095]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:58.429 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:58 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:58.429 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:58 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:58.429 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:53:58 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:58.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:58 smithi080 ceph-mon[32362]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:58.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:58 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:58.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:58 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:58.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:53:58 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:58.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:58 smithi103 ceph-mon[32952]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:53:58.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:58 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:58.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:58 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:53:58.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:53:58 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:00.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:00 smithi040 ceph-mon[28095]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:54:00.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:00 smithi080 ceph-mon[32362]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:54:00.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:00 smithi103 ceph-mon[32952]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:54:01.944 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:01 smithi040 ceph-mon[28095]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:54:01.944 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:01.944 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:01.945 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:01.945 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:01.945 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:01.945 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:01 smithi040 ceph-mon[28095]: from='osd.3 [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T23:54:01.945 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:01 smithi040 ceph-mon[28095]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T23:54:02.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:01 smithi080 ceph-mon[32362]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:54:02.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:02.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:02.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:02.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:02.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:02.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:01 smithi080 ceph-mon[32362]: from='osd.3 [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T23:54:02.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:01 smithi080 ceph-mon[32362]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T23:54:02.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:01 smithi103 ceph-mon[32952]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:54:02.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:02.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:02.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:02.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:02.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:02.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:01 smithi103 ceph-mon[32952]: from='osd.3 [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T23:54:02.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:01 smithi103 ceph-mon[32952]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T23:54:02.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.159+0000 7f7046a00700 1 -- 172.21.15.40:0/1236655054 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f705010b540 con 0x7f70340a3c30 2024-04-26T23:54:02.163 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 3 on host 'smithi040' 2024-04-26T23:54:02.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.163+0000 7f7055200700 1 -- 172.21.15.40:0/1236655054 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f70340a3c30 msgr2=0x7f70340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:54:02.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.163+0000 7f7055200700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f70340a3c30 0x7f70340a60f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f703c00b720 tx=0x7f703c005fd0 comp rx=0 tx=0).stop 2024-04-26T23:54:02.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.163+0000 7f7055200700 1 -- 172.21.15.40:0/1236655054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7050103480 msgr2=0x7f705019b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:54:02.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.163+0000 7f7055200700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7050103480 0x7f705019b800 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f704000e9a0 tx=0x7f704000ecb0 comp rx=0 tx=0).stop 2024-04-26T23:54:02.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.163+0000 7f7055200700 1 -- 172.21.15.40:0/1236655054 shutdown_connections 2024-04-26T23:54:02.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.163+0000 7f7055200700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f70340a3c30 0x7f70340a60f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:02.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.163+0000 7f7055200700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7050105020 0x7f70501a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:02.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.163+0000 7f7055200700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7050103480 0x7f705019b800 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:02.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.163+0000 7f7055200700 1 --2- 172.21.15.40:0/1236655054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7050104680 0x7f705019bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:02.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.163+0000 7f7055200700 1 -- 172.21.15.40:0/1236655054 >> 172.21.15.40:0/1236655054 conn(0x7f70500fea20 msgr2=0x7f7050109e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:54:02.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.164+0000 7f7055200700 1 -- 172.21.15.40:0/1236655054 shutdown_connections 2024-04-26T23:54:02.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:54:02.164+0000 7f7055200700 1 -- 172.21.15.40:0/1236655054 wait complete. 2024-04-26T23:54:02.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:02 smithi040 ceph-mon[28095]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T23:54:02.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:02 smithi040 ceph-mon[28095]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T23:54:02.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:02 smithi040 ceph-mon[28095]: from='osd.3 [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:54:02.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:02 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:54:02.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:02 smithi040 ceph-mon[28095]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:54:02.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:02 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:02.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:02 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:02.804 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:02 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:02.857 DEBUG:teuthology.orchestra.run.smithi040:osd.3> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T23:54:02.862 INFO:tasks.cephadm:Deploying osd.4 on smithi080 with /dev/vg_nvme/lv_3... 2024-04-26T23:54:02.862 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T23:54:03.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:02 smithi080 ceph-mon[32362]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T23:54:03.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:02 smithi080 ceph-mon[32362]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T23:54:03.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:02 smithi080 ceph-mon[32362]: from='osd.3 [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:54:03.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:02 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:54:03.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:02 smithi080 ceph-mon[32362]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:54:03.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:02 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:03.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:02 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:03.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:02 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:03.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:02 smithi103 ceph-mon[32952]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T23:54:03.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:02 smithi103 ceph-mon[32952]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T23:54:03.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:02 smithi103 ceph-mon[32952]: from='osd.3 [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:54:03.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:02 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:54:03.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:02 smithi103 ceph-mon[32952]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:54:03.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:02 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:03.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:02 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:03.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:02 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:03.102 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:54:03.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:03 smithi080 ceph-mon[32362]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:54:03.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:03 smithi080 ceph-mon[32362]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-26T23:54:03.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:03 smithi080 ceph-mon[32362]: osdmap e25: 4 total, 3 up, 4 in 2024-04-26T23:54:03.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:54:03.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:54:03.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:03 smithi080 ceph-mon[32362]: from='osd.3 ' entity='osd.3' 2024-04-26T23:54:03.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:03 smithi040 ceph-mon[28095]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:54:03.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:03 smithi040 ceph-mon[28095]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-26T23:54:03.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:03 smithi040 ceph-mon[28095]: osdmap e25: 4 total, 3 up, 4 in 2024-04-26T23:54:03.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:54:03.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:54:03.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:03 smithi040 ceph-mon[28095]: from='osd.3 ' entity='osd.3' 2024-04-26T23:54:03.969 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 26 23:54:03 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3[47395]: 2024-04-26T23:54:03.612+0000 7fc608600700 -1 osd.3 0 waiting for initial osdmap 2024-04-26T23:54:03.969 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 26 23:54:03 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3[47395]: 2024-04-26T23:54:03.643+0000 7fc5fe400700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:54:04.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:03 smithi103 ceph-mon[32952]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:54:04.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:03 smithi103 ceph-mon[32952]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-26T23:54:04.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:03 smithi103 ceph-mon[32952]: osdmap e25: 4 total, 3 up, 4 in 2024-04-26T23:54:04.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:54:04.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:54:04.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:03 smithi103 ceph-mon[32952]: from='osd.3 ' entity='osd.3' 2024-04-26T23:54:04.844 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:04 smithi080 ceph-mon[32362]: purged_snaps scrub starts 2024-04-26T23:54:04.844 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:04 smithi080 ceph-mon[32362]: purged_snaps scrub ok 2024-04-26T23:54:04.844 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:04 smithi080 ceph-mon[32362]: osd.3 [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893] boot 2024-04-26T23:54:04.844 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:04 smithi080 ceph-mon[32362]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T23:54:04.844 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:54:04.844 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:04 smithi080 ceph-mon[32362]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T23:54:05.012 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:04 smithi040 ceph-mon[28095]: purged_snaps scrub starts 2024-04-26T23:54:05.012 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:04 smithi040 ceph-mon[28095]: purged_snaps scrub ok 2024-04-26T23:54:05.012 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:04 smithi040 ceph-mon[28095]: osd.3 [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893] boot 2024-04-26T23:54:05.012 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:04 smithi040 ceph-mon[28095]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T23:54:05.012 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:54:05.012 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:04 smithi040 ceph-mon[28095]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T23:54:05.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:04 smithi103 ceph-mon[32952]: purged_snaps scrub starts 2024-04-26T23:54:05.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:04 smithi103 ceph-mon[32952]: purged_snaps scrub ok 2024-04-26T23:54:05.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:04 smithi103 ceph-mon[32952]: osd.3 [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893] boot 2024-04-26T23:54:05.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:04 smithi103 ceph-mon[32952]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T23:54:05.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:54:05.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:04 smithi103 ceph-mon[32952]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T23:54:05.210 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T23:54:05.211 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T23:54:05.211 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-04-26T23:54:05.211 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-04-26T23:54:05.211 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112797 s, 930 MB/s 2024-04-26T23:54:05.211 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-04-26T23:54:05.892 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi080:vg_nvme/lv_3 2024-04-26T23:54:05.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:05 smithi040 ceph-mon[28095]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:54:05.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:05 smithi040 ceph-mon[28095]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T23:54:06.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:05 smithi080 ceph-mon[32362]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:54:06.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:05 smithi080 ceph-mon[32362]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T23:54:06.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:05 smithi103 ceph-mon[32952]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:54:06.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:05 smithi103 ceph-mon[32952]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T23:54:06.132 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:54:07.736 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.732+0000 7f1de5fa3700 1 -- 172.21.15.80:0/2674023180 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1de0103580 msgr2=0x7f1de01039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:54:07.736 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.732+0000 7f1de5fa3700 1 --2- 172.21.15.80:0/2674023180 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1de0103580 0x7f1de01039e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1dc80099c0 tx=0x7f1dc8009cd0 comp rx=0 tx=0).stop 2024-04-26T23:54:07.736 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.732+0000 7f1de5fa3700 1 -- 172.21.15.80:0/2674023180 shutdown_connections 2024-04-26T23:54:07.736 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.732+0000 7f1de5fa3700 1 --2- 172.21.15.80:0/2674023180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de0103f20 0x7f1de010a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:07.737 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.732+0000 7f1de5fa3700 1 --2- 172.21.15.80:0/2674023180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1de0102380 0x7f1de01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:07.737 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.732+0000 7f1de5fa3700 1 --2- 172.21.15.80:0/2674023180 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1de0103580 0x7f1de01039e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:07.737 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.732+0000 7f1de5fa3700 1 -- 172.21.15.80:0/2674023180 >> 172.21.15.80:0/2674023180 conn(0x7f1de00fd920 msgr2=0x7f1de00ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:54:07.737 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.732+0000 7f1de5fa3700 1 -- 172.21.15.80:0/2674023180 shutdown_connections 2024-04-26T23:54:07.737 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.732+0000 7f1de5fa3700 1 -- 172.21.15.80:0/2674023180 wait complete. 2024-04-26T23:54:07.738 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:07 smithi080 ceph-mon[32362]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T23:54:07.738 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:07 smithi080 ceph-mon[32362]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:07.742 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.733+0000 7f1de5fa3700 1 Processor -- start 2024-04-26T23:54:07.742 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.733+0000 7f1de5fa3700 1 -- start start 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.733+0000 7f1de5fa3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1de0102380 0x7f1de01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.733+0000 7f1de5fa3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de0103580 0x7f1de0198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.733+0000 7f1de5fa3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1de0103f20 0x7f1de019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.733+0000 7f1de5fa3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1de010df10 con 0x7f1de0103580 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.733+0000 7f1de5fa3700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1de010dd90 con 0x7f1de0103f20 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.733+0000 7f1de5fa3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1de010e090 con 0x7f1de0102380 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1de0103f20 0x7f1de019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddffff700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1de0103f20 0x7f1de019fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:51348/0 (socket says 172.21.15.80:51348) 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddffff700 1 -- 172.21.15.80:0/840100279 learned_addr learned my addr 172.21.15.80:0/840100279 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddf7fe700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1de0102380 0x7f1de01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddffff700 1 -- 172.21.15.80:0/840100279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1de0102380 msgr2=0x7f1de01985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddffff700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1de0102380 0x7f1de01985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddffff700 1 -- 172.21.15.80:0/840100279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de0103580 msgr2=0x7f1de0198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:54:07.743 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddffff700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de0103580 0x7f1de0198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddffff700 1 -- 172.21.15.80:0/840100279 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de01a0240 con 0x7f1de0103f20 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddffff700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1de0103f20 0x7f1de019fba0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800b970 tx=0x7f1dd800bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddd7fa700 1 -- 172.21.15.80:0/840100279 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8019070 con 0x7f1de0103f20 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1de5fa3700 1 -- 172.21.15.80:0/840100279 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1de006b160 con 0x7f1de0103f20 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddd7fa700 1 -- 172.21.15.80:0/840100279 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dd800cde0 con 0x7f1de0103f20 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1ddd7fa700 1 -- 172.21.15.80:0/840100279 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8007370 con 0x7f1de0103f20 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.734+0000 7f1de5fa3700 1 -- 172.21.15.80:0/840100279 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1de006b770 con 0x7f1de0103f20 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.735+0000 7f1de5fa3700 1 -- 172.21.15.80:0/840100279 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1de006a110 con 0x7f1de0103f20 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.737+0000 7f1ddd7fa700 1 -- 172.21.15.80:0/840100279 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1dd801c070 con 0x7f1de0103f20 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.737+0000 7f1ddd7fa700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1dcc0a3c40 0x7f1dcc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.738+0000 7f1ddd7fa700 1 -- 172.21.15.80:0/840100279 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 3669+0+0 (secure 0 0 0) 0x7f1dd80ce660 con 0x7f1de0103f20 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.738+0000 7f1ddf7fe700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1dcc0a3c40 0x7f1dcc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:54:07.744 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.739+0000 7f1ddf7fe700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1dcc0a3c40 0x7f1dcc0a6100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1de01033e0 tx=0x7f1dd0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:54:07.746 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.743+0000 7f1ddd7fa700 1 -- 172.21.15.80:0/840100279 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1dd809c260 con 0x7f1de0103f20 2024-04-26T23:54:07.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:07 smithi103 ceph-mon[32952]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T23:54:07.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:07 smithi103 ceph-mon[32952]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:07.951 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:07.948+0000 7f1de5fa3700 1 -- 172.21.15.80:0/840100279 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1de006ba90 con 0x7f1dcc0a3c40 2024-04-26T23:54:07.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:07 smithi040 ceph-mon[28095]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T23:54:07.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:07 smithi040 ceph-mon[28095]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:08.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:08 smithi103 ceph-mon[32952]: osdmap e30: 4 total, 4 up, 4 in 2024-04-26T23:54:08.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:08 smithi103 ceph-mon[32952]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:54:08.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:08 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:54:08.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:08 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:54:08.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:08 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:08.860 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:08 smithi080 ceph-mon[32362]: osdmap e30: 4 total, 4 up, 4 in 2024-04-26T23:54:08.860 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:08 smithi080 ceph-mon[32362]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:54:08.860 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:08 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:54:08.860 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:08 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:54:08.860 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:08 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:08.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:08 smithi040 ceph-mon[28095]: osdmap e30: 4 total, 4 up, 4 in 2024-04-26T23:54:08.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:08 smithi040 ceph-mon[28095]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:54:08.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:08 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:54:08.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:08 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:54:08.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:08 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:09.791 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:09 smithi080 ceph-mon[32362]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T23:54:09.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:09 smithi103 ceph-mon[32952]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T23:54:09.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:09 smithi040 ceph-mon[28095]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T23:54:10.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: Detected new or changed devices on smithi040 2024-04-26T23:54:10.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:10.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:10.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:10.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:10.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi040 to 8547M 2024-04-26T23:54:10.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:10.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:10.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:10.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:10.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: from='client.? 172.21.15.80:0/3348496498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e645527-e694-4cd0-99d7-035f5c36bed0"}]: dispatch 2024-04-26T23:54:10.969 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e645527-e694-4cd0-99d7-035f5c36bed0"}]: dispatch 2024-04-26T23:54:10.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e645527-e694-4cd0-99d7-035f5c36bed0"}]': finished 2024-04-26T23:54:10.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T23:54:10.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:10 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:10.994 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: Detected new or changed devices on smithi040 2024-04-26T23:54:10.994 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:10.994 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:10.994 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:10.994 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:10.994 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi040 to 8547M 2024-04-26T23:54:10.994 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:10.995 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:10.995 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:10.995 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:10.995 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: from='client.? 172.21.15.80:0/3348496498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e645527-e694-4cd0-99d7-035f5c36bed0"}]: dispatch 2024-04-26T23:54:10.995 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e645527-e694-4cd0-99d7-035f5c36bed0"}]: dispatch 2024-04-26T23:54:10.995 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e645527-e694-4cd0-99d7-035f5c36bed0"}]': finished 2024-04-26T23:54:10.995 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T23:54:10.995 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:10 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:11.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: Detected new or changed devices on smithi040 2024-04-26T23:54:11.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:11.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi040 to 8547M 2024-04-26T23:54:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: from='client.? 172.21.15.80:0/3348496498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e645527-e694-4cd0-99d7-035f5c36bed0"}]: dispatch 2024-04-26T23:54:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e645527-e694-4cd0-99d7-035f5c36bed0"}]: dispatch 2024-04-26T23:54:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e645527-e694-4cd0-99d7-035f5c36bed0"}]': finished 2024-04-26T23:54:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T23:54:11.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:10 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:11.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:11 smithi040 ceph-mon[28095]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-26T23:54:11.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:11 smithi040 ceph-mon[28095]: from='client.? 172.21.15.80:0/1907670503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:54:12.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:11 smithi080 ceph-mon[32362]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-26T23:54:12.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:11 smithi080 ceph-mon[32362]: from='client.? 172.21.15.80:0/1907670503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:54:12.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:11 smithi103 ceph-mon[32952]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-26T23:54:12.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:11 smithi103 ceph-mon[32952]: from='client.? 172.21.15.80:0/1907670503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:54:13.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:13 smithi040 ceph-mon[28095]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T23:54:14.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:13 smithi080 ceph-mon[32362]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T23:54:14.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:13 smithi103 ceph-mon[32952]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T23:54:15.837 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:15 smithi080 ceph-mon[32362]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T23:54:16.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:15 smithi103 ceph-mon[32952]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T23:54:16.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:15 smithi040 ceph-mon[28095]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T23:54:17.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:17 smithi080 ceph-mon[32362]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T23:54:18.021 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:17 smithi040 ceph-mon[28095]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T23:54:18.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:17 smithi103 ceph-mon[32952]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T23:54:18.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:18 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T23:54:18.923 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:18 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:18.923 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:18 smithi080 ceph-mon[32362]: Deploying daemon osd.4 on smithi080 2024-04-26T23:54:19.054 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:18 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T23:54:19.054 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:18 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:19.054 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:18 smithi040 ceph-mon[28095]: Deploying daemon osd.4 on smithi080 2024-04-26T23:54:19.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:18 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T23:54:19.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:18 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:19.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:18 smithi103 ceph-mon[32952]: Deploying daemon osd.4 on smithi080 2024-04-26T23:54:19.894 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:19 smithi080 ceph-mon[32362]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:20.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:19 smithi103 ceph-mon[32952]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:19 smithi040 ceph-mon[28095]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:22.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:21 smithi080 ceph-mon[32362]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:22.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:21 smithi103 ceph-mon[32952]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:22.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:21 smithi040 ceph-mon[28095]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:24.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:23 smithi080 ceph-mon[32362]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:24.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:24.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:24.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:23 smithi103 ceph-mon[32952]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:23 smithi040 ceph-mon[28095]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:26.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:25 smithi080 ceph-mon[32362]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:26.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:25 smithi103 ceph-mon[32952]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:26.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:25 smithi040 ceph-mon[28095]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:27.284 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.281+0000 7f1ddd7fa700 1 -- 172.21.15.80:0/840100279 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1de006ba90 con 0x7f1dcc0a3c40 2024-04-26T23:54:27.284 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 4 on host 'smithi080' 2024-04-26T23:54:27.286 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.283+0000 7f1de5fa3700 1 -- 172.21.15.80:0/840100279 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1dcc0a3c40 msgr2=0x7f1dcc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:54:27.286 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.283+0000 7f1de5fa3700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1dcc0a3c40 0x7f1dcc0a6100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1de01033e0 tx=0x7f1dd0009190 comp rx=0 tx=0).stop 2024-04-26T23:54:27.286 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.283+0000 7f1de5fa3700 1 -- 172.21.15.80:0/840100279 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1de0103f20 msgr2=0x7f1de019fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:54:27.286 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.283+0000 7f1de5fa3700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1de0103f20 0x7f1de019fba0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800b970 tx=0x7f1dd800bc80 comp rx=0 tx=0).stop 2024-04-26T23:54:27.287 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.284+0000 7f1de5fa3700 1 -- 172.21.15.80:0/840100279 shutdown_connections 2024-04-26T23:54:27.287 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.284+0000 7f1de5fa3700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1dcc0a3c40 0x7f1dcc0a6100 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:27.287 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.284+0000 7f1de5fa3700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1de0103f20 0x7f1de019fba0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:27.287 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.284+0000 7f1de5fa3700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1de0102380 0x7f1de01985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:27.287 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.284+0000 7f1de5fa3700 1 --2- 172.21.15.80:0/840100279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1de0103580 0x7f1de0198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:27.287 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.284+0000 7f1de5fa3700 1 -- 172.21.15.80:0/840100279 >> 172.21.15.80:0/840100279 conn(0x7f1de00fd920 msgr2=0x7f1de01068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:54:27.287 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.284+0000 7f1de5fa3700 1 -- 172.21.15.80:0/840100279 shutdown_connections 2024-04-26T23:54:27.287 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:54:27.284+0000 7f1de5fa3700 1 -- 172.21.15.80:0/840100279 wait complete. 2024-04-26T23:54:27.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:27 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:27 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:27 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:27.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:27 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:27.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:27 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:27 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:27 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:27.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:27 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:27 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:27 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:27 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:27.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:27 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:27.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:27 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:27 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:27 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:27.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:27 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:27 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:27 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:27 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:27.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:27 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:27.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:27 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:27 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:27 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:27.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:27 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:27.897 DEBUG:teuthology.orchestra.run.smithi080:osd.4> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T23:54:27.901 INFO:tasks.cephadm:Deploying osd.5 on smithi103 with /dev/vg_nvme/lv_3... 2024-04-26T23:54:27.901 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T23:54:28.149 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:54:28.458 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:28 smithi080 ceph-mon[32362]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:28.459 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:28 smithi080 ceph-mon[32362]: from='osd.4 [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T23:54:28.459 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:28 smithi080 ceph-mon[32362]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T23:54:28.546 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:28 smithi103 ceph-mon[32952]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:28.546 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:28 smithi103 ceph-mon[32952]: from='osd.4 [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T23:54:28.546 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:28 smithi103 ceph-mon[32952]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T23:54:28.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:28 smithi040 ceph-mon[28095]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:28.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:28 smithi040 ceph-mon[28095]: from='osd.4 [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T23:54:28.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:28 smithi040 ceph-mon[28095]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T23:54:29.448 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:29 smithi080 ceph-mon[32362]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T23:54:29.448 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:29 smithi080 ceph-mon[32362]: from='osd.4 [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:54:29.448 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:29 smithi080 ceph-mon[32362]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T23:54:29.448 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:29 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:29.448 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:29 smithi080 ceph-mon[32362]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:54:29.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:29 smithi103 ceph-mon[32952]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T23:54:29.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:29 smithi103 ceph-mon[32952]: from='osd.4 [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:54:29.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:29 smithi103 ceph-mon[32952]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T23:54:29.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:29 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:29.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:29 smithi103 ceph-mon[32952]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:54:29.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:29 smithi040 ceph-mon[28095]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T23:54:29.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:29 smithi040 ceph-mon[28095]: from='osd.4 [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:54:29.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:29 smithi040 ceph-mon[28095]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T23:54:29.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:29 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:29.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:29 smithi040 ceph-mon[28095]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:54:30.178 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T23:54:30.178 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T23:54:30.178 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-04-26T23:54:30.178 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-04-26T23:54:30.179 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113373 s, 925 MB/s 2024-04-26T23:54:30.179 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-04-26T23:54:30.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:30 smithi040 ceph-mon[28095]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:30.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:30 smithi040 ceph-mon[28095]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-04-26T23:54:30.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:30 smithi040 ceph-mon[28095]: osdmap e33: 5 total, 4 up, 5 in 2024-04-26T23:54:30.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:30 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:30.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:30 smithi040 ceph-mon[28095]: from='osd.4 ' entity='osd.4' 2024-04-26T23:54:30.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:30 smithi080 ceph-mon[32362]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:30.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:30 smithi080 ceph-mon[32362]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-04-26T23:54:30.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:30 smithi080 ceph-mon[32362]: osdmap e33: 5 total, 4 up, 5 in 2024-04-26T23:54:30.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:30 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:30.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:30 smithi080 ceph-mon[32362]: from='osd.4 ' entity='osd.4' 2024-04-26T23:54:30.553 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 26 23:54:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4[43898]: 2024-04-26T23:54:30.118+0000 7f5bd7b0e700 -1 osd.4 0 waiting for initial osdmap 2024-04-26T23:54:30.553 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 26 23:54:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4[43898]: 2024-04-26T23:54:30.125+0000 7f5bcf8f9700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:54:30.803 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:30 smithi103 ceph-mon[32952]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:30.803 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:30 smithi103 ceph-mon[32952]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-04-26T23:54:30.803 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:30 smithi103 ceph-mon[32952]: osdmap e33: 5 total, 4 up, 5 in 2024-04-26T23:54:30.803 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:30 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:30.803 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:30 smithi103 ceph-mon[32952]: from='osd.4 ' entity='osd.4' 2024-04-26T23:54:30.831 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_3 2024-04-26T23:54:31.066 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:54:31.472 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: purged_snaps scrub starts 2024-04-26T23:54:31.473 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: purged_snaps scrub ok 2024-04-26T23:54:31.473 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:31.473 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:31.473 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:31.473 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:31.473 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:31.473 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:31.473 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:31.473 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:31.474 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:31.474 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: osd.4 [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790] boot 2024-04-26T23:54:31.474 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T23:54:31.474 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: purged_snaps scrub starts 2024-04-26T23:54:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: purged_snaps scrub ok 2024-04-26T23:54:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:31.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:31.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:31.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: osd.4 [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790] boot 2024-04-26T23:54:31.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T23:54:31.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:31.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: purged_snaps scrub starts 2024-04-26T23:54:31.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: purged_snaps scrub ok 2024-04-26T23:54:31.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:31.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:31.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:31.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:31.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:31.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:31.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:31.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:31.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:31.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: osd.4 [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790] boot 2024-04-26T23:54:31.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T23:54:31.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:54:32.514 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:32 smithi103 ceph-mon[32952]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:32.515 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:32 smithi103 ceph-mon[32952]: Detected new or changed devices on smithi080 2024-04-26T23:54:32.515 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:32 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi080 to 8544M 2024-04-26T23:54:32.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:32 smithi040 ceph-mon[28095]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:32.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:32 smithi040 ceph-mon[28095]: Detected new or changed devices on smithi080 2024-04-26T23:54:32.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:32 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi080 to 8544M 2024-04-26T23:54:32.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.730+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/2011923618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e4103480 msgr2=0x7fb7e41038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:54:32.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.730+0000 7fb7ec6e3700 1 --2- 172.21.15.103:0/2011923618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e4103480 0x7fb7e41038a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009990 tx=0x7fb7d4009ca0 comp rx=0 tx=0).stop 2024-04-26T23:54:32.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.730+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/2011923618 shutdown_connections 2024-04-26T23:54:32.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.730+0000 7fb7ec6e3700 1 --2- 172.21.15.103:0/2011923618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e4105020 0x7fb7e410b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:32.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.730+0000 7fb7ec6e3700 1 --2- 172.21.15.103:0/2011923618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e4103480 0x7fb7e41038a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:32.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.730+0000 7fb7ec6e3700 1 --2- 172.21.15.103:0/2011923618 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e4104680 0x7fb7e4104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:32.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.730+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/2011923618 >> 172.21.15.103:0/2011923618 conn(0x7fb7e40fea20 msgr2=0x7fb7e4100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:54:32.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.730+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/2011923618 shutdown_connections 2024-04-26T23:54:32.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.730+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/2011923618 wait complete. 2024-04-26T23:54:32.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.731+0000 7fb7ec6e3700 1 Processor -- start 2024-04-26T23:54:32.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.731+0000 7fb7ec6e3700 1 -- start start 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.731+0000 7fb7ec6e3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e4103480 0x7fb7e41996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.731+0000 7fb7ec6e3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e4104680 0x7fb7e4199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.731+0000 7fb7ec6e3700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e4105020 0x7fb7e41a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.731+0000 7fb7ec6e3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e410efd0 con 0x7fb7e4103480 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.731+0000 7fb7ec6e3700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e410ee50 con 0x7fb7e4105020 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.731+0000 7fb7ec6e3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e410f150 con 0x7fb7e4104680 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.731+0000 7fb7e9c7e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e4104680 0x7fb7e4199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.731+0000 7fb7e9c7e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e4104680 0x7fb7e4199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33438/0 (socket says 172.21.15.103:33438) 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.731+0000 7fb7e9c7e700 1 -- 172.21.15.103:0/642158625 learned_addr learned my addr 172.21.15.103:0/642158625 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.731+0000 7fb7eac80700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e4105020 0x7fb7e41a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.732+0000 7fb7e9c7e700 1 -- 172.21.15.103:0/642158625 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e4105020 msgr2=0x7fb7e41a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.732+0000 7fb7e9c7e700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e4105020 0x7fb7e41a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.732+0000 7fb7e9c7e700 1 -- 172.21.15.103:0/642158625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e4103480 msgr2=0x7fb7e41996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:54:32.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.732+0000 7fb7e9c7e700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e4103480 0x7fb7e41996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:32.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.732+0000 7fb7e9c7e700 1 -- 172.21.15.103:0/642158625 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e41a1310 con 0x7fb7e4104680 2024-04-26T23:54:32.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.732+0000 7fb7e9c7e700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e4104680 0x7fb7e4199bf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000c950 tx=0x7fb7e000cd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:54:32.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.732+0000 7fb7db7fe700 1 -- 172.21.15.103:0/642158625 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0007ac0 con 0x7fb7e4104680 2024-04-26T23:54:32.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.732+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/642158625 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e41a1600 con 0x7fb7e4104680 2024-04-26T23:54:32.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.732+0000 7fb7db7fe700 1 -- 172.21.15.103:0/642158625 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7e0004ce0 con 0x7fb7e4104680 2024-04-26T23:54:32.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.732+0000 7fb7db7fe700 1 -- 172.21.15.103:0/642158625 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e00055a0 con 0x7fb7e4104680 2024-04-26T23:54:32.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.732+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/642158625 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e41a1b40 con 0x7fb7e4104680 2024-04-26T23:54:32.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.733+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/642158625 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7e406a110 con 0x7fb7e4104680 2024-04-26T23:54:32.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.735+0000 7fb7db7fe700 1 -- 172.21.15.103:0/642158625 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb7e0007c60 con 0x7fb7e4104680 2024-04-26T23:54:32.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.735+0000 7fb7db7fe700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7d00a3c40 0x7fb7d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:54:32.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.735+0000 7fb7db7fe700 1 -- 172.21.15.103:0/642158625 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7fb7e00cf370 con 0x7fb7e4104680 2024-04-26T23:54:32.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.736+0000 7fb7ea47f700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7d00a3c40 0x7fb7d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:54:32.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.737+0000 7fb7ea47f700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7d00a3c40 0x7fb7d00a6100 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4003790 tx=0x7fb7d4003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:54:32.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.739+0000 7fb7db7fe700 1 -- 172.21.15.103:0/642158625 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb7e0014030 con 0x7fb7e4104680 2024-04-26T23:54:32.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:32 smithi080 ceph-mon[32362]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:54:32.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:32 smithi080 ceph-mon[32362]: Detected new or changed devices on smithi080 2024-04-26T23:54:32.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:32 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi080 to 8544M 2024-04-26T23:54:32.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:32.943+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/642158625 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb7e4063860 con 0x7fb7d00a3c40 2024-04-26T23:54:33.717 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:33 smithi103 ceph-mon[32952]: osdmap e35: 5 total, 5 up, 5 in 2024-04-26T23:54:33.717 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:33 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:54:33.717 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:33 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:54:33.717 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:33 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:33.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:33 smithi040 ceph-mon[28095]: osdmap e35: 5 total, 5 up, 5 in 2024-04-26T23:54:33.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:33 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:54:33.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:33 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:54:33.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:33 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:33.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:33 smithi080 ceph-mon[32362]: osdmap e35: 5 total, 5 up, 5 in 2024-04-26T23:54:33.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:33 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:54:33.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:33 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:54:33.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:33 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:34.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:34 smithi040 ceph-mon[28095]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:34.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:34 smithi040 ceph-mon[28095]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:54:34.731 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:34 smithi103 ceph-mon[32952]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:34.731 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:34 smithi103 ceph-mon[32952]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:54:34.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:34 smithi080 ceph-mon[32362]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:34.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:34 smithi080 ceph-mon[32362]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:54:36.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:36 smithi040 ceph-mon[28095]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:36.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:36 smithi040 ceph-mon[28095]: from='client.? 172.21.15.103:0/2537075664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f6e9855-4856-4a8a-a884-1826a9c29327"}]: dispatch 2024-04-26T23:54:36.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:36 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f6e9855-4856-4a8a-a884-1826a9c29327"}]: dispatch 2024-04-26T23:54:36.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:36 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f6e9855-4856-4a8a-a884-1826a9c29327"}]': finished 2024-04-26T23:54:36.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:36 smithi040 ceph-mon[28095]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T23:54:36.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:54:36.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:36 smithi080 ceph-mon[32362]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:36.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:36 smithi080 ceph-mon[32362]: from='client.? 172.21.15.103:0/2537075664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f6e9855-4856-4a8a-a884-1826a9c29327"}]: dispatch 2024-04-26T23:54:36.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:36 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f6e9855-4856-4a8a-a884-1826a9c29327"}]: dispatch 2024-04-26T23:54:36.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:36 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f6e9855-4856-4a8a-a884-1826a9c29327"}]': finished 2024-04-26T23:54:36.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:36 smithi080 ceph-mon[32362]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T23:54:36.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:54:36.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:36 smithi103 ceph-mon[32952]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:36.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:36 smithi103 ceph-mon[32952]: from='client.? 172.21.15.103:0/2537075664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f6e9855-4856-4a8a-a884-1826a9c29327"}]: dispatch 2024-04-26T23:54:36.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:36 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f6e9855-4856-4a8a-a884-1826a9c29327"}]: dispatch 2024-04-26T23:54:36.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:36 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f6e9855-4856-4a8a-a884-1826a9c29327"}]': finished 2024-04-26T23:54:36.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:36 smithi103 ceph-mon[32952]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T23:54:36.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:54:37.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:37 smithi040 ceph-mon[28095]: from='client.? 172.21.15.103:0/3557953366' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:54:37.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:37 smithi080 ceph-mon[32362]: from='client.? 172.21.15.103:0/3557953366' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:54:37.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:37 smithi103 ceph-mon[32952]: from='client.? 172.21.15.103:0/3557953366' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:54:38.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:38 smithi040 ceph-mon[28095]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:38.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:38 smithi080 ceph-mon[32362]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:38.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:38 smithi103 ceph-mon[32952]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:40.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:40 smithi040 ceph-mon[28095]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:40.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:40 smithi080 ceph-mon[32362]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:40.823 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:40 smithi103 ceph-mon[32952]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:42.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:42 smithi040 ceph-mon[28095]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:42.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:42 smithi080 ceph-mon[32362]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:42.832 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:42 smithi103 ceph-mon[32952]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:43.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:43 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T23:54:43.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:43 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:43.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:43 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T23:54:43.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:43 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:43.842 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:43 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T23:54:43.842 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:43 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:44.707 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:44 smithi103 ceph-mon[32952]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:44.707 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:44 smithi103 ceph-mon[32952]: Deploying daemon osd.5 on smithi103 2024-04-26T23:54:44.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:44 smithi040 ceph-mon[28095]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:44.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:44 smithi040 ceph-mon[28095]: Deploying daemon osd.5 on smithi103 2024-04-26T23:54:44.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:44 smithi080 ceph-mon[32362]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:44.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:44 smithi080 ceph-mon[32362]: Deploying daemon osd.5 on smithi103 2024-04-26T23:54:46.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:46 smithi040 ceph-mon[28095]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:46.770 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:46 smithi103 ceph-mon[32952]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:46.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:46 smithi080 ceph-mon[32362]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:48.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:48 smithi103 ceph-mon[32952]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:48.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:48 smithi040 ceph-mon[28095]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:48.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:48 smithi080 ceph-mon[32362]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:49.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:49.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:49.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:49.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:49 smithi080 ceph-mon[32362]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:49.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:49.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:49.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:49.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:49 smithi103 ceph-mon[32952]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:49.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:49.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:49.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:49.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:49 smithi040 ceph-mon[28095]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:52.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.332+0000 7fb7db7fe700 1 -- 172.21.15.103:0/642158625 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb7e4063860 con 0x7fb7d00a3c40 2024-04-26T23:54:52.336 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 5 on host 'smithi103' 2024-04-26T23:54:52.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.334+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/642158625 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7d00a3c40 msgr2=0x7fb7d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:54:52.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.334+0000 7fb7ec6e3700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7d00a3c40 0x7fb7d00a6100 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4003790 tx=0x7fb7d4003860 comp rx=0 tx=0).stop 2024-04-26T23:54:52.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.335+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/642158625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e4104680 msgr2=0x7fb7e4199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:54:52.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.335+0000 7fb7ec6e3700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e4104680 0x7fb7e4199bf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000c950 tx=0x7fb7e000cd10 comp rx=0 tx=0).stop 2024-04-26T23:54:52.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.335+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/642158625 shutdown_connections 2024-04-26T23:54:52.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.335+0000 7fb7ec6e3700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7d00a3c40 0x7fb7d00a6100 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:52.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.335+0000 7fb7ec6e3700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e4105020 0x7fb7e41a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:52.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.335+0000 7fb7ec6e3700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e4103480 0x7fb7e41996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:52.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.335+0000 7fb7ec6e3700 1 --2- 172.21.15.103:0/642158625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e4104680 0x7fb7e4199bf0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:54:52.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.335+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/642158625 >> 172.21.15.103:0/642158625 conn(0x7fb7e40fea20 msgr2=0x7fb7e4107af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:54:52.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.335+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/642158625 shutdown_connections 2024-04-26T23:54:52.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:54:52.335+0000 7fb7ec6e3700 1 -- 172.21.15.103:0/642158625 wait complete. 2024-04-26T23:54:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:52 smithi103 ceph-mon[32952]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:52.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:52 smithi040 ceph-mon[28095]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:52.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:52.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:52.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:52.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:52 smithi080 ceph-mon[32362]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:52.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:52.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:52.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:52.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:52.966 DEBUG:teuthology.orchestra.run.smithi103:osd.5> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T23:54:52.969 INFO:tasks.cephadm:Deploying osd.6 on smithi040 with /dev/vg_nvme/lv_2... 2024-04-26T23:54:52.969 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T23:54:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:54:53.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:53 smithi103 ceph-mon[32952]: from='osd.5 [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T23:54:53.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:53 smithi103 ceph-mon[32952]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T23:54:53.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:53 smithi040 ceph-mon[28095]: from='osd.5 [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T23:54:53.743 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:53 smithi040 ceph-mon[28095]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T23:54:53.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:53 smithi080 ceph-mon[32362]: from='osd.5 [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T23:54:53.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:53 smithi080 ceph-mon[32362]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T23:54:54.636 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:54 smithi040 ceph-mon[28095]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:54.637 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:54 smithi040 ceph-mon[28095]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T23:54:54.637 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:54 smithi040 ceph-mon[28095]: from='osd.5 [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:54:54.637 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:54 smithi040 ceph-mon[28095]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T23:54:54.637 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:54:54.637 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:54 smithi040 ceph-mon[28095]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:54:54.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:54 smithi080 ceph-mon[32362]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:54.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:54 smithi080 ceph-mon[32362]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T23:54:54.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:54 smithi080 ceph-mon[32362]: from='osd.5 [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:54:54.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:54 smithi080 ceph-mon[32362]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T23:54:54.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:54 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:54:54.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:54 smithi080 ceph-mon[32362]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:54:54.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:54 smithi103 ceph-mon[32952]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:54.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:54 smithi103 ceph-mon[32952]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T23:54:54.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:54 smithi103 ceph-mon[32952]: from='osd.5 [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:54:54.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:54 smithi103 ceph-mon[32952]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T23:54:54.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:54:54.850 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:54 smithi103 ceph-mon[32952]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:54:54.850 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 26 23:54:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-26T23:54:54.476+0000 7f4ad3e00700 -1 osd.5 0 waiting for initial osdmap 2024-04-26T23:54:54.850 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 26 23:54:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-26T23:54:54.482+0000 7f4acfbf3700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:54:55.704 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:55 smithi103 ceph-mon[32952]: purged_snaps scrub starts 2024-04-26T23:54:55.704 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:55 smithi103 ceph-mon[32952]: purged_snaps scrub ok 2024-04-26T23:54:55.704 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:55 smithi103 ceph-mon[32952]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T23:54:55.704 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:55 smithi103 ceph-mon[32952]: osdmap e38: 6 total, 5 up, 6 in 2024-04-26T23:54:55.704 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:54:55.704 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:55 smithi103 ceph-mon[32952]: osd.5 [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438] boot 2024-04-26T23:54:55.704 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:55 smithi103 ceph-mon[32952]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T23:54:55.704 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:54:55.710 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:55 smithi040 ceph-mon[28095]: purged_snaps scrub starts 2024-04-26T23:54:55.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:55 smithi040 ceph-mon[28095]: purged_snaps scrub ok 2024-04-26T23:54:55.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:55 smithi040 ceph-mon[28095]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T23:54:55.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:55 smithi040 ceph-mon[28095]: osdmap e38: 6 total, 5 up, 6 in 2024-04-26T23:54:55.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:54:55.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:55 smithi040 ceph-mon[28095]: osd.5 [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438] boot 2024-04-26T23:54:55.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:55 smithi040 ceph-mon[28095]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T23:54:55.711 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:54:55.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:55 smithi080 ceph-mon[32362]: purged_snaps scrub starts 2024-04-26T23:54:55.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:55 smithi080 ceph-mon[32362]: purged_snaps scrub ok 2024-04-26T23:54:55.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:55 smithi080 ceph-mon[32362]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T23:54:55.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:55 smithi080 ceph-mon[32362]: osdmap e38: 6 total, 5 up, 6 in 2024-04-26T23:54:55.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:54:55.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:55 smithi080 ceph-mon[32362]: osd.5 [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438] boot 2024-04-26T23:54:55.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:55 smithi080 ceph-mon[32362]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T23:54:55.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:54:56.583 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T23:54:56.584 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T23:54:56.584 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-26T23:54:56.584 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-26T23:54:56.584 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.03323 s, 316 MB/s 2024-04-26T23:54:56.584 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-26T23:54:56.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:56 smithi080 ceph-mon[32362]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:57.017 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:57.017 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:57.017 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:57.017 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:57.017 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:57.017 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:57.017 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:57.017 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:57.017 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:56 smithi080 ceph-mon[32362]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T23:54:57.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:56 smithi040 ceph-mon[28095]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:57.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:57.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:57.019 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:57.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:57.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:57.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:57.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:57.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:57.020 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:56 smithi040 ceph-mon[28095]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T23:54:57.020 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:56 smithi103 ceph-mon[32952]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:54:57.020 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:57.020 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:57.021 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:57.021 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:54:57.021 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:57.021 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:57.021 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:57.021 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:54:57.021 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:56 smithi103 ceph-mon[32952]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T23:54:57.536 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_2 2024-04-26T23:54:57.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:57 smithi080 ceph-mon[32362]: Detected new or changed devices on smithi103 2024-04-26T23:54:57.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:57 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi103 to 8544M 2024-04-26T23:54:57.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:57 smithi080 ceph-mon[32362]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T23:54:57.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:57 smithi103 ceph-mon[32952]: Detected new or changed devices on smithi103 2024-04-26T23:54:57.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:57 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi103 to 8544M 2024-04-26T23:54:57.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:57 smithi103 ceph-mon[32952]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T23:54:57.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:57 smithi040 ceph-mon[28095]: Detected new or changed devices on smithi103 2024-04-26T23:54:57.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:57 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi103 to 8544M 2024-04-26T23:54:57.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:57 smithi040 ceph-mon[28095]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T23:54:57.992 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:54:58.772 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:58 smithi103 ceph-mon[32952]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:54:58.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:58 smithi080 ceph-mon[32362]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:54:58.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:58 smithi040 ceph-mon[28095]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:54:59.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:59 smithi080 ceph-mon[32362]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T23:54:59.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:54:59 smithi080 ceph-mon[32362]: pgmap v159: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:54:59.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:59 smithi040 ceph-mon[28095]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T23:54:59.814 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:54:59 smithi040 ceph-mon[28095]: pgmap v159: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:54:59.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:59 smithi103 ceph-mon[32952]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T23:54:59.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:54:59 smithi103 ceph-mon[32952]: pgmap v159: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:55:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.503+0000 7f62f1800700 1 -- 172.21.15.40:0/3834414954 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62ec109090 msgr2=0x7f62ec10b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:55:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.503+0000 7f62f1800700 1 --2- 172.21.15.40:0/3834414954 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62ec109090 0x7f62ec10b500 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f62e400b0a0 tx=0x7f62e400b3b0 comp rx=0 tx=0).stop 2024-04-26T23:55:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.503+0000 7f62f1800700 1 -- 172.21.15.40:0/3834414954 shutdown_connections 2024-04-26T23:55:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.503+0000 7f62f1800700 1 --2- 172.21.15.40:0/3834414954 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62ec109090 0x7f62ec10b500 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.503+0000 7f62f1800700 1 --2- 172.21.15.40:0/3834414954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62ec103e30 0x7f62ec106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.503+0000 7f62f1800700 1 --2- 172.21.15.40:0/3834414954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ec106760 0x7f62ec108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.503+0000 7f62f1800700 1 -- 172.21.15.40:0/3834414954 >> 172.21.15.40:0/3834414954 conn(0x7f62ec0fd840 msgr2=0x7f62ec0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:55:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.504+0000 7f62f1800700 1 -- 172.21.15.40:0/3834414954 shutdown_connections 2024-04-26T23:55:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.504+0000 7f62f1800700 1 -- 172.21.15.40:0/3834414954 wait complete. 2024-04-26T23:55:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.505+0000 7f62f1800700 1 Processor -- start 2024-04-26T23:55:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.505+0000 7f62f1800700 1 -- start start 2024-04-26T23:55:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.506+0000 7f62f1800700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62ec103e30 0x7f62ec199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:55:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.506+0000 7f62f1800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62ec106760 0x7f62ec199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:55:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.506+0000 7f62f1800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ec109090 0x7f62ec1a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:55:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.506+0000 7f62f1800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec10ee60 con 0x7f62ec109090 2024-04-26T23:55:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.506+0000 7f62f1800700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec10ece0 con 0x7f62ec103e30 2024-04-26T23:55:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.506+0000 7f62f1800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec10efe0 con 0x7f62ec106760 2024-04-26T23:55:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.506+0000 7f62f0e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ec109090 0x7f62ec1a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:55:00.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.506+0000 7f62ebe00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62ec103e30 0x7f62ec199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:55:00.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.506+0000 7f62eb400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62ec106760 0x7f62ec199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:55:00.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.506+0000 7f62ebe00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62ec103e30 0x7f62ec199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:50684/0 (socket says 172.21.15.40:50684) 2024-04-26T23:55:00.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.506+0000 7f62ebe00700 1 -- 172.21.15.40:0/3457908499 learned_addr learned my addr 172.21.15.40:0/3457908499 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:55:00.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.507+0000 7f62ebe00700 1 -- 172.21.15.40:0/3457908499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62ec106760 msgr2=0x7f62ec199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:55:00.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.507+0000 7f62ebe00700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62ec106760 0x7f62ec199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:00.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.507+0000 7f62ebe00700 1 -- 172.21.15.40:0/3457908499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ec109090 msgr2=0x7f62ec1a09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:55:00.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.507+0000 7f62ebe00700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ec109090 0x7f62ec1a09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:00.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.507+0000 7f62ebe00700 1 -- 172.21.15.40:0/3457908499 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec1a1090 con 0x7f62ec103e30 2024-04-26T23:55:00.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.507+0000 7f62f0e00700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ec109090 0x7f62ec1a09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:55:00.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.507+0000 7f62ebe00700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62ec103e30 0x7f62ec199430 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00e9c0 tx=0x7f62dc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:55:00.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.507+0000 7f62e8c00700 1 -- 172.21.15.40:0/3457908499 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc00cbd0 con 0x7f62ec103e30 2024-04-26T23:55:00.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.507+0000 7f62e8c00700 1 -- 172.21.15.40:0/3457908499 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62dc00ee60 con 0x7f62ec103e30 2024-04-26T23:55:00.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.507+0000 7f62f1800700 1 -- 172.21.15.40:0/3457908499 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec1a1380 con 0x7f62ec103e30 2024-04-26T23:55:00.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.507+0000 7f62e8c00700 1 -- 172.21.15.40:0/3457908499 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc005550 con 0x7f62ec103e30 2024-04-26T23:55:00.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.507+0000 7f62f1800700 1 -- 172.21.15.40:0/3457908499 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec1a18c0 con 0x7f62ec103e30 2024-04-26T23:55:00.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.508+0000 7f62f1800700 1 -- 172.21.15.40:0/3457908499 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ec06a110 con 0x7f62ec103e30 2024-04-26T23:55:00.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.510+0000 7f62e8c00700 1 -- 172.21.15.40:0/3457908499 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f62dc004750 con 0x7f62ec103e30 2024-04-26T23:55:00.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.512+0000 7f62e8c00700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f62d00a3cc0 0x7f62d00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:55:00.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.512+0000 7f62e8c00700 1 -- 172.21.15.40:0/3457908499 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4533+0+0 (secure 0 0 0) 0x7f62dc0cf7b0 con 0x7f62ec103e30 2024-04-26T23:55:00.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.512+0000 7f62eb400700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f62d00a3cc0 0x7f62d00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:55:00.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.513+0000 7f62eb400700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f62d00a3cc0 0x7f62d00a6180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009c60 tx=0x7f62d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:55:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.516+0000 7f62e8c00700 1 -- 172.21.15.40:0/3457908499 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f62dc09cfb0 con 0x7f62ec103e30 2024-04-26T23:55:00.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:00.768+0000 7f62f1800700 1 -- 172.21.15.40:0/3457908499 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f62ec063860 con 0x7f62d00a3cc0 2024-04-26T23:55:00.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:00 smithi040 ceph-mon[28095]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T23:55:00.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:00 smithi040 ceph-mon[28095]: osdmap e43: 6 total, 6 up, 6 in 2024-04-26T23:55:00.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:00 smithi080 ceph-mon[32362]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T23:55:00.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:00 smithi080 ceph-mon[32362]: osdmap e43: 6 total, 6 up, 6 in 2024-04-26T23:55:00.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:00 smithi103 ceph-mon[32952]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T23:55:00.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:00 smithi103 ceph-mon[32952]: osdmap e43: 6 total, 6 up, 6 in 2024-04-26T23:55:01.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:01 smithi080 ceph-mon[32362]: pgmap v161: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:55:01.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:01 smithi080 ceph-mon[32362]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:55:01.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:55:01.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:55:01.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:01.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:01 smithi103 ceph-mon[32952]: pgmap v161: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:55:01.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:01 smithi103 ceph-mon[32952]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:55:01.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:55:01.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:55:01.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:01.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:01 smithi040 ceph-mon[28095]: pgmap v161: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:55:01.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:01 smithi040 ceph-mon[28095]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:55:01.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:55:01.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:55:01.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:03.787 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:03 smithi040 ceph-mon[28095]: pgmap v162: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:55:04.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:03 smithi080 ceph-mon[32362]: pgmap v162: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:55:04.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:03 smithi103 ceph-mon[32952]: pgmap v162: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:55:05.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:04 smithi080 ceph-mon[32362]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T23:55:05.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:04 smithi080 ceph-mon[32362]: Cluster is now healthy 2024-04-26T23:55:05.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:04 smithi103 ceph-mon[32952]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T23:55:05.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:04 smithi103 ceph-mon[32952]: Cluster is now healthy 2024-04-26T23:55:05.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:04 smithi040 ceph-mon[28095]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T23:55:05.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:04 smithi040 ceph-mon[28095]: Cluster is now healthy 2024-04-26T23:55:05.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:05 smithi040 ceph-mon[28095]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-26T23:55:05.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:05 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2283813201' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6180405f-9925-491b-aee8-ecaa9fd39d15"}]: dispatch 2024-04-26T23:55:05.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:05 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2283813201' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6180405f-9925-491b-aee8-ecaa9fd39d15"}]': finished 2024-04-26T23:55:05.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:05 smithi040 ceph-mon[28095]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T23:55:05.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:05 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:06.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:05 smithi080 ceph-mon[32362]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-26T23:55:06.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:05 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2283813201' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6180405f-9925-491b-aee8-ecaa9fd39d15"}]: dispatch 2024-04-26T23:55:06.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:05 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2283813201' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6180405f-9925-491b-aee8-ecaa9fd39d15"}]': finished 2024-04-26T23:55:06.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:05 smithi080 ceph-mon[32362]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T23:55:06.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:05 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:06.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:05 smithi103 ceph-mon[32952]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-26T23:55:06.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:05 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2283813201' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6180405f-9925-491b-aee8-ecaa9fd39d15"}]: dispatch 2024-04-26T23:55:06.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:05 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2283813201' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6180405f-9925-491b-aee8-ecaa9fd39d15"}]': finished 2024-04-26T23:55:06.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:05 smithi103 ceph-mon[32952]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T23:55:06.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:05 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:06.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:06 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2523360017' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:55:07.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:06 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2523360017' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:55:07.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:06 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2523360017' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:55:08.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:07 smithi080 ceph-mon[32362]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T23:55:08.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:07 smithi103 ceph-mon[32952]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T23:55:08.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:07 smithi040 ceph-mon[28095]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T23:55:10.026 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:09 smithi103 ceph-mon[32952]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-26T23:55:10.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:09 smithi080 ceph-mon[32362]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-26T23:55:10.191 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:09 smithi040 ceph-mon[28095]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-26T23:55:11.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:11 smithi040 ceph-mon[28095]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:55:12.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:11 smithi080 ceph-mon[32362]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:55:12.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:11 smithi103 ceph-mon[32952]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:55:14.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:13 smithi080 ceph-mon[32362]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:55:14.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:13 smithi103 ceph-mon[32952]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:55:14.170 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:13 smithi040 ceph-mon[28095]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:55:14.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T23:55:14.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:15.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T23:55:15.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:15.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T23:55:15.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:16.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:15 smithi040 ceph-mon[28095]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:16.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:15 smithi040 ceph-mon[28095]: Deploying daemon osd.6 on smithi040 2024-04-26T23:55:16.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:15 smithi080 ceph-mon[32362]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:16.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:15 smithi080 ceph-mon[32362]: Deploying daemon osd.6 on smithi040 2024-04-26T23:55:16.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:15 smithi103 ceph-mon[32952]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:16.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:15 smithi103 ceph-mon[32952]: Deploying daemon osd.6 on smithi040 2024-04-26T23:55:18.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:17 smithi080 ceph-mon[32362]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:18.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:17 smithi103 ceph-mon[32952]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:18.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:17 smithi040 ceph-mon[28095]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:20.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:19 smithi080 ceph-mon[32362]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:20.070 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:19 smithi040 ceph-mon[28095]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:20.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:19 smithi103 ceph-mon[32952]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:22.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:21 smithi080 ceph-mon[32362]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:22.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:21 smithi103 ceph-mon[32952]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:22.201 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:21 smithi040 ceph-mon[28095]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:24.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:23 smithi080 ceph-mon[32362]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:24.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:24.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:24.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:55:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:23 smithi103 ceph-mon[32952]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:55:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:23 smithi040 ceph-mon[28095]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:55:26.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:25 smithi080 ceph-mon[32362]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:26.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:25 smithi103 ceph-mon[32952]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:26.107 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:25 smithi040 ceph-mon[28095]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:28.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:28 smithi040 ceph-mon[28095]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:28.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:28.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:28.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:28 smithi040 ceph-mon[28095]: from='osd.6 [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T23:55:28.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:28 smithi040 ceph-mon[28095]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T23:55:28.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.779+0000 7f62e8c00700 1 -- 172.21.15.40:0/3457908499 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f62ec063860 con 0x7f62d00a3cc0 2024-04-26T23:55:28.783 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 6 on host 'smithi040' 2024-04-26T23:55:28.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.783+0000 7f62f1800700 1 -- 172.21.15.40:0/3457908499 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f62d00a3cc0 msgr2=0x7f62d00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:55:28.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.783+0000 7f62f1800700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f62d00a3cc0 0x7f62d00a6180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009c60 tx=0x7f62d8009260 comp rx=0 tx=0).stop 2024-04-26T23:55:28.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.783+0000 7f62f1800700 1 -- 172.21.15.40:0/3457908499 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62ec103e30 msgr2=0x7f62ec199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:55:28.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.783+0000 7f62f1800700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62ec103e30 0x7f62ec199430 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00e9c0 tx=0x7f62dc00ecd0 comp rx=0 tx=0).stop 2024-04-26T23:55:28.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.783+0000 7f62f1800700 1 -- 172.21.15.40:0/3457908499 shutdown_connections 2024-04-26T23:55:28.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.783+0000 7f62f1800700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f62d00a3cc0 0x7f62d00a6180 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.783+0000 7f62f1800700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62ec109090 0x7f62ec1a09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.783+0000 7f62f1800700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f62ec103e30 0x7f62ec199430 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.783+0000 7f62f1800700 1 --2- 172.21.15.40:0/3457908499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62ec106760 0x7f62ec199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.784+0000 7f62f1800700 1 -- 172.21.15.40:0/3457908499 >> 172.21.15.40:0/3457908499 conn(0x7f62ec0fd840 msgr2=0x7f62ec1047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:55:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.784+0000 7f62f1800700 1 -- 172.21.15.40:0/3457908499 shutdown_connections 2024-04-26T23:55:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:55:28.784+0000 7f62f1800700 1 -- 172.21.15.40:0/3457908499 wait complete. 2024-04-26T23:55:28.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:28 smithi080 ceph-mon[32362]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:28.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:28.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:28.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:28 smithi080 ceph-mon[32362]: from='osd.6 [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T23:55:28.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:28 smithi080 ceph-mon[32362]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T23:55:28.801 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:28 smithi103 ceph-mon[32952]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:28.801 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:28.801 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:28.802 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:28 smithi103 ceph-mon[32952]: from='osd.6 [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T23:55:28.802 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:28 smithi103 ceph-mon[32952]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T23:55:29.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:29 smithi040 ceph-mon[28095]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T23:55:29.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:29 smithi040 ceph-mon[28095]: from='osd.6 [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:55:29.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:29 smithi040 ceph-mon[28095]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T23:55:29.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:29 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:29.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:29 smithi040 ceph-mon[28095]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:55:29.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:29 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:29.619 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:29 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:29.680 DEBUG:teuthology.orchestra.run.smithi040:osd.6> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T23:55:29.686 INFO:tasks.cephadm:Deploying osd.7 on smithi080 with /dev/vg_nvme/lv_2... 2024-04-26T23:55:29.686 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T23:55:29.713 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:29 smithi080 ceph-mon[32362]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T23:55:29.713 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:29 smithi080 ceph-mon[32362]: from='osd.6 [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:55:29.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:29 smithi080 ceph-mon[32362]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T23:55:29.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:29 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:29.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:29 smithi080 ceph-mon[32362]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:55:29.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:29 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:29.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:29 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:29.817 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:29 smithi103 ceph-mon[32952]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T23:55:29.817 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:29 smithi103 ceph-mon[32952]: from='osd.6 [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:55:29.817 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:29 smithi103 ceph-mon[32952]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T23:55:29.817 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:29 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:29.817 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:29 smithi103 ceph-mon[32952]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:55:29.817 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:29 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:29.817 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:29 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:29.939 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:55:30.444 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 26 23:55:30 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-26T23:55:30.284+0000 7f87d9e00700 -1 osd.6 0 waiting for initial osdmap 2024-04-26T23:55:30.445 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 26 23:55:30 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-26T23:55:30.318+0000 7f87cfc00700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:55:30.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:30 smithi040 ceph-mon[28095]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:30.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:30 smithi040 ceph-mon[28095]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-26T23:55:30.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:30 smithi040 ceph-mon[28095]: osdmap e46: 7 total, 6 up, 7 in 2024-04-26T23:55:30.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:30 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:30.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:30 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:30.719 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:30 smithi040 ceph-mon[28095]: from='osd.6 ' entity='osd.6' 2024-04-26T23:55:30.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:30 smithi080 ceph-mon[32362]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:30.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:30 smithi080 ceph-mon[32362]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-26T23:55:30.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:30 smithi080 ceph-mon[32362]: osdmap e46: 7 total, 6 up, 7 in 2024-04-26T23:55:30.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:30 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:30.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:30 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:30.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:30 smithi080 ceph-mon[32362]: from='osd.6 ' entity='osd.6' 2024-04-26T23:55:30.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:30 smithi103 ceph-mon[32952]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:30.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:30 smithi103 ceph-mon[32952]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-26T23:55:30.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:30 smithi103 ceph-mon[32952]: osdmap e46: 7 total, 6 up, 7 in 2024-04-26T23:55:30.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:30 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:30.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:30 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:30.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:30 smithi103 ceph-mon[32952]: from='osd.6 ' entity='osd.6' 2024-04-26T23:55:31.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:31 smithi040 ceph-mon[28095]: purged_snaps scrub starts 2024-04-26T23:55:31.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:31 smithi040 ceph-mon[28095]: purged_snaps scrub ok 2024-04-26T23:55:31.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:31.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:31 smithi040 ceph-mon[28095]: osd.6 [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516] boot 2024-04-26T23:55:31.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:31 smithi040 ceph-mon[28095]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T23:55:31.720 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:31.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:31 smithi080 ceph-mon[32362]: purged_snaps scrub starts 2024-04-26T23:55:31.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:31 smithi080 ceph-mon[32362]: purged_snaps scrub ok 2024-04-26T23:55:31.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:31.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:31 smithi080 ceph-mon[32362]: osd.6 [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516] boot 2024-04-26T23:55:31.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:31 smithi080 ceph-mon[32362]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T23:55:31.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:31.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:31 smithi103 ceph-mon[32952]: purged_snaps scrub starts 2024-04-26T23:55:31.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:31 smithi103 ceph-mon[32952]: purged_snaps scrub ok 2024-04-26T23:55:31.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:31.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:31 smithi103 ceph-mon[32952]: osd.6 [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516] boot 2024-04-26T23:55:31.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:31 smithi103 ceph-mon[32952]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T23:55:31.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:55:31.976 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T23:55:31.976 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T23:55:31.976 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-04-26T23:55:31.976 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-04-26T23:55:31.976 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0165966 s, 632 MB/s 2024-04-26T23:55:31.976 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-04-26T23:55:32.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:32 smithi080 ceph-mon[32362]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:32.685 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi080:vg_nvme/lv_2 2024-04-26T23:55:32.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:32 smithi040 ceph-mon[28095]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:32.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:32 smithi103 ceph-mon[32952]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:55:32.969 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:55:33.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:33 smithi040 ceph-mon[28095]: osdmap e48: 7 total, 7 up, 7 in 2024-04-26T23:55:33.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:33 smithi080 ceph-mon[32362]: osdmap e48: 7 total, 7 up, 7 in 2024-04-26T23:55:33.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:33 smithi103 ceph-mon[32952]: osdmap e48: 7 total, 7 up, 7 in 2024-04-26T23:55:34.674 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.670+0000 7fc49039b700 1 -- 172.21.15.80:0/2017284454 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc488101730 msgr2=0x7fc488101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:55:34.674 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.670+0000 7fc49039b700 1 --2- 172.21.15.80:0/2017284454 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc488101730 0x7fc488101b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc484009990 tx=0x7fc484009ca0 comp rx=0 tx=0).stop 2024-04-26T23:55:34.674 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.670+0000 7fc49039b700 1 -- 172.21.15.80:0/2017284454 shutdown_connections 2024-04-26T23:55:34.674 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.670+0000 7fc49039b700 1 --2- 172.21.15.80:0/2017284454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc488105150 0x7fc48810ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:34.674 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.670+0000 7fc49039b700 1 --2- 172.21.15.80:0/2017284454 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc488101730 0x7fc488101b50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:34.674 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.670+0000 7fc49039b700 1 --2- 172.21.15.80:0/2017284454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4881047b0 0x7fc488104c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:34.674 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.670+0000 7fc49039b700 1 -- 172.21.15.80:0/2017284454 >> 172.21.15.80:0/2017284454 conn(0x7fc4880fd0b0 msgr2=0x7fc4880ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:55:34.674 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.670+0000 7fc49039b700 1 -- 172.21.15.80:0/2017284454 shutdown_connections 2024-04-26T23:55:34.674 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.670+0000 7fc49039b700 1 -- 172.21.15.80:0/2017284454 wait complete. 2024-04-26T23:55:34.675 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.670+0000 7fc49039b700 1 Processor -- start 2024-04-26T23:55:34.675 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.670+0000 7fc49039b700 1 -- start start 2024-04-26T23:55:34.675 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc49039b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc488101730 0x7fc48819b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:55:34.675 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc49039b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4881047b0 0x7fc48819bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:55:34.675 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc49039b700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc488105150 0x7fc4881a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:55:34.675 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc49039b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc48810ef80 con 0x7fc4881047b0 2024-04-26T23:55:34.675 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc49039b700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fc48810ee00 con 0x7fc488105150 2024-04-26T23:55:34.675 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc49039b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc48810f100 con 0x7fc488101730 2024-04-26T23:55:34.675 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc48e938700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc488105150 0x7fc4881a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:55:34.675 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc48e938700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc488105150 0x7fc4881a2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:37654/0 (socket says 172.21.15.80:37654) 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc48e938700 1 -- 172.21.15.80:0/1797854041 learned_addr learned my addr 172.21.15.80:0/1797854041 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc48e137700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc488101730 0x7fc48819b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc48e938700 1 -- 172.21.15.80:0/1797854041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc488101730 msgr2=0x7fc48819b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc48e938700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc488101730 0x7fc48819b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc48e938700 1 -- 172.21.15.80:0/1797854041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4881047b0 msgr2=0x7fc48819bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc48e938700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4881047b0 0x7fc48819bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc48e938700 1 -- 172.21.15.80:0/1797854041 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4881a3470 con 0x7fc488105150 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc48e938700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc488105150 0x7fc4881a2dd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc48000ef30 tx=0x7fc48000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc47f7fe700 1 -- 172.21.15.80:0/1797854041 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4800092e0 con 0x7fc488105150 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc47f7fe700 1 -- 172.21.15.80:0/1797854041 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc480004bb0 con 0x7fc488105150 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc49039b700 1 -- 172.21.15.80:0/1797854041 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4881a3700 con 0x7fc488105150 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc47f7fe700 1 -- 172.21.15.80:0/1797854041 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480004280 con 0x7fc488105150 2024-04-26T23:55:34.676 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.671+0000 7fc49039b700 1 -- 172.21.15.80:0/1797854041 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4881a3cc0 con 0x7fc488105150 2024-04-26T23:55:34.677 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.672+0000 7fc49039b700 1 -- 172.21.15.80:0/1797854041 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc48806a110 con 0x7fc488105150 2024-04-26T23:55:34.677 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.673+0000 7fc47f7fe700 1 -- 172.21.15.80:0/1797854041 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc48001b030 con 0x7fc488105150 2024-04-26T23:55:34.677 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.674+0000 7fc47f7fe700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4740a3c40 0x7fc4740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:55:34.677 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.674+0000 7fc47f7fe700 1 -- 172.21.15.80:0/1797854041 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7fc4800cfb90 con 0x7fc488105150 2024-04-26T23:55:34.677 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.675+0000 7fc48e137700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4740a3c40 0x7fc4740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:55:34.678 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.676+0000 7fc48e137700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4740a3c40 0x7fc4740a6100 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc48400b2e0 tx=0x7fc484005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:55:34.680 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.678+0000 7fc47f7fe700 1 -- 172.21.15.80:0/1797854041 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc480022020 con 0x7fc488105150 2024-04-26T23:55:34.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:34 smithi080 ceph-mon[32362]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:34.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:34 smithi103 ceph-mon[32952]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:34.898 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:34.895+0000 7fc49039b700 1 -- 172.21.15.80:0/1797854041 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc48810b6f0 con 0x7fc4740a3c40 2024-04-26T23:55:34.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:34 smithi040 ceph-mon[28095]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:35.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:35.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: Detected new or changed devices on smithi040 2024-04-26T23:55:35.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:35.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:35.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:55:35.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:55:35.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:55:35.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi040 to 5698M 2024-04-26T23:55:35.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:35.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:35.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:55:35.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:35.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:55:35.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:55:35.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:55:35.803 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:35.803 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:55:35.803 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:35.803 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:55:35.803 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:35 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:36.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:36.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: Detected new or changed devices on smithi040 2024-04-26T23:55:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:55:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:55:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:55:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi040 to 5698M 2024-04-26T23:55:36.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:55:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:55:36.101 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:55:36.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:55:36.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:36.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:55:36.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:36.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:55:36.102 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:35 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:36.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:36.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: Detected new or changed devices on smithi040 2024-04-26T23:55:36.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:36.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:36.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:55:36.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:55:36.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:55:36.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi040 to 5698M 2024-04-26T23:55:36.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:36.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:36.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:55:36.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:36.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:55:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:55:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:55:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:55:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:55:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:35 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:38.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:37 smithi080 ceph-mon[32362]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:38.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:37 smithi080 ceph-mon[32362]: from='client.? 172.21.15.80:0/4026096352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cd5c915-ae45-499a-8910-8619855fec2f"}]: dispatch 2024-04-26T23:55:38.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:37 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cd5c915-ae45-499a-8910-8619855fec2f"}]: dispatch 2024-04-26T23:55:38.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:37 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cd5c915-ae45-499a-8910-8619855fec2f"}]': finished 2024-04-26T23:55:38.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:37 smithi080 ceph-mon[32362]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T23:55:38.052 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:37 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:55:38.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:37 smithi103 ceph-mon[32952]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:38.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:37 smithi103 ceph-mon[32952]: from='client.? 172.21.15.80:0/4026096352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cd5c915-ae45-499a-8910-8619855fec2f"}]: dispatch 2024-04-26T23:55:38.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:37 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cd5c915-ae45-499a-8910-8619855fec2f"}]: dispatch 2024-04-26T23:55:38.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:37 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cd5c915-ae45-499a-8910-8619855fec2f"}]': finished 2024-04-26T23:55:38.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:37 smithi103 ceph-mon[32952]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T23:55:38.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:37 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:55:38.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:37 smithi040 ceph-mon[28095]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:38.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:37 smithi040 ceph-mon[28095]: from='client.? 172.21.15.80:0/4026096352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cd5c915-ae45-499a-8910-8619855fec2f"}]: dispatch 2024-04-26T23:55:38.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:37 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cd5c915-ae45-499a-8910-8619855fec2f"}]: dispatch 2024-04-26T23:55:38.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:37 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cd5c915-ae45-499a-8910-8619855fec2f"}]': finished 2024-04-26T23:55:38.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:37 smithi040 ceph-mon[28095]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T23:55:38.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:37 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:55:39.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:38 smithi080 ceph-mon[32362]: from='client.? 172.21.15.80:0/4035670093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:55:39.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:38 smithi103 ceph-mon[32952]: from='client.? 172.21.15.80:0/4035670093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:55:39.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:38 smithi040 ceph-mon[28095]: from='client.? 172.21.15.80:0/4035670093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:55:40.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:39 smithi080 ceph-mon[32362]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:40.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:39 smithi103 ceph-mon[32952]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:40.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:39 smithi040 ceph-mon[28095]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:42.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:41 smithi080 ceph-mon[32362]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:42.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:41 smithi103 ceph-mon[32952]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:42.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:41 smithi040 ceph-mon[28095]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:43.870 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:43 smithi080 ceph-mon[32362]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:44.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:43 smithi103 ceph-mon[32952]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:44.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:43 smithi040 ceph-mon[28095]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:46.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:45 smithi080 ceph-mon[32362]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:46.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T23:55:46.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:46.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:45 smithi080 ceph-mon[32362]: Deploying daemon osd.7 on smithi080 2024-04-26T23:55:46.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:45 smithi103 ceph-mon[32952]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:46.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T23:55:46.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:46.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:45 smithi103 ceph-mon[32952]: Deploying daemon osd.7 on smithi080 2024-04-26T23:55:46.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:45 smithi040 ceph-mon[28095]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:46.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T23:55:46.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:46.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:45 smithi040 ceph-mon[28095]: Deploying daemon osd.7 on smithi080 2024-04-26T23:55:47.939 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:47 smithi080 ceph-mon[32362]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:48.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:47 smithi103 ceph-mon[32952]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:48.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:47 smithi040 ceph-mon[28095]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:50.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:49 smithi080 ceph-mon[32362]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:50.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:49 smithi103 ceph-mon[32952]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:50.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:49 smithi040 ceph-mon[28095]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:52.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:52 smithi080 ceph-mon[32362]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:52.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:52.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:52.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:55:52.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:52 smithi103 ceph-mon[32952]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:52.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:52.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:52.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:55:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:52 smithi040 ceph-mon[28095]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:55:54.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:54 smithi080 ceph-mon[32362]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:54.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:54 smithi103 ceph-mon[32952]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:54.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:54 smithi040 ceph-mon[28095]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:54.784 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.782+0000 7fc47f7fe700 1 -- 172.21.15.80:0/1797854041 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc48810b6f0 con 0x7fc4740a3c40 2024-04-26T23:55:54.784 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 7 on host 'smithi080' 2024-04-26T23:55:54.787 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.784+0000 7fc49039b700 1 -- 172.21.15.80:0/1797854041 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4740a3c40 msgr2=0x7fc4740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:55:54.787 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.784+0000 7fc49039b700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4740a3c40 0x7fc4740a6100 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc48400b2e0 tx=0x7fc484005fb0 comp rx=0 tx=0).stop 2024-04-26T23:55:54.787 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.784+0000 7fc49039b700 1 -- 172.21.15.80:0/1797854041 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc488105150 msgr2=0x7fc4881a2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:55:54.787 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.784+0000 7fc49039b700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc488105150 0x7fc4881a2dd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc48000ef30 tx=0x7fc48000c3a0 comp rx=0 tx=0).stop 2024-04-26T23:55:54.787 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.784+0000 7fc49039b700 1 -- 172.21.15.80:0/1797854041 shutdown_connections 2024-04-26T23:55:54.787 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.785+0000 7fc49039b700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4740a3c40 0x7fc4740a6100 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:54.787 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.785+0000 7fc49039b700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc488105150 0x7fc4881a2dd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:54.787 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.785+0000 7fc49039b700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc488101730 0x7fc48819b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:54.787 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.785+0000 7fc49039b700 1 --2- 172.21.15.80:0/1797854041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4881047b0 0x7fc48819bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:55:54.787 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.785+0000 7fc49039b700 1 -- 172.21.15.80:0/1797854041 >> 172.21.15.80:0/1797854041 conn(0x7fc4880fd0b0 msgr2=0x7fc488109fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:55:54.787 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.785+0000 7fc49039b700 1 -- 172.21.15.80:0/1797854041 shutdown_connections 2024-04-26T23:55:54.788 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:55:54.785+0000 7fc49039b700 1 -- 172.21.15.80:0/1797854041 wait complete. 2024-04-26T23:55:55.497 DEBUG:teuthology.orchestra.run.smithi080:osd.7> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T23:55:55.499 INFO:tasks.cephadm:Deploying osd.8 on smithi103 with /dev/vg_nvme/lv_2... 2024-04-26T23:55:55.499 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T23:55:55.748 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:55:55.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:55 smithi103 ceph-mon[32952]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:55.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:55.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:55.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:55 smithi103 ceph-mon[32952]: from='osd.7 [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T23:55:55.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:55 smithi103 ceph-mon[32952]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T23:55:55.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:55.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:56.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:55 smithi080 ceph-mon[32362]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:56.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:56.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:56.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:55 smithi080 ceph-mon[32362]: from='osd.7 [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T23:55:56.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:55 smithi080 ceph-mon[32362]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T23:55:56.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:56.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:56.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:55 smithi040 ceph-mon[28095]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:56.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:56.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:56.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:55 smithi040 ceph-mon[28095]: from='osd.7 [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T23:55:56.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:55 smithi040 ceph-mon[28095]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T23:55:56.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:56.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:55:56.824 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:56 smithi103 ceph-mon[32952]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T23:55:56.824 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:56 smithi103 ceph-mon[32952]: from='osd.7 [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:55:56.825 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:56 smithi103 ceph-mon[32952]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T23:55:56.825 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:55:56.825 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:56 smithi103 ceph-mon[32952]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:55:57.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:56 smithi080 ceph-mon[32362]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T23:55:57.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:56 smithi080 ceph-mon[32362]: from='osd.7 [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:55:57.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:56 smithi080 ceph-mon[32362]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T23:55:57.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:55:57.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:56 smithi080 ceph-mon[32362]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:55:57.051 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 26 23:55:56 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-26T23:55:56.759+0000 7f7bef76b700 -1 osd.7 0 waiting for initial osdmap 2024-04-26T23:55:57.051 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 26 23:55:56 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-26T23:55:56.767+0000 7f7beb55e700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:55:57.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:56 smithi040 ceph-mon[28095]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T23:55:57.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:56 smithi040 ceph-mon[28095]: from='osd.7 [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:55:57.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:56 smithi040 ceph-mon[28095]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T23:55:57.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:55:57.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:56 smithi040 ceph-mon[28095]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:55:57.670 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T23:55:57.670 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T23:55:57.670 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-04-26T23:55:57.670 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-04-26T23:55:57.670 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128205 s, 818 MB/s 2024-04-26T23:55:57.670 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 2024-04-26T23:55:57.670 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-04-26T23:55:58.014 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:57 smithi103 ceph-mon[32952]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:58.015 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:57 smithi103 ceph-mon[32952]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-04-26T23:55:58.015 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:57 smithi103 ceph-mon[32952]: osdmap e51: 8 total, 7 up, 8 in 2024-04-26T23:55:58.015 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:55:58.015 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:57 smithi103 ceph-mon[32952]: osd.7 [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332] boot 2024-04-26T23:55:58.015 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:57 smithi103 ceph-mon[32952]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T23:55:58.015 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:55:58.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:57 smithi080 ceph-mon[32362]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:58.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:57 smithi080 ceph-mon[32362]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-04-26T23:55:58.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:57 smithi080 ceph-mon[32362]: osdmap e51: 8 total, 7 up, 8 in 2024-04-26T23:55:58.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:55:58.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:57 smithi080 ceph-mon[32362]: osd.7 [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332] boot 2024-04-26T23:55:58.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:57 smithi080 ceph-mon[32362]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T23:55:58.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:55:58.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:57 smithi040 ceph-mon[28095]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:55:58.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:57 smithi040 ceph-mon[28095]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-04-26T23:55:58.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:57 smithi040 ceph-mon[28095]: osdmap e51: 8 total, 7 up, 8 in 2024-04-26T23:55:58.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:55:58.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:57 smithi040 ceph-mon[28095]: osd.7 [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332] boot 2024-04-26T23:55:58.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:57 smithi040 ceph-mon[28095]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T23:55:58.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:55:58.342 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_2 2024-04-26T23:55:58.579 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:55:58.944 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:58 smithi103 ceph-mon[32952]: purged_snaps scrub starts 2024-04-26T23:55:58.944 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:58 smithi103 ceph-mon[32952]: purged_snaps scrub ok 2024-04-26T23:55:58.944 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:55:58 smithi103 ceph-mon[32952]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T23:55:59.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:58 smithi080 ceph-mon[32362]: purged_snaps scrub starts 2024-04-26T23:55:59.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:58 smithi080 ceph-mon[32362]: purged_snaps scrub ok 2024-04-26T23:55:59.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:55:58 smithi080 ceph-mon[32362]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T23:55:59.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:58 smithi040 ceph-mon[28095]: purged_snaps scrub starts 2024-04-26T23:55:59.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:58 smithi040 ceph-mon[28095]: purged_snaps scrub ok 2024-04-26T23:55:59.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:55:58 smithi040 ceph-mon[28095]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T23:56:00.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.203+0000 7fa92b484700 1 -- 172.21.15.103:0/307873073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa924104680 msgr2=0x7fa924104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:56:00.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.203+0000 7fa92b484700 1 --2- 172.21.15.103:0/307873073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa924104680 0x7fa924104ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa914009990 tx=0x7fa914009ca0 comp rx=0 tx=0).stop 2024-04-26T23:56:00.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.203+0000 7fa92b484700 1 -- 172.21.15.103:0/307873073 shutdown_connections 2024-04-26T23:56:00.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.203+0000 7fa92b484700 1 --2- 172.21.15.103:0/307873073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa924105020 0x7fa92410b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:00.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.203+0000 7fa92b484700 1 --2- 172.21.15.103:0/307873073 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa924103480 0x7fa9241038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:00.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.203+0000 7fa92b484700 1 --2- 172.21.15.103:0/307873073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa924104680 0x7fa924104ae0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:00.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.203+0000 7fa92b484700 1 -- 172.21.15.103:0/307873073 >> 172.21.15.103:0/307873073 conn(0x7fa9240fea20 msgr2=0x7fa924100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:56:00.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa92b484700 1 -- 172.21.15.103:0/307873073 shutdown_connections 2024-04-26T23:56:00.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa92b484700 1 -- 172.21.15.103:0/307873073 wait complete. 2024-04-26T23:56:00.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa92b484700 1 Processor -- start 2024-04-26T23:56:00.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa92b484700 1 -- start start 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa92b484700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa924103480 0x7fa9240771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa92b484700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa924104680 0x7fa924075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa92b484700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa924105020 0x7fa924075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa92b484700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa92410ef70 con 0x7fa924104680 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa92b484700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa92410edf0 con 0x7fa924103480 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa92b484700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa92410f0f0 con 0x7fa924105020 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa929a21700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa924105020 0x7fa924075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa929a21700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa924105020 0x7fa924075d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45094/0 (socket says 172.21.15.103:45094) 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.204+0000 7fa929a21700 1 -- 172.21.15.103:0/1543809003 learned_addr learned my addr 172.21.15.103:0/1543809003 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.205+0000 7fa929220700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa924103480 0x7fa9240771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.205+0000 7fa929a21700 1 -- 172.21.15.103:0/1543809003 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa924103480 msgr2=0x7fa9240771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.205+0000 7fa929a21700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa924103480 0x7fa9240771c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.205+0000 7fa929a21700 1 -- 172.21.15.103:0/1543809003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa924104680 msgr2=0x7fa924075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:56:00.209 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.205+0000 7fa929a21700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa924104680 0x7fa924075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:00.210 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.205+0000 7fa929a21700 1 -- 172.21.15.103:0/1543809003 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa924076600 con 0x7fa924105020 2024-04-26T23:56:00.210 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.205+0000 7fa929a21700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa924105020 0x7fa924075d50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00b970 tx=0x7fa91c00bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:56:00.210 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.205+0000 7fa91a7fc700 1 -- 172.21.15.103:0/1543809003 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91c019070 con 0x7fa924105020 2024-04-26T23:56:00.210 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.205+0000 7fa91a7fc700 1 -- 172.21.15.103:0/1543809003 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa91c014440 con 0x7fa924105020 2024-04-26T23:56:00.210 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.205+0000 7fa92b484700 1 -- 172.21.15.103:0/1543809003 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9241a3710 con 0x7fa924105020 2024-04-26T23:56:00.210 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.205+0000 7fa91a7fc700 1 -- 172.21.15.103:0/1543809003 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91c00ce60 con 0x7fa924105020 2024-04-26T23:56:00.210 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.205+0000 7fa92b484700 1 -- 172.21.15.103:0/1543809003 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9241a3c20 con 0x7fa924105020 2024-04-26T23:56:00.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.206+0000 7fa92b484700 1 -- 172.21.15.103:0/1543809003 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa92406a110 con 0x7fa924105020 2024-04-26T23:56:00.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.207+0000 7fa91a7fc700 1 -- 172.21.15.103:0/1543809003 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa91c01c070 con 0x7fa924105020 2024-04-26T23:56:00.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.208+0000 7fa91a7fc700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa9100a3c40 0x7fa9100a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:56:00.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.208+0000 7fa91a7fc700 1 -- 172.21.15.103:0/1543809003 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7fa91c0cecb0 con 0x7fa924105020 2024-04-26T23:56:00.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.208+0000 7fa929220700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa9100a3c40 0x7fa9100a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:56:00.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.209+0000 7fa929220700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa9100a3c40 0x7fa9100a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa920005fd0 tx=0x7fa920005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:56:00.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.213+0000 7fa91a7fc700 1 -- 172.21.15.103:0/1543809003 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa91c09c1d0 con 0x7fa924105020 2024-04-26T23:56:00.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:00.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: Detected new or changed devices on smithi080 2024-04-26T23:56:00.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:00.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:00.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:00.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi080 to 5696M 2024-04-26T23:56:00.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:00.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:00.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:00.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:00.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:00.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:00 smithi103 ceph-mon[32952]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T23:56:00.423 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:00.421+0000 7fa92b484700 1 -- 172.21.15.103:0/1543809003 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa924063860 con 0x7fa9100a3c40 2024-04-26T23:56:00.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:00.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: Detected new or changed devices on smithi080 2024-04-26T23:56:00.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:00.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:00.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:00.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi080 to 5696M 2024-04-26T23:56:00.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:00.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:00.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:00.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:00.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:00.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:00 smithi040 ceph-mon[28095]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T23:56:00.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:00.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: Detected new or changed devices on smithi080 2024-04-26T23:56:00.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:00.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:00.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:00.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi080 to 5696M 2024-04-26T23:56:00.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:00.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:00.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:00.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:00.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:00.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:00.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:00 smithi080 ceph-mon[32362]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T23:56:01.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:01 smithi103 ceph-mon[32952]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:56:01.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:56:01.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:56:01.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:01.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:01 smithi103 ceph-mon[32952]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T23:56:01.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:01 smithi040 ceph-mon[28095]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:56:01.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:56:01.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:56:01.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:01.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:01 smithi040 ceph-mon[28095]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T23:56:01.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:01 smithi080 ceph-mon[32362]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:56:01.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:56:01.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:56:01.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:01.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:01 smithi080 ceph-mon[32362]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T23:56:02.328 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:02 smithi103 ceph-mon[32952]: pgmap v202: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:02.328 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:02 smithi103 ceph-mon[32952]: osdmap e56: 8 total, 8 up, 8 in 2024-04-26T23:56:02.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:02 smithi040 ceph-mon[28095]: pgmap v202: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:02.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:02 smithi040 ceph-mon[28095]: osdmap e56: 8 total, 8 up, 8 in 2024-04-26T23:56:02.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:02 smithi080 ceph-mon[32362]: pgmap v202: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:02.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:02 smithi080 ceph-mon[32362]: osdmap e56: 8 total, 8 up, 8 in 2024-04-26T23:56:04.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:04 smithi040 ceph-mon[28095]: pgmap v205: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:04.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:04 smithi040 ceph-mon[28095]: from='client.? 172.21.15.103:0/4248462238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63176b7b-984c-4aac-be69-3fe6d316ac7e"}]: dispatch 2024-04-26T23:56:04.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:04 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63176b7b-984c-4aac-be69-3fe6d316ac7e"}]: dispatch 2024-04-26T23:56:04.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:04 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63176b7b-984c-4aac-be69-3fe6d316ac7e"}]': finished 2024-04-26T23:56:04.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:04 smithi040 ceph-mon[28095]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T23:56:04.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:04.470 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:04 smithi040 ceph-mon[28095]: from='client.? 172.21.15.103:0/315058540' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:56:04.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:04 smithi080 ceph-mon[32362]: pgmap v205: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:04.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:04 smithi080 ceph-mon[32362]: from='client.? 172.21.15.103:0/4248462238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63176b7b-984c-4aac-be69-3fe6d316ac7e"}]: dispatch 2024-04-26T23:56:04.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:04 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63176b7b-984c-4aac-be69-3fe6d316ac7e"}]: dispatch 2024-04-26T23:56:04.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:04 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63176b7b-984c-4aac-be69-3fe6d316ac7e"}]': finished 2024-04-26T23:56:04.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:04 smithi080 ceph-mon[32362]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T23:56:04.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:04.552 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:04 smithi080 ceph-mon[32362]: from='client.? 172.21.15.103:0/315058540' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:56:04.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:04 smithi103 ceph-mon[32952]: pgmap v205: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:04.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:04 smithi103 ceph-mon[32952]: from='client.? 172.21.15.103:0/4248462238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63176b7b-984c-4aac-be69-3fe6d316ac7e"}]: dispatch 2024-04-26T23:56:04.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:04 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63176b7b-984c-4aac-be69-3fe6d316ac7e"}]: dispatch 2024-04-26T23:56:04.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:04 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63176b7b-984c-4aac-be69-3fe6d316ac7e"}]': finished 2024-04-26T23:56:04.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:04 smithi103 ceph-mon[32952]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T23:56:04.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:04.600 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:04 smithi103 ceph-mon[32952]: from='client.? 172.21.15.103:0/315058540' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:56:06.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:06 smithi040 ceph-mon[28095]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:06.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:06 smithi080 ceph-mon[32362]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:06.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:06 smithi103 ceph-mon[32952]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:08.387 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:08 smithi103 ceph-mon[32952]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:08.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:08 smithi040 ceph-mon[28095]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:08.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:08 smithi080 ceph-mon[32362]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:10.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:10 smithi040 ceph-mon[28095]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:10.516 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:10 smithi103 ceph-mon[32952]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:10.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:10 smithi080 ceph-mon[32362]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:11.444 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:11 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T23:56:11.444 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:11 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:11.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:11 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T23:56:11.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:11 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:11.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:11 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T23:56:11.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:11 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:12.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:12 smithi103 ceph-mon[32952]: Deploying daemon osd.8 on smithi103 2024-04-26T23:56:12.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:12 smithi103 ceph-mon[32952]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:12.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:12 smithi040 ceph-mon[28095]: Deploying daemon osd.8 on smithi103 2024-04-26T23:56:12.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:12 smithi040 ceph-mon[28095]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:12.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:12 smithi080 ceph-mon[32362]: Deploying daemon osd.8 on smithi103 2024-04-26T23:56:12.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:12 smithi080 ceph-mon[32362]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:14.330 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:14 smithi103 ceph-mon[32952]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:14.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:14 smithi040 ceph-mon[28095]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:14.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:14 smithi080 ceph-mon[32362]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:16.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:16 smithi103 ceph-mon[32952]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:16.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:16 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:16.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:16 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:16.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:16 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:16.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:16 smithi040 ceph-mon[28095]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:16.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:16 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:16.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:16 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:16.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:16 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:16.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:16 smithi080 ceph-mon[32362]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:16.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:16 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:16.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:16 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:16.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:16 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:18.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:18 smithi103 ceph-mon[32952]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:18.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:18 smithi040 ceph-mon[28095]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:18.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:18 smithi080 ceph-mon[32362]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:19.761 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 8 on host 'smithi103' 2024-04-26T23:56:19.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.759+0000 7fa91a7fc700 1 -- 172.21.15.103:0/1543809003 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa924063860 con 0x7fa9100a3c40 2024-04-26T23:56:19.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.762+0000 7fa92b484700 1 -- 172.21.15.103:0/1543809003 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa9100a3c40 msgr2=0x7fa9100a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:56:19.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.762+0000 7fa92b484700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa9100a3c40 0x7fa9100a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa920005fd0 tx=0x7fa920005e50 comp rx=0 tx=0).stop 2024-04-26T23:56:19.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.762+0000 7fa92b484700 1 -- 172.21.15.103:0/1543809003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa924105020 msgr2=0x7fa924075d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:56:19.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.762+0000 7fa92b484700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa924105020 0x7fa924075d50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa91c00b970 tx=0x7fa91c00bd30 comp rx=0 tx=0).stop 2024-04-26T23:56:19.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.762+0000 7fa92b484700 1 -- 172.21.15.103:0/1543809003 shutdown_connections 2024-04-26T23:56:19.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.762+0000 7fa92b484700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa9100a3c40 0x7fa9100a6100 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:19.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.762+0000 7fa92b484700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa924105020 0x7fa924075d50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:19.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.762+0000 7fa92b484700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa924103480 0x7fa9240771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:19.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.762+0000 7fa92b484700 1 --2- 172.21.15.103:0/1543809003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa924104680 0x7fa924075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:19.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.762+0000 7fa92b484700 1 -- 172.21.15.103:0/1543809003 >> 172.21.15.103:0/1543809003 conn(0x7fa9240fea20 msgr2=0x7fa9241079d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:56:19.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.762+0000 7fa92b484700 1 -- 172.21.15.103:0/1543809003 shutdown_connections 2024-04-26T23:56:19.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:56:19.762+0000 7fa92b484700 1 -- 172.21.15.103:0/1543809003 wait complete. 2024-04-26T23:56:19.952 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:19 smithi103 ceph-mon[32952]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:19.953 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:19.953 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:19.953 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:19.953 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:19.953 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:19.953 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:19.953 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:19.953 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:19 smithi040 ceph-mon[28095]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:20.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:20.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:20.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:20.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:19 smithi080 ceph-mon[32362]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:20.462 DEBUG:teuthology.orchestra.run.smithi103:osd.8> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.8.service 2024-04-26T23:56:20.464 INFO:tasks.cephadm:Deploying osd.9 on smithi040 with /dev/vg_nvme/lv_1... 2024-04-26T23:56:20.464 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T23:56:20.935 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:56:21.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:20 smithi103 ceph-mon[32952]: from='osd.8 [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T23:56:21.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:20 smithi103 ceph-mon[32952]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T23:56:21.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:20 smithi040 ceph-mon[28095]: from='osd.8 [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T23:56:21.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:20 smithi040 ceph-mon[28095]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T23:56:21.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:20 smithi080 ceph-mon[32362]: from='osd.8 [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T23:56:21.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:20 smithi080 ceph-mon[32362]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T23:56:22.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:21 smithi040 ceph-mon[28095]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:22.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:21 smithi040 ceph-mon[28095]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T23:56:22.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:21 smithi040 ceph-mon[28095]: from='osd.8 [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:56:22.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:21 smithi040 ceph-mon[28095]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T23:56:22.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:21 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:22.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:21 smithi040 ceph-mon[28095]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:56:22.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:21 smithi080 ceph-mon[32362]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:22.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:21 smithi080 ceph-mon[32362]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T23:56:22.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:21 smithi080 ceph-mon[32362]: from='osd.8 [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:56:22.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:21 smithi080 ceph-mon[32362]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T23:56:22.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:21 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:22.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:21 smithi080 ceph-mon[32362]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:56:22.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:21 smithi103 ceph-mon[32952]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:22.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:21 smithi103 ceph-mon[32952]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T23:56:22.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:21 smithi103 ceph-mon[32952]: from='osd.8 [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:56:22.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:21 smithi103 ceph-mon[32952]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T23:56:22.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:21 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:22.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:21 smithi103 ceph-mon[32952]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:56:22.938 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 26 23:56:22 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-26T23:56:22.647+0000 7ff2f1de3700 -1 osd.8 0 waiting for initial osdmap 2024-04-26T23:56:22.939 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 26 23:56:22 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-26T23:56:22.659+0000 7ff2e9bce700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:56:22.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:22 smithi040 ceph-mon[28095]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T23:56:22.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:22 smithi040 ceph-mon[28095]: osdmap e59: 9 total, 8 up, 9 in 2024-04-26T23:56:22.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:22 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:22.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:22 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:22.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:22 smithi040 ceph-mon[28095]: from='osd.8 ' entity='osd.8' 2024-04-26T23:56:23.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:22 smithi080 ceph-mon[32362]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T23:56:23.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:22 smithi080 ceph-mon[32362]: osdmap e59: 9 total, 8 up, 9 in 2024-04-26T23:56:23.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:22 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:23.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:22 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:23.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:22 smithi080 ceph-mon[32362]: from='osd.8 ' entity='osd.8' 2024-04-26T23:56:23.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:22 smithi103 ceph-mon[32952]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T23:56:23.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:22 smithi103 ceph-mon[32952]: osdmap e59: 9 total, 8 up, 9 in 2024-04-26T23:56:23.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:22 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:23.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:22 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:23.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:22 smithi103 ceph-mon[32952]: from='osd.8 ' entity='osd.8' 2024-04-26T23:56:24.035 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T23:56:24.035 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T23:56:24.035 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-04-26T23:56:24.035 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-04-26T23:56:24.035 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0525976 s, 199 MB/s 2024-04-26T23:56:24.036 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: purged_snaps scrub starts 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: purged_snaps scrub ok 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: osd.8 [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511] boot 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:24.099 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:24.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:24.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:24.100 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:23 smithi103 ceph-mon[32952]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T23:56:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: purged_snaps scrub starts 2024-04-26T23:56:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: purged_snaps scrub ok 2024-04-26T23:56:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: osd.8 [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511] boot 2024-04-26T23:56:24.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T23:56:24.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:24.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:24.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:24.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:24.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:24.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:24.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:24.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:24.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:24.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:24.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:23 smithi040 ceph-mon[28095]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T23:56:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: purged_snaps scrub starts 2024-04-26T23:56:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: purged_snaps scrub ok 2024-04-26T23:56:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:56:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: osd.8 [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511] boot 2024-04-26T23:56:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T23:56:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:56:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:24.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:24.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:56:24.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:24.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:24.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:24.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:24.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:23 smithi080 ceph-mon[32362]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T23:56:25.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi040:vg_nvme/lv_1 2024-04-26T23:56:25.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:24 smithi040 ceph-mon[28095]: Detected new or changed devices on smithi103 2024-04-26T23:56:25.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:24 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi103 to 5696M 2024-04-26T23:56:25.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:24 smithi040 ceph-mon[28095]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T23:56:25.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:24 smithi080 ceph-mon[32362]: Detected new or changed devices on smithi103 2024-04-26T23:56:25.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:24 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi103 to 5696M 2024-04-26T23:56:25.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:24 smithi080 ceph-mon[32362]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T23:56:25.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:24 smithi103 ceph-mon[32952]: Detected new or changed devices on smithi103 2024-04-26T23:56:25.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:24 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi103 to 5696M 2024-04-26T23:56:25.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:24 smithi103 ceph-mon[32952]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T23:56:25.599 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:56:26.170 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:25 smithi040 ceph-mon[28095]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 352 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:26.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:25 smithi080 ceph-mon[32362]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 352 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:26.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:25 smithi103 ceph-mon[32952]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 352 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:27.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:26 smithi040 ceph-mon[28095]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T23:56:27.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:26 smithi080 ceph-mon[32362]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T23:56:27.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:26 smithi103 ceph-mon[32952]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T23:56:27.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.731+0000 7feb64c00700 1 -- 172.21.15.40:0/739608052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb60106760 msgr2=0x7feb60108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:56:27.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.731+0000 7feb64c00700 1 --2- 172.21.15.40:0/739608052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb60106760 0x7feb60108b50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feb4c009990 tx=0x7feb4c009ca0 comp rx=0 tx=0).stop 2024-04-26T23:56:27.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.731+0000 7feb64c00700 1 -- 172.21.15.40:0/739608052 shutdown_connections 2024-04-26T23:56:27.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.731+0000 7feb64c00700 1 --2- 172.21.15.40:0/739608052 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb60109090 0x7feb6010b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:27.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.731+0000 7feb64c00700 1 --2- 172.21.15.40:0/739608052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb60103e30 0x7feb60106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:27.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.731+0000 7feb64c00700 1 --2- 172.21.15.40:0/739608052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb60106760 0x7feb60108b50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:27.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.731+0000 7feb64c00700 1 -- 172.21.15.40:0/739608052 >> 172.21.15.40:0/739608052 conn(0x7feb600fd840 msgr2=0x7feb600ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:56:27.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.731+0000 7feb64c00700 1 -- 172.21.15.40:0/739608052 shutdown_connections 2024-04-26T23:56:27.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.731+0000 7feb64c00700 1 -- 172.21.15.40:0/739608052 wait complete. 2024-04-26T23:56:27.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.732+0000 7feb64c00700 1 Processor -- start 2024-04-26T23:56:27.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.732+0000 7feb64c00700 1 -- start start 2024-04-26T23:56:27.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.732+0000 7feb64c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb60103e30 0x7feb6019b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:56:27.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.732+0000 7feb64c00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb60106760 0x7feb6019bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:56:27.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.732+0000 7feb64c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb60109090 0x7feb601a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:56:27.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.732+0000 7feb64c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb6010eeb0 con 0x7feb60109090 2024-04-26T23:56:27.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.732+0000 7feb64c00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7feb6010ed30 con 0x7feb60106760 2024-04-26T23:56:27.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.732+0000 7feb64c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb6010f030 con 0x7feb60103e30 2024-04-26T23:56:27.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.733+0000 7feb5fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb60109090 0x7feb601a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:56:27.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.733+0000 7feb5ea00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb60106760 0x7feb6019bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:56:27.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.733+0000 7feb5ea00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb60106760 0x7feb6019bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:39636/0 (socket says 172.21.15.40:39636) 2024-04-26T23:56:27.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.733+0000 7feb5fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb60109090 0x7feb601a2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47306/0 (socket says 172.21.15.40:47306) 2024-04-26T23:56:27.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.733+0000 7feb5ea00700 1 -- 172.21.15.40:0/3975048884 learned_addr learned my addr 172.21.15.40:0/3975048884 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:56:27.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.733+0000 7feb5f400700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb60103e30 0x7feb6019b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:56:27.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.733+0000 7feb5ea00700 1 -- 172.21.15.40:0/3975048884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb60103e30 msgr2=0x7feb6019b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:56:27.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.733+0000 7feb5ea00700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb60103e30 0x7feb6019b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:27.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.733+0000 7feb5ea00700 1 -- 172.21.15.40:0/3975048884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb60109090 msgr2=0x7feb601a2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:56:27.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.733+0000 7feb5ea00700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb60109090 0x7feb601a2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:27.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.733+0000 7feb5ea00700 1 -- 172.21.15.40:0/3975048884 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb601a3390 con 0x7feb60106760 2024-04-26T23:56:27.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.733+0000 7feb5fe00700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb60109090 0x7feb601a2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:56:27.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.734+0000 7feb5ea00700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb60106760 0x7feb6019bc70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feb4c00b760 tx=0x7feb4c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:56:27.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.734+0000 7feb57e00700 1 -- 172.21.15.40:0/3975048884 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c027070 con 0x7feb60106760 2024-04-26T23:56:27.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.734+0000 7feb64c00700 1 -- 172.21.15.40:0/3975048884 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb601a3620 con 0x7feb60106760 2024-04-26T23:56:27.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.734+0000 7feb57e00700 1 -- 172.21.15.40:0/3975048884 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb4c022470 con 0x7feb60106760 2024-04-26T23:56:27.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.734+0000 7feb57e00700 1 -- 172.21.15.40:0/3975048884 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c017650 con 0x7feb60106760 2024-04-26T23:56:27.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.734+0000 7feb64c00700 1 -- 172.21.15.40:0/3975048884 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb601a3ad0 con 0x7feb60106760 2024-04-26T23:56:27.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.735+0000 7feb64c00700 1 -- 172.21.15.40:0/3975048884 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb6006a110 con 0x7feb60106760 2024-04-26T23:56:27.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.737+0000 7feb57e00700 1 -- 172.21.15.40:0/3975048884 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7feb4c009d10 con 0x7feb60106760 2024-04-26T23:56:27.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.739+0000 7feb57e00700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feb440a3cc0 0x7feb440a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:56:27.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.739+0000 7feb57e00700 1 -- 172.21.15.40:0/3975048884 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5829+0+0 (secure 0 0 0) 0x7feb4c0d0a80 con 0x7feb60106760 2024-04-26T23:56:27.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.739+0000 7feb5f400700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feb440a3cc0 0x7feb440a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:56:27.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.740+0000 7feb5f400700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feb440a3cc0 0x7feb440a6180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feb50004630 tx=0x7feb5000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:56:27.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.741+0000 7feb57e00700 1 -- 172.21.15.40:0/3975048884 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7feb4c030070 con 0x7feb60106760 2024-04-26T23:56:27.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:27.986+0000 7feb64c00700 1 -- 172.21.15.40:0/3975048884 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7feb60063860 con 0x7feb440a3cc0 2024-04-26T23:56:28.249 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:27 smithi103 ceph-mon[32952]: pgmap v224: 1 pgs: 1 peering; 577 KiB data, 352 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:28.250 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:27 smithi103 ceph-mon[32952]: osdmap e64: 9 total, 9 up, 9 in 2024-04-26T23:56:28.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:27 smithi080 ceph-mon[32362]: pgmap v224: 1 pgs: 1 peering; 577 KiB data, 352 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:28.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:27 smithi080 ceph-mon[32362]: osdmap e64: 9 total, 9 up, 9 in 2024-04-26T23:56:28.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:27 smithi040 ceph-mon[28095]: pgmap v224: 1 pgs: 1 peering; 577 KiB data, 352 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:28.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:27 smithi040 ceph-mon[28095]: osdmap e64: 9 total, 9 up, 9 in 2024-04-26T23:56:29.252 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:28 smithi040 ceph-mon[28095]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:56:29.252 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:56:29.252 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:56:29.252 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:29.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:28 smithi080 ceph-mon[32362]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:56:29.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:56:29.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:56:29.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:29.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:28 smithi103 ceph-mon[32952]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:56:29.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:56:29.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:56:29.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:30.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:29 smithi040 ceph-mon[28095]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-04-26T23:56:30.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:29 smithi080 ceph-mon[32362]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-04-26T23:56:30.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:29 smithi103 ceph-mon[32952]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-04-26T23:56:32.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:31 smithi080 ceph-mon[32362]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T23:56:32.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:31 smithi103 ceph-mon[32952]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T23:56:32.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:31 smithi040 ceph-mon[28095]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T23:56:33.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:32 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3605949630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ed158cb-03c5-4930-b529-7d017cf5c31e"}]: dispatch 2024-04-26T23:56:33.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:32 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ed158cb-03c5-4930-b529-7d017cf5c31e"}]: dispatch 2024-04-26T23:56:33.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:32 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ed158cb-03c5-4930-b529-7d017cf5c31e"}]': finished 2024-04-26T23:56:33.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:32 smithi040 ceph-mon[28095]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T23:56:33.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:33.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:32 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3605949630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ed158cb-03c5-4930-b529-7d017cf5c31e"}]: dispatch 2024-04-26T23:56:33.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:32 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ed158cb-03c5-4930-b529-7d017cf5c31e"}]: dispatch 2024-04-26T23:56:33.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:32 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ed158cb-03c5-4930-b529-7d017cf5c31e"}]': finished 2024-04-26T23:56:33.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:32 smithi080 ceph-mon[32362]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T23:56:33.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:33.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:32 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3605949630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ed158cb-03c5-4930-b529-7d017cf5c31e"}]: dispatch 2024-04-26T23:56:33.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:32 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ed158cb-03c5-4930-b529-7d017cf5c31e"}]: dispatch 2024-04-26T23:56:33.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:32 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ed158cb-03c5-4930-b529-7d017cf5c31e"}]': finished 2024-04-26T23:56:33.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:32 smithi103 ceph-mon[32952]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T23:56:33.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:34.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:33 smithi080 ceph-mon[32362]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-26T23:56:34.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:33 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/173976123' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:56:34.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:33 smithi103 ceph-mon[32952]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-26T23:56:34.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:33 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/173976123' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:56:34.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:33 smithi040 ceph-mon[28095]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-26T23:56:34.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:33 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/173976123' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:56:36.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:35 smithi080 ceph-mon[32362]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T23:56:36.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:35 smithi103 ceph-mon[32952]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T23:56:36.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:35 smithi040 ceph-mon[28095]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T23:56:38.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:37 smithi080 ceph-mon[32362]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-26T23:56:38.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:37 smithi103 ceph-mon[32952]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-26T23:56:38.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:37 smithi040 ceph-mon[28095]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-26T23:56:40.129 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:40 smithi040 ceph-mon[28095]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:40.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:40 smithi080 ceph-mon[32362]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:40.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:40 smithi103 ceph-mon[32952]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:42.129 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:42 smithi040 ceph-mon[28095]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:42.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:42 smithi080 ceph-mon[32362]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:42.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:42 smithi103 ceph-mon[32952]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:43.289 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:43 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T23:56:43.289 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:43 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:43.289 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:43 smithi040 ceph-mon[28095]: Deploying daemon osd.9 on smithi040 2024-04-26T23:56:43.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:43 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T23:56:43.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:43 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:43.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:43 smithi080 ceph-mon[32362]: Deploying daemon osd.9 on smithi040 2024-04-26T23:56:43.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:43 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T23:56:43.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:43 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:43.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:43 smithi103 ceph-mon[32952]: Deploying daemon osd.9 on smithi040 2024-04-26T23:56:44.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:44 smithi040 ceph-mon[28095]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:44.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:44 smithi080 ceph-mon[32362]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:44.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:44 smithi103 ceph-mon[32952]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:46.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:46 smithi040 ceph-mon[28095]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:46.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:46 smithi080 ceph-mon[32362]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:46.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:46 smithi103 ceph-mon[32952]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:48.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:48 smithi080 ceph-mon[32362]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:48.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:48 smithi103 ceph-mon[32952]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:48.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:48 smithi040 ceph-mon[28095]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:50.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:50 smithi080 ceph-mon[32362]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:50.338 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:50 smithi040 ceph-mon[28095]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:50.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:50 smithi103 ceph-mon[32952]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:52.656 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:52 smithi040 ceph-mon[28095]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:52.656 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:52.656 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:52.656 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:52 smithi080 ceph-mon[32362]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:52.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:52 smithi103 ceph-mon[32952]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:52.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:52.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:52.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:54.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:54 smithi040 ceph-mon[28095]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:54.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:54 smithi080 ceph-mon[32362]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:54.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:54 smithi103 ceph-mon[32952]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:56.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:56 smithi103 ceph-mon[32952]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:56.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:56.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:56.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:56 smithi103 ceph-mon[32952]: from='osd.9 [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T23:56:56.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:56 smithi103 ceph-mon[32952]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T23:56:56.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:56 smithi040 ceph-mon[28095]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:56.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:56.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:56.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:56 smithi040 ceph-mon[28095]: from='osd.9 [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T23:56:56.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:56 smithi040 ceph-mon[28095]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T23:56:56.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.703+0000 7feb57e00700 1 -- 172.21.15.40:0/3975048884 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7feb60063860 con 0x7feb440a3cc0 2024-04-26T23:56:56.706 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 9 on host 'smithi040' 2024-04-26T23:56:56.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.707+0000 7feb64c00700 1 -- 172.21.15.40:0/3975048884 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feb440a3cc0 msgr2=0x7feb440a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:56:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.707+0000 7feb64c00700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feb440a3cc0 0x7feb440a6180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feb50004630 tx=0x7feb5000b3c0 comp rx=0 tx=0).stop 2024-04-26T23:56:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.707+0000 7feb64c00700 1 -- 172.21.15.40:0/3975048884 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb60106760 msgr2=0x7feb6019bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:56:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.707+0000 7feb64c00700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb60106760 0x7feb6019bc70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feb4c00b760 tx=0x7feb4c0047b0 comp rx=0 tx=0).stop 2024-04-26T23:56:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.707+0000 7feb64c00700 1 -- 172.21.15.40:0/3975048884 shutdown_connections 2024-04-26T23:56:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.707+0000 7feb64c00700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feb440a3cc0 0x7feb440a6180 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.707+0000 7feb64c00700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb60109090 0x7feb601a2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.707+0000 7feb64c00700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb60103e30 0x7feb6019b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.707+0000 7feb64c00700 1 --2- 172.21.15.40:0/3975048884 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb60106760 0x7feb6019bc70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:56:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.707+0000 7feb64c00700 1 -- 172.21.15.40:0/3975048884 >> 172.21.15.40:0/3975048884 conn(0x7feb600fd840 msgr2=0x7feb601035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:56:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.708+0000 7feb64c00700 1 -- 172.21.15.40:0/3975048884 shutdown_connections 2024-04-26T23:56:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:56:56.708+0000 7feb64c00700 1 -- 172.21.15.40:0/3975048884 wait complete. 2024-04-26T23:56:56.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:56 smithi080 ceph-mon[32362]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:56.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:56.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:56.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:56 smithi080 ceph-mon[32362]: from='osd.9 [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T23:56:56.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:56 smithi080 ceph-mon[32362]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T23:56:57.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:57 smithi040 ceph-mon[28095]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T23:56:57.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:57 smithi040 ceph-mon[28095]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T23:56:57.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:57.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:57 smithi040 ceph-mon[28095]: from='osd.9 [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:56:57.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:57 smithi040 ceph-mon[28095]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:56:57.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:57.735 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:57 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:57.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:57 smithi080 ceph-mon[32362]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T23:56:57.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:57 smithi080 ceph-mon[32362]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T23:56:57.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:57.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:57 smithi080 ceph-mon[32362]: from='osd.9 [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:56:57.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:57 smithi080 ceph-mon[32362]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:56:57.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:57.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:57 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:57.807 DEBUG:teuthology.orchestra.run.smithi040:osd.9> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.9.service 2024-04-26T23:56:57.812 INFO:tasks.cephadm:Deploying osd.10 on smithi080 with /dev/vg_nvme/lv_1... 2024-04-26T23:56:57.812 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T23:56:57.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:57 smithi103 ceph-mon[32952]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T23:56:57.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:57 smithi103 ceph-mon[32952]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T23:56:57.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:57.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:57 smithi103 ceph-mon[32952]: from='osd.9 [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:56:57.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:57 smithi103 ceph-mon[32952]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-04-26T23:56:57.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:57.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:57 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:56:58.058 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:56:58.468 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 26 23:56:58 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-26T23:56:58.325+0000 7f6ac3800700 -1 osd.9 0 waiting for initial osdmap 2024-04-26T23:56:58.468 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 26 23:56:58 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-26T23:56:58.365+0000 7f6ab9600700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:56:58.750 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:58 smithi103 ceph-mon[32952]: purged_snaps scrub starts 2024-04-26T23:56:58.750 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:58 smithi103 ceph-mon[32952]: purged_snaps scrub ok 2024-04-26T23:56:58.750 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:58 smithi103 ceph-mon[32952]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:58.750 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:58 smithi103 ceph-mon[32952]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-26T23:56:58.750 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:58 smithi103 ceph-mon[32952]: osdmap e67: 10 total, 9 up, 10 in 2024-04-26T23:56:58.750 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:58 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:58.750 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:58 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:58.750 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:58 smithi103 ceph-mon[32952]: from='osd.9 ' entity='osd.9' 2024-04-26T23:56:58.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:58 smithi080 ceph-mon[32362]: purged_snaps scrub starts 2024-04-26T23:56:58.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:58 smithi080 ceph-mon[32362]: purged_snaps scrub ok 2024-04-26T23:56:58.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:58 smithi080 ceph-mon[32362]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:58.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:58 smithi080 ceph-mon[32362]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-26T23:56:58.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:58 smithi080 ceph-mon[32362]: osdmap e67: 10 total, 9 up, 10 in 2024-04-26T23:56:58.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:58 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:58.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:58 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:58.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:58 smithi080 ceph-mon[32362]: from='osd.9 ' entity='osd.9' 2024-04-26T23:56:58.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:58 smithi040 ceph-mon[28095]: purged_snaps scrub starts 2024-04-26T23:56:58.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:58 smithi040 ceph-mon[28095]: purged_snaps scrub ok 2024-04-26T23:56:58.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:58 smithi040 ceph-mon[28095]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:56:58.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:58 smithi040 ceph-mon[28095]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-04-26T23:56:58.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:58 smithi040 ceph-mon[28095]: osdmap e67: 10 total, 9 up, 10 in 2024-04-26T23:56:58.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:58 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:58.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:58 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:58.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:58 smithi040 ceph-mon[28095]: from='osd.9 ' entity='osd.9' 2024-04-26T23:56:59.761 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:59 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:59.762 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:59 smithi103 ceph-mon[32952]: osd.9 [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825] boot 2024-04-26T23:56:59.762 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:59 smithi103 ceph-mon[32952]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T23:56:59.762 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:56:59 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:59.779 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:59 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:59.779 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:59 smithi080 ceph-mon[32362]: osd.9 [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825] boot 2024-04-26T23:56:59.779 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:59 smithi080 ceph-mon[32362]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T23:56:59.779 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:56:59 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:59.938 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:59 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:56:59.938 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:59 smithi040 ceph-mon[28095]: osd.9 [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825] boot 2024-04-26T23:56:59.938 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:59 smithi040 ceph-mon[28095]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T23:56:59.938 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:56:59 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:57:00.163 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T23:57:00.163 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T23:57:00.163 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-04-26T23:57:00.163 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-04-26T23:57:00.163 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116371 s, 901 MB/s 2024-04-26T23:57:00.163 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-04-26T23:57:00.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:00 smithi080 ceph-mon[32362]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:57:00.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:00 smithi080 ceph-mon[32362]: osdmap e69: 10 total, 10 up, 10 in 2024-04-26T23:57:00.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:00 smithi103 ceph-mon[32952]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:57:00.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:00 smithi103 ceph-mon[32952]: osdmap e69: 10 total, 10 up, 10 in 2024-04-26T23:57:00.927 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi080:vg_nvme/lv_1 2024-04-26T23:57:00.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:00 smithi040 ceph-mon[28095]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:57:00.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:00 smithi040 ceph-mon[28095]: osdmap e69: 10 total, 10 up, 10 in 2024-04-26T23:57:01.166 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:57:02.705 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:02 smithi080 ceph-mon[32362]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:02.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:02 smithi040 ceph-mon[28095]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:02.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:02 smithi103 ceph-mon[32952]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:02.941 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.939+0000 7f7d0557c700 1 -- 172.21.15.80:0/2760211203 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d00103f00 msgr2=0x7f7d001062f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:02.941 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.939+0000 7f7d0557c700 1 --2- 172.21.15.80:0/2760211203 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d00103f00 0x7f7d001062f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000b0a0 tx=0x7f7cf000b3b0 comp rx=0 tx=0).stop 2024-04-26T23:57:02.941 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.939+0000 7f7d0557c700 1 -- 172.21.15.80:0/2760211203 shutdown_connections 2024-04-26T23:57:02.941 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.939+0000 7f7d0557c700 1 --2- 172.21.15.80:0/2760211203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d00109160 0x7f7d0010b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:02.941 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.939+0000 7f7d0557c700 1 --2- 172.21.15.80:0/2760211203 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d00103f00 0x7f7d001062f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:02.941 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.939+0000 7f7d0557c700 1 --2- 172.21.15.80:0/2760211203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d00106830 0x7f7d00108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:02.942 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.939+0000 7f7d0557c700 1 -- 172.21.15.80:0/2760211203 >> 172.21.15.80:0/2760211203 conn(0x7f7d000fd910 msgr2=0x7f7d000ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:57:02.942 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.939+0000 7f7d0557c700 1 -- 172.21.15.80:0/2760211203 shutdown_connections 2024-04-26T23:57:02.942 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.939+0000 7f7d0557c700 1 -- 172.21.15.80:0/2760211203 wait complete. 2024-04-26T23:57:02.942 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.940+0000 7f7d0557c700 1 Processor -- start 2024-04-26T23:57:02.943 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.940+0000 7f7d0557c700 1 -- start start 2024-04-26T23:57:02.943 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.940+0000 7f7d0557c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d00103f00 0x7f7d00199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:02.944 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.940+0000 7f7d0557c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d00106830 0x7f7d00199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:02.944 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.940+0000 7f7d0557c700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d00109160 0x7f7d001a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:02.944 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.940+0000 7f7d0557c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0010eff0 con 0x7f7d00103f00 2024-04-26T23:57:02.944 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.940+0000 7f7d0557c700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0010ee70 con 0x7f7d00109160 2024-04-26T23:57:02.944 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.940+0000 7f7d0557c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0010f170 con 0x7f7d00106830 2024-04-26T23:57:02.944 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.940+0000 7f7cff7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d00109160 0x7f7d001a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:02.944 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.940+0000 7f7cff7fe700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d00109160 0x7f7d001a0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.80:42058/0 (socket says 172.21.15.80:42058) 2024-04-26T23:57:02.944 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.940+0000 7f7cff7fe700 1 -- 172.21.15.80:0/2270956921 learned_addr learned my addr 172.21.15.80:0/2270956921 (peer_addr_for_me v2:172.21.15.80:0/0) 2024-04-26T23:57:02.944 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7cfe7fc700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d00106830 0x7f7d00199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:02.944 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7cff7fe700 1 -- 172.21.15.80:0/2270956921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d00106830 msgr2=0x7f7d00199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:02.944 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7cff7fe700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d00106830 0x7f7d00199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:02.944 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7cff7fe700 1 -- 172.21.15.80:0/2270956921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d00103f00 msgr2=0x7f7d00199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:02.945 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7cfeffd700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d00103f00 0x7f7d00199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:02.945 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7cff7fe700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d00103f00 0x7f7d00199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:02.945 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7cff7fe700 1 -- 172.21.15.80:0/2270956921 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d001a1370 con 0x7f7d00109160 2024-04-26T23:57:02.945 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7cff7fe700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d00109160 0x7f7d001a0cd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800b480 tx=0x7f7cf800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:57:02.945 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7cf7fff700 1 -- 172.21.15.80:0/2270956921 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8015020 con 0x7f7d00109160 2024-04-26T23:57:02.945 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7d0557c700 1 -- 172.21.15.80:0/2270956921 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d001a1660 con 0x7f7d00109160 2024-04-26T23:57:02.945 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7cf7fff700 1 -- 172.21.15.80:0/2270956921 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cf8011ce0 con 0x7f7d00109160 2024-04-26T23:57:02.945 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7cf7fff700 1 -- 172.21.15.80:0/2270956921 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf800f2e0 con 0x7f7d00109160 2024-04-26T23:57:02.945 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.941+0000 7f7d0557c700 1 -- 172.21.15.80:0/2270956921 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d001a1c20 con 0x7f7d00109160 2024-04-26T23:57:02.946 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.942+0000 7f7d0557c700 1 -- 172.21.15.80:0/2270956921 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d0006a110 con 0x7f7d00109160 2024-04-26T23:57:02.946 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.943+0000 7f7cf7fff700 1 -- 172.21.15.80:0/2270956921 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7cf8018020 con 0x7f7d00109160 2024-04-26T23:57:02.946 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.944+0000 7f7cf7fff700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7cec0a3c40 0x7f7cec0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:02.946 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.944+0000 7f7cf7fff700 1 -- 172.21.15.80:0/2270956921 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f7cf80cf440 con 0x7f7d00109160 2024-04-26T23:57:02.946 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.944+0000 7f7cfeffd700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7cec0a3c40 0x7f7cec0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:02.947 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.945+0000 7f7cfeffd700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7cec0a3c40 0x7f7cec0a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000b830 tx=0x7f7cf00060c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:57:02.950 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:02.949+0000 7f7cf7fff700 1 -- 172.21.15.80:0/2270956921 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7cf801f020 con 0x7f7d00109160 2024-04-26T23:57:03.159 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:03.157+0000 7f7d0557c700 1 -- 172.21.15.80:0/2270956921 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7d00063860 con 0x7f7cec0a3c40 2024-04-26T23:57:04.085 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:04.085 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: Detected new or changed devices on smithi040 2024-04-26T23:57:04.085 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi040 to 4273M 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:57:04.086 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:04.087 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:04.087 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:04.087 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:04.087 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:04.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: Detected new or changed devices on smithi040 2024-04-26T23:57:04.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:04.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi040 to 4273M 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:04.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:04.220 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:04.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: Detected new or changed devices on smithi040 2024-04-26T23:57:04.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi040 to 4273M 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:57:04.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:04.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:04.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:04.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:04.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:06.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:05 smithi040 ceph-mon[28095]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:06.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:05 smithi080 ceph-mon[32362]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:06.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:05 smithi103 ceph-mon[32952]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:07.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:06 smithi040 ceph-mon[28095]: from='client.? 172.21.15.80:0/1717755760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1ee814-e69d-492a-9b3c-64d53ba2e22c"}]: dispatch 2024-04-26T23:57:07.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:06 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1ee814-e69d-492a-9b3c-64d53ba2e22c"}]: dispatch 2024-04-26T23:57:07.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:06 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba1ee814-e69d-492a-9b3c-64d53ba2e22c"}]': finished 2024-04-26T23:57:07.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:06 smithi040 ceph-mon[28095]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T23:57:07.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:06 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:57:07.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:06 smithi040 ceph-mon[28095]: from='client.? 172.21.15.80:0/2919904299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:57:07.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:06 smithi080 ceph-mon[32362]: from='client.? 172.21.15.80:0/1717755760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1ee814-e69d-492a-9b3c-64d53ba2e22c"}]: dispatch 2024-04-26T23:57:07.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:06 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1ee814-e69d-492a-9b3c-64d53ba2e22c"}]: dispatch 2024-04-26T23:57:07.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:06 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba1ee814-e69d-492a-9b3c-64d53ba2e22c"}]': finished 2024-04-26T23:57:07.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:06 smithi080 ceph-mon[32362]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T23:57:07.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:06 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:57:07.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:06 smithi080 ceph-mon[32362]: from='client.? 172.21.15.80:0/2919904299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:57:07.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:06 smithi103 ceph-mon[32952]: from='client.? 172.21.15.80:0/1717755760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1ee814-e69d-492a-9b3c-64d53ba2e22c"}]: dispatch 2024-04-26T23:57:07.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:06 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba1ee814-e69d-492a-9b3c-64d53ba2e22c"}]: dispatch 2024-04-26T23:57:07.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:06 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba1ee814-e69d-492a-9b3c-64d53ba2e22c"}]': finished 2024-04-26T23:57:07.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:06 smithi103 ceph-mon[32952]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T23:57:07.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:06 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:57:07.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:06 smithi103 ceph-mon[32952]: from='client.? 172.21.15.80:0/2919904299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:57:08.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:08 smithi103 ceph-mon[32952]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:08.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:08 smithi040 ceph-mon[28095]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:08.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:08 smithi080 ceph-mon[32362]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:10.333 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:10 smithi080 ceph-mon[32362]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:10.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:10 smithi103 ceph-mon[32952]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:10.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:10 smithi040 ceph-mon[28095]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:12.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:12 smithi103 ceph-mon[32952]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:12.456 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:12 smithi080 ceph-mon[32362]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:12.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:12 smithi040 ceph-mon[28095]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:14.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:14 smithi080 ceph-mon[32362]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:14.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T23:57:14.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:14.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:14 smithi103 ceph-mon[32952]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:14.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T23:57:14.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:14.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:14 smithi040 ceph-mon[28095]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:14.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T23:57:14.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:15.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:15 smithi080 ceph-mon[32362]: Deploying daemon osd.10 on smithi080 2024-04-26T23:57:15.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:15 smithi040 ceph-mon[28095]: Deploying daemon osd.10 on smithi080 2024-04-26T23:57:15.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:15 smithi103 ceph-mon[32952]: Deploying daemon osd.10 on smithi080 2024-04-26T23:57:16.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:16 smithi080 ceph-mon[32362]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:16.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:16 smithi040 ceph-mon[28095]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:16.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:16 smithi103 ceph-mon[32952]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:18.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:18 smithi080 ceph-mon[32362]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:18.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:18 smithi040 ceph-mon[28095]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:18.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:18 smithi103 ceph-mon[32952]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:20.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:20 smithi040 ceph-mon[28095]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:20.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:20 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:20.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:20 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:20.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:20 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:20.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:20 smithi080 ceph-mon[32362]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:20.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:20 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:20.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:20 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:20.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:20 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:20.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:20 smithi103 ceph-mon[32952]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:20.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:20 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:20.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:20 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:20.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:20 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:22.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:22 smithi080 ceph-mon[32362]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:22.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:22 smithi040 ceph-mon[28095]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:22.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:22 smithi103 ceph-mon[32952]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:23.856 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.854+0000 7f7cf7fff700 1 -- 172.21.15.80:0/2270956921 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f7d00063860 con 0x7f7cec0a3c40 2024-04-26T23:57:23.856 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 10 on host 'smithi080' 2024-04-26T23:57:23.858 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.856+0000 7f7d0557c700 1 -- 172.21.15.80:0/2270956921 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7cec0a3c40 msgr2=0x7f7cec0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:23.859 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.857+0000 7f7d0557c700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7cec0a3c40 0x7f7cec0a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000b830 tx=0x7f7cf00060c0 comp rx=0 tx=0).stop 2024-04-26T23:57:23.859 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.857+0000 7f7d0557c700 1 -- 172.21.15.80:0/2270956921 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d00109160 msgr2=0x7f7d001a0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:23.859 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.857+0000 7f7d0557c700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d00109160 0x7f7d001a0cd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800b480 tx=0x7f7cf800b840 comp rx=0 tx=0).stop 2024-04-26T23:57:23.859 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.857+0000 7f7d0557c700 1 -- 172.21.15.80:0/2270956921 shutdown_connections 2024-04-26T23:57:23.859 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.857+0000 7f7d0557c700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7cec0a3c40 0x7f7cec0a6100 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:23.859 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.857+0000 7f7d0557c700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7d00109160 0x7f7d001a0cd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:23.859 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.857+0000 7f7d0557c700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d00103f00 0x7f7d00199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:23.859 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.857+0000 7f7d0557c700 1 --2- 172.21.15.80:0/2270956921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d00106830 0x7f7d00199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:23.859 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.857+0000 7f7d0557c700 1 -- 172.21.15.80:0/2270956921 >> 172.21.15.80:0/2270956921 conn(0x7f7d000fd910 msgr2=0x7f7d00104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:57:23.859 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.857+0000 7f7d0557c700 1 -- 172.21.15.80:0/2270956921 shutdown_connections 2024-04-26T23:57:23.859 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-26T23:57:23.857+0000 7f7d0557c700 1 -- 172.21.15.80:0/2270956921 wait complete. 2024-04-26T23:57:24.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:24 smithi080 ceph-mon[32362]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:24.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:24 smithi080 ceph-mon[32362]: from='osd.10 [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T23:57:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:24 smithi080 ceph-mon[32362]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T23:57:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:24.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:24 smithi103 ceph-mon[32952]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:24.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:24.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:24.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:24 smithi103 ceph-mon[32952]: from='osd.10 [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T23:57:24.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:24 smithi103 ceph-mon[32952]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T23:57:24.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:24.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:24 smithi040 ceph-mon[28095]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:24.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:24.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:24.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:24 smithi040 ceph-mon[28095]: from='osd.10 [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T23:57:24.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:24 smithi040 ceph-mon[28095]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T23:57:24.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:24.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:24.553 DEBUG:teuthology.orchestra.run.smithi080:osd.10> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.10.service 2024-04-26T23:57:24.555 INFO:tasks.cephadm:Deploying osd.11 on smithi103 with /dev/vg_nvme/lv_1... 2024-04-26T23:57:24.556 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T23:57:24.803 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:57:25.177 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:25 smithi103 ceph-mon[32952]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T23:57:25.177 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:25 smithi103 ceph-mon[32952]: from='osd.10 [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:57:25.177 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:25 smithi103 ceph-mon[32952]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T23:57:25.177 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:25 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:57:25.177 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:25 smithi103 ceph-mon[32952]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:57:25.457 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:25 smithi080 ceph-mon[32362]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T23:57:25.457 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:25 smithi080 ceph-mon[32362]: from='osd.10 [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:57:25.457 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:25 smithi080 ceph-mon[32362]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T23:57:25.457 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:25 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:57:25.457 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:25 smithi080 ceph-mon[32362]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:57:25.458 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 26 23:57:25 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-26T23:57:25.140+0000 7f0e07da0700 -1 osd.10 0 waiting for initial osdmap 2024-04-26T23:57:25.458 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 26 23:57:25 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-26T23:57:25.152+0000 7f0e03b93700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:57:25.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:25 smithi040 ceph-mon[28095]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T23:57:25.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:25 smithi040 ceph-mon[28095]: from='osd.10 [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:57:25.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:25 smithi040 ceph-mon[28095]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T23:57:25.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:25 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:57:25.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:25 smithi040 ceph-mon[28095]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-04-26T23:57:26.207 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:26 smithi103 ceph-mon[32952]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:26.207 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:26 smithi103 ceph-mon[32952]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-04-26T23:57:26.208 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:26 smithi103 ceph-mon[32952]: osdmap e72: 11 total, 10 up, 11 in 2024-04-26T23:57:26.208 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:26 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:57:26.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:26 smithi040 ceph-mon[28095]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:26.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:26 smithi040 ceph-mon[28095]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-04-26T23:57:26.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:26 smithi040 ceph-mon[28095]: osdmap e72: 11 total, 10 up, 11 in 2024-04-26T23:57:26.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:26 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:57:26.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:26 smithi080 ceph-mon[32362]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:57:26.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:26 smithi080 ceph-mon[32362]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-04-26T23:57:26.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:26 smithi080 ceph-mon[32362]: osdmap e72: 11 total, 10 up, 11 in 2024-04-26T23:57:26.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:26 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:57:26.905 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T23:57:26.905 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T23:57:26.905 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-04-26T23:57:26.905 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-04-26T23:57:26.905 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131541 s, 797 MB/s 2024-04-26T23:57:26.905 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-04-26T23:57:27.231 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:27 smithi103 ceph-mon[32952]: purged_snaps scrub starts 2024-04-26T23:57:27.231 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:27 smithi103 ceph-mon[32952]: purged_snaps scrub ok 2024-04-26T23:57:27.231 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:27 smithi103 ceph-mon[32952]: osd.10 [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951] boot 2024-04-26T23:57:27.231 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:27 smithi103 ceph-mon[32952]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T23:57:27.231 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:27 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:57:27.231 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:27 smithi103 ceph-mon[32952]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T23:57:27.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:27 smithi040 ceph-mon[28095]: purged_snaps scrub starts 2024-04-26T23:57:27.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:27 smithi040 ceph-mon[28095]: purged_snaps scrub ok 2024-04-26T23:57:27.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:27 smithi040 ceph-mon[28095]: osd.10 [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951] boot 2024-04-26T23:57:27.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:27 smithi040 ceph-mon[28095]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T23:57:27.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:27 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:57:27.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:27 smithi040 ceph-mon[28095]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T23:57:27.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:27 smithi080 ceph-mon[32362]: purged_snaps scrub starts 2024-04-26T23:57:27.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:27 smithi080 ceph-mon[32362]: purged_snaps scrub ok 2024-04-26T23:57:27.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:27 smithi080 ceph-mon[32362]: osd.10 [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951] boot 2024-04-26T23:57:27.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:27 smithi080 ceph-mon[32362]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T23:57:27.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:27 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:57:27.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:27 smithi080 ceph-mon[32362]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T23:57:27.567 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_1 2024-04-26T23:57:27.817 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:57:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:28 smithi040 ceph-mon[28095]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:57:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:28 smithi040 ceph-mon[28095]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T23:57:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:28.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:28.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:28 smithi080 ceph-mon[32362]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:57:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:28 smithi080 ceph-mon[32362]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T23:57:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:28.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:28.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:28 smithi103 ceph-mon[32952]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:57:28.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:28 smithi103 ceph-mon[32952]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T23:57:28.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:28.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:28.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:28.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:28.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:28.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:28.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:28.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:28.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:28.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:29.291 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:29 smithi103 ceph-mon[32952]: Detected new or changed devices on smithi080 2024-04-26T23:57:29.292 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:29 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi080 to 4272M 2024-04-26T23:57:29.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:29 smithi040 ceph-mon[28095]: Detected new or changed devices on smithi080 2024-04-26T23:57:29.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:29 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi080 to 4272M 2024-04-26T23:57:29.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.520+0000 7fcfcf240700 1 -- 172.21.15.103:0/3467122128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfc8104680 msgr2=0x7fcfc8104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.520+0000 7fcfcf240700 1 --2- 172.21.15.103:0/3467122128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfc8104680 0x7fcfc8104ae0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4009990 tx=0x7fcfb4009ca0 comp rx=0 tx=0).stop 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.520+0000 7fcfcf240700 1 -- 172.21.15.103:0/3467122128 shutdown_connections 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.520+0000 7fcfcf240700 1 --2- 172.21.15.103:0/3467122128 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcfc8105020 0x7fcfc810b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.520+0000 7fcfcf240700 1 --2- 172.21.15.103:0/3467122128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfc8103480 0x7fcfc81038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.520+0000 7fcfcf240700 1 --2- 172.21.15.103:0/3467122128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfc8104680 0x7fcfc8104ae0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.520+0000 7fcfcf240700 1 -- 172.21.15.103:0/3467122128 >> 172.21.15.103:0/3467122128 conn(0x7fcfc80fea20 msgr2=0x7fcfc8100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.520+0000 7fcfcf240700 1 -- 172.21.15.103:0/3467122128 shutdown_connections 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.520+0000 7fcfcf240700 1 -- 172.21.15.103:0/3467122128 wait complete. 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.520+0000 7fcfcf240700 1 Processor -- start 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfcf240700 1 -- start start 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfcf240700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfc8103480 0x7fcfc8199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfcf240700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfc8104680 0x7fcfc8199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfcf240700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcfc8105020 0x7fcfc81a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfcf240700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc810efc0 con 0x7fcfc8104680 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfcf240700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc810ee40 con 0x7fcfc8105020 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfcf240700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc810f140 con 0x7fcfc8103480 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfccfdc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfc8103480 0x7fcfc8199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfccfdc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfc8103480 0x7fcfc8199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35324/0 (socket says 172.21.15.103:35324) 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfccfdc700 1 -- 172.21.15.103:0/3376116292 learned_addr learned my addr 172.21.15.103:0/3376116292 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfcd7dd700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcfc8105020 0x7fcfc81a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfccfdc700 1 -- 172.21.15.103:0/3376116292 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcfc8105020 msgr2=0x7fcfc81a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfccfdc700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcfc8105020 0x7fcfc81a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfccfdc700 1 -- 172.21.15.103:0/3376116292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfc8104680 msgr2=0x7fcfc8199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfccfdc700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfc8104680 0x7fcfc8199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfccfdc700 1 -- 172.21.15.103:0/3376116292 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc81a12c0 con 0x7fcfc8103480 2024-04-26T23:57:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.521+0000 7fcfccfdc700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfc8103480 0x7fcfc8199660 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00b530 tx=0x7fcfbc00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:57:29.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.522+0000 7fcfc5ffb700 1 -- 172.21.15.103:0/3376116292 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc015020 con 0x7fcfc8103480 2024-04-26T23:57:29.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.522+0000 7fcfc5ffb700 1 -- 172.21.15.103:0/3376116292 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfbc011ce0 con 0x7fcfc8103480 2024-04-26T23:57:29.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.522+0000 7fcfcf240700 1 -- 172.21.15.103:0/3376116292 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfc81a15b0 con 0x7fcfc8103480 2024-04-26T23:57:29.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.522+0000 7fcfc5ffb700 1 -- 172.21.15.103:0/3376116292 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc010d60 con 0x7fcfc8103480 2024-04-26T23:57:29.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.522+0000 7fcfcf240700 1 -- 172.21.15.103:0/3376116292 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfc81a1bc0 con 0x7fcfc8103480 2024-04-26T23:57:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.523+0000 7fcfcf240700 1 -- 172.21.15.103:0/3376116292 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfc806a110 con 0x7fcfc8103480 2024-04-26T23:57:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.524+0000 7fcfc5ffb700 1 -- 172.21.15.103:0/3376116292 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcfbc018020 con 0x7fcfc8103480 2024-04-26T23:57:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.524+0000 7fcfc5ffb700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcfb80a3cc0 0x7fcfb80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.524+0000 7fcfc5ffb700 1 -- 172.21.15.103:0/3376116292 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7fcfbc0cf480 con 0x7fcfc8103480 2024-04-26T23:57:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.525+0000 7fcfc7fff700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcfb80a3cc0 0x7fcfb80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:29.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.526+0000 7fcfc7fff700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcfb80a3cc0 0x7fcfb80a6180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400b2e0 tx=0x7fcfb4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:57:29.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.529+0000 7fcfc5ffb700 1 -- 172.21.15.103:0/3376116292 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcfbc01f020 con 0x7fcfc8103480 2024-04-26T23:57:29.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:29 smithi080 ceph-mon[32362]: Detected new or changed devices on smithi080 2024-04-26T23:57:29.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:29 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi080 to 4272M 2024-04-26T23:57:29.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:29.738+0000 7fcfcf240700 1 -- 172.21.15.103:0/3376116292 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fcfc8063860 con 0x7fcfb80a3cc0 2024-04-26T23:57:30.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:30 smithi040 ceph-mon[28095]: pgmap v267: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:57:30.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:30 smithi040 ceph-mon[28095]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T23:57:30.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:30 smithi040 ceph-mon[28095]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T23:57:30.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:30 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:57:30.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:30 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:57:30.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:30 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:30.526 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:30 smithi103 ceph-mon[32952]: pgmap v267: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:57:30.527 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:30 smithi103 ceph-mon[32952]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T23:57:30.527 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:30 smithi103 ceph-mon[32952]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T23:57:30.527 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:30 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:57:30.527 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:30 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:57:30.527 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:30 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:30.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:30 smithi080 ceph-mon[32362]: pgmap v267: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:57:30.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:30 smithi080 ceph-mon[32362]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T23:57:30.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:30 smithi080 ceph-mon[32362]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T23:57:30.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:30 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:57:30.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:30 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:57:30.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:30 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:31.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:31 smithi040 ceph-mon[28095]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:57:31.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:31 smithi040 ceph-mon[28095]: osdmap e77: 11 total, 11 up, 11 in 2024-04-26T23:57:31.483 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:31 smithi103 ceph-mon[32952]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:57:31.483 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:31 smithi103 ceph-mon[32952]: osdmap e77: 11 total, 11 up, 11 in 2024-04-26T23:57:31.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:31 smithi080 ceph-mon[32362]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:57:31.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:31 smithi080 ceph-mon[32362]: osdmap e77: 11 total, 11 up, 11 in 2024-04-26T23:57:32.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:32 smithi040 ceph-mon[28095]: pgmap v270: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:57:32.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:32 smithi080 ceph-mon[32362]: pgmap v270: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:57:32.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:32 smithi103 ceph-mon[32952]: pgmap v270: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:57:33.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:33 smithi040 ceph-mon[28095]: from='client.? 172.21.15.103:0/3279324133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a4a8429-ed19-4687-9a13-251dfbf74878"}]: dispatch 2024-04-26T23:57:33.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:33 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a4a8429-ed19-4687-9a13-251dfbf74878"}]: dispatch 2024-04-26T23:57:33.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:33 smithi040 ceph-mon[28095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a4a8429-ed19-4687-9a13-251dfbf74878"}]': finished 2024-04-26T23:57:33.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:33 smithi040 ceph-mon[28095]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T23:57:33.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:33 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:33.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:33 smithi040 ceph-mon[28095]: from='client.? 172.21.15.103:0/3286159617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:57:33.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:33 smithi080 ceph-mon[32362]: from='client.? 172.21.15.103:0/3279324133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a4a8429-ed19-4687-9a13-251dfbf74878"}]: dispatch 2024-04-26T23:57:33.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:33 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a4a8429-ed19-4687-9a13-251dfbf74878"}]: dispatch 2024-04-26T23:57:33.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:33 smithi080 ceph-mon[32362]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a4a8429-ed19-4687-9a13-251dfbf74878"}]': finished 2024-04-26T23:57:33.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:33 smithi080 ceph-mon[32362]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T23:57:33.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:33 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:33.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:33 smithi080 ceph-mon[32362]: from='client.? 172.21.15.103:0/3286159617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:57:33.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:33 smithi103 ceph-mon[32952]: from='client.? 172.21.15.103:0/3279324133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a4a8429-ed19-4687-9a13-251dfbf74878"}]: dispatch 2024-04-26T23:57:33.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:33 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a4a8429-ed19-4687-9a13-251dfbf74878"}]: dispatch 2024-04-26T23:57:33.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:33 smithi103 ceph-mon[32952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a4a8429-ed19-4687-9a13-251dfbf74878"}]': finished 2024-04-26T23:57:33.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:33 smithi103 ceph-mon[32952]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T23:57:33.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:33 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:33.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:33 smithi103 ceph-mon[32952]: from='client.? 172.21.15.103:0/3286159617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:57:34.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:34 smithi080 ceph-mon[32362]: pgmap v272: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:57:34.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:34 smithi103 ceph-mon[32952]: pgmap v272: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:57:34.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:34 smithi040 ceph-mon[28095]: pgmap v272: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:57:35.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:35 smithi103 ceph-mon[32952]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T23:57:35.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:35 smithi103 ceph-mon[32952]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T23:57:35.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:35 smithi103 ceph-mon[32952]: Cluster is now healthy 2024-04-26T23:57:35.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:35 smithi040 ceph-mon[28095]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T23:57:35.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:35 smithi040 ceph-mon[28095]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T23:57:35.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:35 smithi040 ceph-mon[28095]: Cluster is now healthy 2024-04-26T23:57:36.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:35 smithi080 ceph-mon[32362]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T23:57:36.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:35 smithi080 ceph-mon[32362]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T23:57:36.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:35 smithi080 ceph-mon[32362]: Cluster is now healthy 2024-04-26T23:57:38.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:37 smithi080 ceph-mon[32362]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-26T23:57:38.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:37 smithi103 ceph-mon[32952]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-26T23:57:38.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:37 smithi040 ceph-mon[28095]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-26T23:57:40.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:39 smithi080 ceph-mon[32362]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-26T23:57:40.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:39 smithi103 ceph-mon[32952]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-26T23:57:40.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:39 smithi040 ceph-mon[28095]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-26T23:57:41.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:40 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T23:57:41.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:40 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:41.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:40 smithi080 ceph-mon[32362]: Deploying daemon osd.11 on smithi103 2024-04-26T23:57:41.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:40 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T23:57:41.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:40 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:41.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:40 smithi103 ceph-mon[32952]: Deploying daemon osd.11 on smithi103 2024-04-26T23:57:41.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:40 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T23:57:41.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:40 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:41.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:40 smithi040 ceph-mon[28095]: Deploying daemon osd.11 on smithi103 2024-04-26T23:57:41.987 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:41 smithi103 ceph-mon[32952]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:57:42.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:41 smithi080 ceph-mon[32362]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:57:42.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:41 smithi040 ceph-mon[28095]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:57:44.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:43 smithi080 ceph-mon[32362]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-26T23:57:44.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:43 smithi103 ceph-mon[32952]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-26T23:57:44.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:43 smithi040 ceph-mon[28095]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-26T23:57:46.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:46 smithi040 ceph-mon[28095]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T23:57:46.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:46.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:46.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:46 smithi080 ceph-mon[32362]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T23:57:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:46 smithi103 ceph-mon[32952]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T23:57:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:48.708 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:48 smithi103 ceph-mon[32952]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:57:48.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:48 smithi040 ceph-mon[28095]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:57:48.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:48 smithi080 ceph-mon[32362]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:57:49.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.433+0000 7fcfc5ffb700 1 -- 172.21.15.103:0/3376116292 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fcfc8063860 con 0x7fcfb80a3cc0 2024-04-26T23:57:49.434 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 11 on host 'smithi103' 2024-04-26T23:57:49.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.435+0000 7fcfcf240700 1 -- 172.21.15.103:0/3376116292 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcfb80a3cc0 msgr2=0x7fcfb80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:49.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.435+0000 7fcfcf240700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcfb80a3cc0 0x7fcfb80a6180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400b2e0 tx=0x7fcfb4005fb0 comp rx=0 tx=0).stop 2024-04-26T23:57:49.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.435+0000 7fcfcf240700 1 -- 172.21.15.103:0/3376116292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfc8103480 msgr2=0x7fcfc8199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:49.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.435+0000 7fcfcf240700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfc8103480 0x7fcfc8199660 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00b530 tx=0x7fcfbc00b840 comp rx=0 tx=0).stop 2024-04-26T23:57:49.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.435+0000 7fcfcf240700 1 -- 172.21.15.103:0/3376116292 shutdown_connections 2024-04-26T23:57:49.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.435+0000 7fcfcf240700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcfb80a3cc0 0x7fcfb80a6180 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:49.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.435+0000 7fcfcf240700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcfc8105020 0x7fcfc81a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:49.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.435+0000 7fcfcf240700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfc8103480 0x7fcfc8199660 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:49.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.435+0000 7fcfcf240700 1 --2- 172.21.15.103:0/3376116292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfc8104680 0x7fcfc8199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:49.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.435+0000 7fcfcf240700 1 -- 172.21.15.103:0/3376116292 >> 172.21.15.103:0/3376116292 conn(0x7fcfc80fea20 msgr2=0x7fcfc8107a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:57:49.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.435+0000 7fcfcf240700 1 -- 172.21.15.103:0/3376116292 shutdown_connections 2024-04-26T23:57:49.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:57:49.435+0000 7fcfcf240700 1 -- 172.21.15.103:0/3376116292 wait complete. 2024-04-26T23:57:49.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:49.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:49.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:49 smithi103 ceph-mon[32952]: from='osd.11 [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T23:57:49.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:49 smithi103 ceph-mon[32952]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T23:57:49.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:49.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:49.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:49.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:49 smithi080 ceph-mon[32362]: from='osd.11 [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T23:57:49.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:49 smithi080 ceph-mon[32362]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T23:57:49.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:49.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:49.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:49.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:49 smithi040 ceph-mon[28095]: from='osd.11 [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T23:57:49.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:49 smithi040 ceph-mon[28095]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T23:57:49.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:49.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:49.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:50.099 DEBUG:teuthology.orchestra.run.smithi103:osd.11> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.11.service 2024-04-26T23:57:50.101 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-26T23:57:50.101 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T23:57:50.569 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:57:50.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:50 smithi080 ceph-mon[32362]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:57:50.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:50 smithi080 ceph-mon[32362]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T23:57:50.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:50 smithi080 ceph-mon[32362]: from='osd.11 [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:57:50.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:50 smithi080 ceph-mon[32362]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T23:57:50.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:50 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:50.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:50 smithi080 ceph-mon[32362]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:57:50.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:50 smithi103 ceph-mon[32952]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:57:50.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:50 smithi103 ceph-mon[32952]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T23:57:50.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:50 smithi103 ceph-mon[32952]: from='osd.11 [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:57:50.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:50 smithi103 ceph-mon[32952]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T23:57:50.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:50 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:50.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:50 smithi103 ceph-mon[32952]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:57:50.849 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 26 23:57:50 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-26T23:57:50.514+0000 7f6443569700 -1 osd.11 0 waiting for initial osdmap 2024-04-26T23:57:50.849 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 26 23:57:50 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-26T23:57:50.526+0000 7f643f35c700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:57:50.920 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:50 smithi040 ceph-mon[28095]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:57:50.921 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:50 smithi040 ceph-mon[28095]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T23:57:50.921 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:50 smithi040 ceph-mon[28095]: from='osd.11 [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:57:50.921 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:50 smithi040 ceph-mon[28095]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T23:57:50.921 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:50 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:50.921 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:50 smithi040 ceph-mon[28095]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-04-26T23:57:51.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:51 smithi040 ceph-mon[28095]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T23:57:51.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:51 smithi040 ceph-mon[28095]: osdmap e80: 12 total, 11 up, 12 in 2024-04-26T23:57:51.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:51 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:51.773 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:51 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:51.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:51 smithi080 ceph-mon[32362]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T23:57:51.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:51 smithi080 ceph-mon[32362]: osdmap e80: 12 total, 11 up, 12 in 2024-04-26T23:57:51.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:51 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:51.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:51 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:51.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:51 smithi103 ceph-mon[32952]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-04-26T23:57:51.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:51 smithi103 ceph-mon[32952]: osdmap e80: 12 total, 11 up, 12 in 2024-04-26T23:57:51.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:51 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:51.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:51 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:52.665 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:52 smithi040 ceph-mon[28095]: purged_snaps scrub starts 2024-04-26T23:57:52.665 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:52 smithi040 ceph-mon[28095]: purged_snaps scrub ok 2024-04-26T23:57:52.665 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:52 smithi040 ceph-mon[28095]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:57:52.665 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:52.665 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:52 smithi040 ceph-mon[28095]: osd.11 [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397] boot 2024-04-26T23:57:52.665 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:52 smithi040 ceph-mon[28095]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T23:57:52.666 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:52.734 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:52 smithi103 ceph-mon[32952]: purged_snaps scrub starts 2024-04-26T23:57:52.734 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:52 smithi103 ceph-mon[32952]: purged_snaps scrub ok 2024-04-26T23:57:52.734 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:52 smithi103 ceph-mon[32952]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:57:52.735 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:52.735 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:52 smithi103 ceph-mon[32952]: osd.11 [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397] boot 2024-04-26T23:57:52.735 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:52 smithi103 ceph-mon[32952]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T23:57:52.735 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:52 smithi080 ceph-mon[32362]: purged_snaps scrub starts 2024-04-26T23:57:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:52 smithi080 ceph-mon[32362]: purged_snaps scrub ok 2024-04-26T23:57:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:52 smithi080 ceph-mon[32362]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:57:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:52 smithi080 ceph-mon[32362]: osd.11 [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397] boot 2024-04-26T23:57:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:52 smithi080 ceph-mon[32362]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T23:57:52.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:57:52.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.994+0000 7f83b1200700 1 -- 172.21.15.40:0/1540432537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83ac103e30 msgr2=0x7f83ac106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:52.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.995+0000 7f83b1200700 1 --2- 172.21.15.40:0/1540432537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83ac103e30 0x7f83ac106220 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f839c009990 tx=0x7f839c009ca0 comp rx=0 tx=0).stop 2024-04-26T23:57:52.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.995+0000 7f83b1200700 1 -- 172.21.15.40:0/1540432537 shutdown_connections 2024-04-26T23:57:52.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.995+0000 7f83b1200700 1 --2- 172.21.15.40:0/1540432537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83ac109090 0x7f83ac10b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.995+0000 7f83b1200700 1 --2- 172.21.15.40:0/1540432537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83ac103e30 0x7f83ac106220 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.995+0000 7f83b1200700 1 --2- 172.21.15.40:0/1540432537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83ac106760 0x7f83ac108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.995+0000 7f83b1200700 1 -- 172.21.15.40:0/1540432537 >> 172.21.15.40:0/1540432537 conn(0x7f83ac0fd840 msgr2=0x7f83ac0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:57:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.995+0000 7f83b1200700 1 -- 172.21.15.40:0/1540432537 shutdown_connections 2024-04-26T23:57:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.995+0000 7f83b1200700 1 -- 172.21.15.40:0/1540432537 wait complete. 2024-04-26T23:57:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.996+0000 7f83b1200700 1 Processor -- start 2024-04-26T23:57:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.996+0000 7f83b1200700 1 -- start start 2024-04-26T23:57:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.996+0000 7f83b1200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83ac103e30 0x7f83ac199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.996+0000 7f83b1200700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83ac106760 0x7f83ac199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.996+0000 7f83b1200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83ac109090 0x7f83ac1a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.996+0000 7f83b1200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f83ac10eeb0 con 0x7f83ac109090 2024-04-26T23:57:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.996+0000 7f83b1200700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f83ac10ed30 con 0x7f83ac106760 2024-04-26T23:57:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.996+0000 7f83b1200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f83ac10f030 con 0x7f83ac103e30 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.996+0000 7f83aaa00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83ac106760 0x7f83ac199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.997+0000 7f83aaa00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83ac106760 0x7f83ac199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:38076/0 (socket says 172.21.15.40:38076) 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.997+0000 7f83ab400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83ac103e30 0x7f83ac199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.997+0000 7f83aaa00700 1 -- 172.21.15.40:0/2422230343 learned_addr learned my addr 172.21.15.40:0/2422230343 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.997+0000 7f83abe00700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83ac109090 0x7f83ac1a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.997+0000 7f83ab400700 1 -- 172.21.15.40:0/2422230343 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83ac106760 msgr2=0x7f83ac199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.997+0000 7f83ab400700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83ac106760 0x7f83ac199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.997+0000 7f83ab400700 1 -- 172.21.15.40:0/2422230343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83ac109090 msgr2=0x7f83ac1a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.997+0000 7f83ab400700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83ac109090 0x7f83ac1a0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.997+0000 7f83ab400700 1 -- 172.21.15.40:0/2422230343 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83ac1a1160 con 0x7f83ac103e30 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.997+0000 7f83abe00700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83ac109090 0x7f83ac1a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.997+0000 7f83ab400700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83ac103e30 0x7f83ac199500 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f839c009990 tx=0x7f839c0057a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:57:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.998+0000 7f83a3e00700 1 -- 172.21.15.40:0/2422230343 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c026070 con 0x7f83ac103e30 2024-04-26T23:57:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.998+0000 7f83b1200700 1 -- 172.21.15.40:0/2422230343 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83ac1a13f0 con 0x7f83ac103e30 2024-04-26T23:57:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.998+0000 7f83b1200700 1 -- 172.21.15.40:0/2422230343 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83ac1a18a0 con 0x7f83ac103e30 2024-04-26T23:57:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.998+0000 7f83a3e00700 1 -- 172.21.15.40:0/2422230343 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f839c0095c0 con 0x7f83ac103e30 2024-04-26T23:57:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.998+0000 7f83a3e00700 1 -- 172.21.15.40:0/2422230343 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c017d20 con 0x7f83ac103e30 2024-04-26T23:57:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:52.999+0000 7f83b1200700 1 -- 172.21.15.40:0/2422230343 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f838c005360 con 0x7f83ac103e30 2024-04-26T23:57:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.001+0000 7f83a3e00700 1 -- 172.21.15.40:0/2422230343 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f839c029070 con 0x7f83ac103e30 2024-04-26T23:57:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.002+0000 7f83a3e00700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f83900a3c40 0x7f83900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.003+0000 7f83a3e00700 1 -- 172.21.15.40:0/2422230343 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f839c0d0dc0 con 0x7f83ac103e30 2024-04-26T23:57:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.003+0000 7f83aaa00700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f83900a3c40 0x7f83900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.003+0000 7f83aaa00700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f83900a3c40 0x7f83900a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8398009d60 tx=0x7f8398009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:57:53.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.004+0000 7f83a3e00700 1 -- 172.21.15.40:0/2422230343 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f839c05f060 con 0x7f83ac103e30 2024-04-26T23:57:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.243+0000 7f83b1200700 1 -- 172.21.15.40:0/2422230343 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f838c005920 con 0x7f83ac103e30 2024-04-26T23:57:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.243+0000 7f83a3e00700 1 -- 172.21.15.40:0/2422230343 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f839c013070 con 0x7f83ac103e30 2024-04-26T23:57:53.245 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:57:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.246+0000 7f83b1200700 1 -- 172.21.15.40:0/2422230343 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f83900a3c40 msgr2=0x7f83900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.246+0000 7f83b1200700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f83900a3c40 0x7f83900a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8398009d60 tx=0x7f8398009260 comp rx=0 tx=0).stop 2024-04-26T23:57:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.246+0000 7f83b1200700 1 -- 172.21.15.40:0/2422230343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83ac103e30 msgr2=0x7f83ac199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.246+0000 7f83b1200700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83ac103e30 0x7f83ac199500 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f839c009990 tx=0x7f839c0057a0 comp rx=0 tx=0).stop 2024-04-26T23:57:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.246+0000 7f83b1200700 1 -- 172.21.15.40:0/2422230343 shutdown_connections 2024-04-26T23:57:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.246+0000 7f83b1200700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f83900a3c40 0x7f83900a6100 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.246+0000 7f83b1200700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83ac109090 0x7f83ac1a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.246+0000 7f83b1200700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83ac103e30 0x7f83ac199500 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.246+0000 7f83b1200700 1 --2- 172.21.15.40:0/2422230343 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f83ac106760 0x7f83ac199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.246+0000 7f83b1200700 1 -- 172.21.15.40:0/2422230343 >> 172.21.15.40:0/2422230343 conn(0x7f83ac0fd840 msgr2=0x7f83ac104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:57:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.246+0000 7f83b1200700 1 -- 172.21.15.40:0/2422230343 shutdown_connections 2024-04-26T23:57:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:53.247+0000 7f83b1200700 1 -- 172.21.15.40:0/2422230343 wait complete. 2024-04-26T23:57:53.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:53 smithi103 ceph-mon[32952]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T23:57:53.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:53.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:53.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:53.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:53.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:53.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:53.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:53.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:53.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:53.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:53 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:53.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:53 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2422230343' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T23:57:53.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:53 smithi040 ceph-mon[28095]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T23:57:53.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:53.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:53.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:53.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:53.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:53.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:53.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:53.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:53.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:53.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:53 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:53.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:53 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2422230343' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T23:57:54.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:53 smithi080 ceph-mon[32362]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T23:57:54.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:53 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:54.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:53 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:54.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:53 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:54.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:53 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:54.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:53 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:54.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:53 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:57:54.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:53 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:54.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:53 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:54.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:53 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:54.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:53 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:57:54.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:53 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2422230343' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T23:57:54.087 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1714175871,"num_in_osds":12,"osd_in_since":1714175852,"num_remapped_pgs":0} 2024-04-26T23:57:54.088 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T23:57:54.546 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:57:54.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:54 smithi103 ceph-mon[32952]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:57:54.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:54 smithi103 ceph-mon[32952]: Detected new or changed devices on smithi103 2024-04-26T23:57:54.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:54 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi103 to 4272M 2024-04-26T23:57:54.883 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:54 smithi040 ceph-mon[28095]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:57:54.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:54 smithi040 ceph-mon[28095]: Detected new or changed devices on smithi103 2024-04-26T23:57:54.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:54 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi103 to 4272M 2024-04-26T23:57:55.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:54 smithi080 ceph-mon[32362]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:57:55.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:54 smithi080 ceph-mon[32362]: Detected new or changed devices on smithi103 2024-04-26T23:57:55.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:54 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi103 to 4272M 2024-04-26T23:57:55.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:55 smithi103 ceph-mon[32952]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:57:55.890 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:55 smithi040 ceph-mon[28095]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:57:56.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:55 smithi080 ceph-mon[32362]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:57:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.708+0000 7f0faea00700 1 -- 172.21.15.40:0/3702323246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fa8109960 msgr2=0x7f0fa8109de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.708+0000 7f0faea00700 1 --2- 172.21.15.40:0/3702323246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fa8109960 0x7f0fa8109de0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0f94009990 tx=0x7f0f94009ca0 comp rx=0 tx=0).stop 2024-04-26T23:57:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.708+0000 7f0faea00700 1 -- 172.21.15.40:0/3702323246 shutdown_connections 2024-04-26T23:57:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.708+0000 7f0faea00700 1 --2- 172.21.15.40:0/3702323246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0fa810a320 0x7f0fa81167c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.708+0000 7f0faea00700 1 --2- 172.21.15.40:0/3702323246 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fa810e830 0x7f0fa810ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.708+0000 7f0faea00700 1 --2- 172.21.15.40:0/3702323246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fa8109960 0x7f0fa8109de0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.708+0000 7f0faea00700 1 -- 172.21.15.40:0/3702323246 >> 172.21.15.40:0/3702323246 conn(0x7f0fa80791b0 msgr2=0x7f0fa8108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:57:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.709+0000 7f0faea00700 1 -- 172.21.15.40:0/3702323246 shutdown_connections 2024-04-26T23:57:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.709+0000 7f0faea00700 1 -- 172.21.15.40:0/3702323246 wait complete. 2024-04-26T23:57:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.710+0000 7f0faea00700 1 Processor -- start 2024-04-26T23:57:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.710+0000 7f0faea00700 1 -- start start 2024-04-26T23:57:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.710+0000 7f0faea00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fa8109960 0x7f0fa81a0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.710+0000 7f0faea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0fa810a320 0x7f0fa81a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.710+0000 7f0faea00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fa810e830 0x7f0fa81a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.710+0000 7f0faea00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa8119b50 con 0x7f0fa8109960 2024-04-26T23:57:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.710+0000 7f0faea00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa81199d0 con 0x7f0fa810e830 2024-04-26T23:57:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.710+0000 7f0faea00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa8119cd0 con 0x7f0fa810a320 2024-04-26T23:57:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.711+0000 7f0fad600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fa8109960 0x7f0fa81a0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.711+0000 7f0fad600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fa8109960 0x7f0fa81a0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55766/0 (socket says 172.21.15.40:55766) 2024-04-26T23:57:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.711+0000 7f0fad600700 1 -- 172.21.15.40:0/4074152930 learned_addr learned my addr 172.21.15.40:0/4074152930 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:57:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.711+0000 7f0fae000700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fa810e830 0x7f0fa81a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.711+0000 7f0facc00700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0fa810a320 0x7f0fa81a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.711+0000 7f0fad600700 1 -- 172.21.15.40:0/4074152930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0fa810a320 msgr2=0x7f0fa81a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.711+0000 7f0fad600700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0fa810a320 0x7f0fa81a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.711+0000 7f0fad600700 1 -- 172.21.15.40:0/4074152930 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fa810e830 msgr2=0x7f0fa81a4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.711+0000 7f0fad600700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fa810e830 0x7f0fa81a4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.711+0000 7f0fad600700 1 -- 172.21.15.40:0/4074152930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa81a5050 con 0x7f0fa8109960 2024-04-26T23:57:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.711+0000 7f0fae000700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fa810e830 0x7f0fa81a4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:57:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.712+0000 7f0fad600700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fa8109960 0x7f0fa81a0040 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c00e9a0 tx=0x7f0f9c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:57:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.712+0000 7f0fa6000700 1 -- 172.21.15.40:0/4074152930 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c00c9c0 con 0x7f0fa8109960 2024-04-26T23:57:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.712+0000 7f0faea00700 1 -- 172.21.15.40:0/4074152930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fa81a5340 con 0x7f0fa8109960 2024-04-26T23:57:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.712+0000 7f0fa6000700 1 -- 172.21.15.40:0/4074152930 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f9c00cb60 con 0x7f0fa8109960 2024-04-26T23:57:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.712+0000 7f0faea00700 1 -- 172.21.15.40:0/4074152930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fa81acbc0 con 0x7f0fa8109960 2024-04-26T23:57:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.712+0000 7f0fa6000700 1 -- 172.21.15.40:0/4074152930 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c018620 con 0x7f0fa8109960 2024-04-26T23:57:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.713+0000 7f0faea00700 1 -- 172.21.15.40:0/4074152930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fa806a110 con 0x7f0fa8109960 2024-04-26T23:57:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.715+0000 7f0fa6000700 1 -- 172.21.15.40:0/4074152930 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0f9c010430 con 0x7f0fa8109960 2024-04-26T23:57:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.717+0000 7f0fa6000700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f900a3c40 0x7f0f900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:57:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.717+0000 7f0fa6000700 1 -- 172.21.15.40:0/4074152930 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0f9c0d01b0 con 0x7f0fa8109960 2024-04-26T23:57:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.717+0000 7f0facc00700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f900a3c40 0x7f0f900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:57:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.718+0000 7f0facc00700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f900a3c40 0x7f0f900a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8109510 tx=0x7f0f9400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:57:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.721+0000 7f0fa6000700 1 -- 172.21.15.40:0/4074152930 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0f9c09cf90 con 0x7f0fa8109960 2024-04-26T23:57:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.956+0000 7f0faea00700 1 -- 172.21.15.40:0/4074152930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fa802e490 con 0x7f0fa8109960 2024-04-26T23:57:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.958+0000 7f0fa6000700 1 -- 172.21.15.40:0/4074152930 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18418 (secure 0 0 0) 0x7f0f9c085e20 con 0x7f0fa8109960 2024-04-26T23:57:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:57:56.961 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:57:52.532046+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:57:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.962+0000 7f0faea00700 1 -- 172.21.15.40:0/4074152930 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f900a3c40 msgr2=0x7f0f900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.962+0000 7f0faea00700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f900a3c40 0x7f0f900a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8109510 tx=0x7f0f9400b260 comp rx=0 tx=0).stop 2024-04-26T23:57:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.962+0000 7f0faea00700 1 -- 172.21.15.40:0/4074152930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fa8109960 msgr2=0x7f0fa81a0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:57:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.962+0000 7f0faea00700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fa8109960 0x7f0fa81a0040 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c00e9a0 tx=0x7f0f9c00ecb0 comp rx=0 tx=0).stop 2024-04-26T23:57:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.962+0000 7f0faea00700 1 -- 172.21.15.40:0/4074152930 shutdown_connections 2024-04-26T23:57:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.962+0000 7f0faea00700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f900a3c40 0x7f0f900a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.962+0000 7f0faea00700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0fa810e830 0x7f0fa81a4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.962+0000 7f0faea00700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fa8109960 0x7f0fa81a0040 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.962+0000 7f0faea00700 1 --2- 172.21.15.40:0/4074152930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0fa810a320 0x7f0fa81a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:57:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.962+0000 7f0faea00700 1 -- 172.21.15.40:0/4074152930 >> 172.21.15.40:0/4074152930 conn(0x7f0fa80791b0 msgr2=0x7f0fa8079870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:57:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.962+0000 7f0faea00700 1 -- 172.21.15.40:0/4074152930 shutdown_connections 2024-04-26T23:57:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:57:56.962+0000 7f0faea00700 1 -- 172.21.15.40:0/4074152930 wait complete. 2024-04-26T23:57:57.794 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:53:32.708818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:57:57.794 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T23:57:58.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:57 smithi080 ceph-mon[32362]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:57:58.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:57 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/4074152930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:57:58.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:57 smithi103 ceph-mon[32952]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:57:58.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:57 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/4074152930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:57:58.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:57 smithi040 ceph-mon[28095]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:57:58.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:57 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/4074152930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:57:58.251 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:57:59.955 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:57:59 smithi040 ceph-mon[28095]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:00.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:57:59 smithi080 ceph-mon[32362]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:00.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:57:59 smithi103 ceph-mon[32952]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:00.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.503+0000 7f4cbfe00700 1 -- 172.21.15.40:0/1414463277 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cb8105640 msgr2=0x7f4cb8105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:00.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.503+0000 7f4cbfe00700 1 --2- 172.21.15.40:0/1414463277 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cb8105640 0x7f4cb8105ac0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4009990 tx=0x7f4ca4009ca0 comp rx=0 tx=0).stop 2024-04-26T23:58:00.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.503+0000 7f4cbfe00700 1 -- 172.21.15.40:0/1414463277 shutdown_connections 2024-04-26T23:58:00.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.503+0000 7f4cbfe00700 1 --2- 172.21.15.40:0/1414463277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cb8106000 0x7f4cb8112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:00.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.503+0000 7f4cbfe00700 1 --2- 172.21.15.40:0/1414463277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cb810b640 0x7f4cb810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:00.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.503+0000 7f4cbfe00700 1 --2- 172.21.15.40:0/1414463277 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cb8105640 0x7f4cb8105ac0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:00.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.503+0000 7f4cbfe00700 1 -- 172.21.15.40:0/1414463277 >> 172.21.15.40:0/1414463277 conn(0x7f4cb8101130 msgr2=0x7f4cb8103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:00.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.503+0000 7f4cbfe00700 1 -- 172.21.15.40:0/1414463277 shutdown_connections 2024-04-26T23:58:00.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.503+0000 7f4cbfe00700 1 -- 172.21.15.40:0/1414463277 wait complete. 2024-04-26T23:58:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.504+0000 7f4cbfe00700 1 Processor -- start 2024-04-26T23:58:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.504+0000 7f4cbfe00700 1 -- start start 2024-04-26T23:58:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbfe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cb8105640 0x7f4cb81a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbfe00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cb8106000 0x7f4cb81a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbfe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cb810b640 0x7f4cb819bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbfe00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb8115890 con 0x7f4cb810b640 2024-04-26T23:58:00.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbfe00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb8115710 con 0x7f4cb8106000 2024-04-26T23:58:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbfe00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb8115a10 con 0x7f4cb8105640 2024-04-26T23:58:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbd400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cb8105640 0x7f4cb81a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbde00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cb810b640 0x7f4cb819bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbd400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cb8105640 0x7f4cb81a1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36348/0 (socket says 172.21.15.40:36348) 2024-04-26T23:58:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbd400700 1 -- 172.21.15.40:0/3472471608 learned_addr learned my addr 172.21.15.40:0/3472471608 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbca00700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cb8106000 0x7f4cb81a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbd400700 1 -- 172.21.15.40:0/3472471608 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cb8106000 msgr2=0x7f4cb81a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbd400700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cb8106000 0x7f4cb81a1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbd400700 1 -- 172.21.15.40:0/3472471608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cb810b640 msgr2=0x7f4cb819bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.505+0000 7f4cbd400700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cb810b640 0x7f4cb819bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.506+0000 7f4cbd400700 1 -- 172.21.15.40:0/3472471608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb819c410 con 0x7f4cb8105640 2024-04-26T23:58:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.506+0000 7f4cbd400700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cb8105640 0x7f4cb81a1a60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00e9c0 tx=0x7f4cac00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.506+0000 7f4cb6000700 1 -- 172.21.15.40:0/3472471608 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac00cbd0 con 0x7f4cb8105640 2024-04-26T23:58:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.506+0000 7f4cbfe00700 1 -- 172.21.15.40:0/3472471608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb819c700 con 0x7f4cb8105640 2024-04-26T23:58:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.506+0000 7f4cb6000700 1 -- 172.21.15.40:0/3472471608 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cac00ee60 con 0x7f4cb8105640 2024-04-26T23:58:00.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.506+0000 7f4cb6000700 1 -- 172.21.15.40:0/3472471608 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac0055c0 con 0x7f4cb8105640 2024-04-26T23:58:00.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.506+0000 7f4cbfe00700 1 -- 172.21.15.40:0/3472471608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb81a89e0 con 0x7f4cb8105640 2024-04-26T23:58:00.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.508+0000 7f4cbfe00700 1 -- 172.21.15.40:0/3472471608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cb8106d80 con 0x7f4cb8105640 2024-04-26T23:58:00.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.509+0000 7f4cb6000700 1 -- 172.21.15.40:0/3472471608 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4cac004750 con 0x7f4cb8105640 2024-04-26T23:58:00.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.511+0000 7f4cb6000700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4ca00a3c40 0x7f4ca00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:00.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.511+0000 7f4cb6000700 1 -- 172.21.15.40:0/3472471608 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4cac0d01e0 con 0x7f4cb8105640 2024-04-26T23:58:00.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.511+0000 7f4cbca00700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4ca00a3c40 0x7f4ca00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:00.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.512+0000 7f4cbca00700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4ca00a3c40 0x7f4ca00a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4cb819d280 tx=0x7f4ca400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:00.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.515+0000 7f4cb6000700 1 -- 172.21.15.40:0/3472471608 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4cac09cfc0 con 0x7f4cb8105640 2024-04-26T23:58:00.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.747+0000 7f4cbfe00700 1 -- 172.21.15.40:0/3472471608 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4cb819cff0 con 0x7f4cb8105640 2024-04-26T23:58:00.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.748+0000 7f4cb6000700 1 -- 172.21.15.40:0/3472471608 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f4cac09c8c0 con 0x7f4cb8105640 2024-04-26T23:58:00.750 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:58:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.751+0000 7f4cbfe00700 1 -- 172.21.15.40:0/3472471608 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4ca00a3c40 msgr2=0x7f4ca00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.751+0000 7f4cbfe00700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4ca00a3c40 0x7f4ca00a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4cb819d280 tx=0x7f4ca400b260 comp rx=0 tx=0).stop 2024-04-26T23:58:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.751+0000 7f4cbfe00700 1 -- 172.21.15.40:0/3472471608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cb8105640 msgr2=0x7f4cb81a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.751+0000 7f4cbfe00700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cb8105640 0x7f4cb81a1a60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00e9c0 tx=0x7f4cac00ecd0 comp rx=0 tx=0).stop 2024-04-26T23:58:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.752+0000 7f4cbfe00700 1 -- 172.21.15.40:0/3472471608 shutdown_connections 2024-04-26T23:58:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.752+0000 7f4cbfe00700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4ca00a3c40 0x7f4ca00a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.752+0000 7f4cbfe00700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cb810b640 0x7f4cb819bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.752+0000 7f4cbfe00700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cb8105640 0x7f4cb81a1a60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.752+0000 7f4cbfe00700 1 --2- 172.21.15.40:0/3472471608 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cb8106000 0x7f4cb81a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.752+0000 7f4cbfe00700 1 -- 172.21.15.40:0/3472471608 >> 172.21.15.40:0/3472471608 conn(0x7f4cb8101130 msgr2=0x7f4cb8101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.752+0000 7f4cbfe00700 1 -- 172.21.15.40:0/3472471608 shutdown_connections 2024-04-26T23:58:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:00.752+0000 7f4cbfe00700 1 -- 172.21.15.40:0/3472471608 wait complete. 2024-04-26T23:58:01.618 INFO:tasks.cephadm:Adding mds.a on smithi040 2024-04-26T23:58:01.618 INFO:tasks.cephadm:Adding mds.d on smithi040 2024-04-26T23:58:01.618 INFO:tasks.cephadm:Adding mds.g on smithi040 2024-04-26T23:58:01.619 INFO:tasks.cephadm:Adding mds.j on smithi040 2024-04-26T23:58:01.619 INFO:tasks.cephadm:Adding mds.b on smithi080 2024-04-26T23:58:01.619 INFO:tasks.cephadm:Adding mds.e on smithi080 2024-04-26T23:58:01.619 INFO:tasks.cephadm:Adding mds.h on smithi080 2024-04-26T23:58:01.619 INFO:tasks.cephadm:Adding mds.k on smithi080 2024-04-26T23:58:01.619 INFO:tasks.cephadm:Adding mds.c on smithi103 2024-04-26T23:58:01.619 INFO:tasks.cephadm:Adding mds.f on smithi103 2024-04-26T23:58:01.619 INFO:tasks.cephadm:Adding mds.i on smithi103 2024-04-26T23:58:01.619 INFO:tasks.cephadm:Adding mds.l on smithi103 2024-04-26T23:58:01.619 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi080=b;smithi080=e;smithi080=h;smithi080=k;smithi103=c;smithi103=f;smithi103=i;smithi103=l' 2024-04-26T23:58:01.870 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:58:01.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:01 smithi040 ceph-mon[28095]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:01.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:01 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3472471608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:58:01.991 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:01 smithi103 ceph-mon[32952]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:01.992 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:01 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3472471608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:58:02.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:01 smithi080 ceph-mon[32362]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:02.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:01 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3472471608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:58:03.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.558+0000 7f767bf00700 1 -- 172.21.15.103:0/3474601343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7674102370 msgr2=0x7f7674102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:03.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.558+0000 7f767bf00700 1 --2- 172.21.15.103:0/3474601343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7674102370 0x7f7674102790 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7670009990 tx=0x7f7670009ca0 comp rx=0 tx=0).stop 2024-04-26T23:58:03.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.558+0000 7f767bf00700 1 -- 172.21.15.103:0/3474601343 shutdown_connections 2024-04-26T23:58:03.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.558+0000 7f767bf00700 1 --2- 172.21.15.103:0/3474601343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7674103f10 0x7f767410a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.558+0000 7f767bf00700 1 --2- 172.21.15.103:0/3474601343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7674102370 0x7f7674102790 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.558+0000 7f767bf00700 1 --2- 172.21.15.103:0/3474601343 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7674103570 0x7f76741039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.558+0000 7f767bf00700 1 -- 172.21.15.103:0/3474601343 >> 172.21.15.103:0/3474601343 conn(0x7f76740fd950 msgr2=0x7f76740ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.558+0000 7f767bf00700 1 -- 172.21.15.103:0/3474601343 shutdown_connections 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.558+0000 7f767bf00700 1 -- 172.21.15.103:0/3474601343 wait complete. 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.559+0000 7f767bf00700 1 Processor -- start 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.559+0000 7f767bf00700 1 -- start start 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.559+0000 7f767bf00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7674102370 0x7f7674199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.559+0000 7f767bf00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7674103570 0x7f7674199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.559+0000 7f767bf00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7674103f10 0x7f76741a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.559+0000 7f767bf00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f767410de70 con 0x7f7674103570 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.559+0000 7f767bf00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f767410dcf0 con 0x7f7674103f10 2024-04-26T23:58:03.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.559+0000 7f767bf00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f767410dff0 con 0x7f7674102370 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.559+0000 7f7679c9c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7674102370 0x7f7674199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.559+0000 7f7679c9c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7674102370 0x7f7674199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56950/0 (socket says 172.21.15.103:56950) 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.559+0000 7f7679c9c700 1 -- 172.21.15.103:0/2597480891 learned_addr learned my addr 172.21.15.103:0/2597480891 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.559+0000 7f767a49d700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7674103f10 0x7f76741a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.560+0000 7f7679c9c700 1 -- 172.21.15.103:0/2597480891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7674103f10 msgr2=0x7f76741a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.560+0000 7f7679c9c700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7674103f10 0x7f76741a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.560+0000 7f7679c9c700 1 -- 172.21.15.103:0/2597480891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7674103570 msgr2=0x7f7674199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.560+0000 7f7679c9c700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7674103570 0x7f7674199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.560+0000 7f7679c9c700 1 -- 172.21.15.103:0/2597480891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76741a1240 con 0x7f7674102370 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.560+0000 7f767a49d700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7674103f10 0x7f76741a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.560+0000 7f7679c9c700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7674102370 0x7f7674199630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f76700098e0 tx=0x7f7670003c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.560+0000 7f766affd700 1 -- 172.21.15.103:0/2597480891 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7670027070 con 0x7f7674102370 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.560+0000 7f766affd700 1 -- 172.21.15.103:0/2597480891 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f767000bd90 con 0x7f7674102370 2024-04-26T23:58:03.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.560+0000 7f767bf00700 1 -- 172.21.15.103:0/2597480891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76741a14d0 con 0x7f7674102370 2024-04-26T23:58:03.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.560+0000 7f766affd700 1 -- 172.21.15.103:0/2597480891 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767000f460 con 0x7f7674102370 2024-04-26T23:58:03.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.560+0000 7f767bf00700 1 -- 172.21.15.103:0/2597480891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76741a1920 con 0x7f7674102370 2024-04-26T23:58:03.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.561+0000 7f767bf00700 1 -- 172.21.15.103:0/2597480891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f767406a140 con 0x7f7674102370 2024-04-26T23:58:03.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.562+0000 7f766affd700 1 -- 172.21.15.103:0/2597480891 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f767002a070 con 0x7f7674102370 2024-04-26T23:58:03.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.563+0000 7f766affd700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f76600a3c40 0x7f76600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:03.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.563+0000 7f766affd700 1 -- 172.21.15.103:0/2597480891 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f76700d15a0 con 0x7f7674102370 2024-04-26T23:58:03.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.563+0000 7f767949b700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f76600a3c40 0x7f76600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:03.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.564+0000 7f767949b700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f76600a3c40 0x7f76600a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7664005fd0 tx=0x7f7664005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:03.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.566+0000 7f766affd700 1 -- 172.21.15.103:0/2597480891 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f767406a140 con 0x7f7674102370 2024-04-26T23:58:03.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.774+0000 7f767bf00700 1 -- 172.21.15.103:0/2597480891 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi080=b;smithi080=e;smithi080=h;smithi080=k;smithi103=c;smithi103=f;smithi103=i;smithi103=l", "target": ["mon-mgr", ""]}) -- 0x7f767406c3a0 con 0x7f76600a3c40 2024-04-26T23:58:03.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.791+0000 7f766affd700 1 -- 172.21.15.103:0/2597480891 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f767406c3a0 con 0x7f76600a3c40 2024-04-26T23:58:03.793 INFO:teuthology.orchestra.run.smithi103.stdout:Scheduled mds.all update... 2024-04-26T23:58:03.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.793+0000 7f767bf00700 1 -- 172.21.15.103:0/2597480891 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f76600a3c40 msgr2=0x7f76600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:03.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.793+0000 7f767bf00700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f76600a3c40 0x7f76600a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7664005fd0 tx=0x7f7664005d00 comp rx=0 tx=0).stop 2024-04-26T23:58:03.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.793+0000 7f767bf00700 1 -- 172.21.15.103:0/2597480891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7674102370 msgr2=0x7f7674199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:03.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.793+0000 7f767bf00700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7674102370 0x7f7674199630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f76700098e0 tx=0x7f7670003c00 comp rx=0 tx=0).stop 2024-04-26T23:58:03.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.793+0000 7f767bf00700 1 -- 172.21.15.103:0/2597480891 shutdown_connections 2024-04-26T23:58:03.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.793+0000 7f767bf00700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f76600a3c40 0x7f76600a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:03.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.793+0000 7f767bf00700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7674103f10 0x7f76741a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:03.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.793+0000 7f767bf00700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7674102370 0x7f7674199630 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:03.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.793+0000 7f767bf00700 1 --2- 172.21.15.103:0/2597480891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7674103570 0x7f7674199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:03.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.793+0000 7f767bf00700 1 -- 172.21.15.103:0/2597480891 >> 172.21.15.103:0/2597480891 conn(0x7f76740fd950 msgr2=0x7f767406bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:03.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.794+0000 7f767bf00700 1 -- 172.21.15.103:0/2597480891 shutdown_connections 2024-04-26T23:58:03.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-26T23:58:03.794+0000 7f767bf00700 1 -- 172.21.15.103:0/2597480891 wait complete. 2024-04-26T23:58:04.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:03 smithi080 ceph-mon[32362]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:04.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:03 smithi103 ceph-mon[32952]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:04.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:03 smithi040 ceph-mon[28095]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:04.522 DEBUG:teuthology.orchestra.run.smithi040:mds.a> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.a.service 2024-04-26T23:58:04.527 DEBUG:teuthology.orchestra.run.smithi040:mds.d> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.d.service 2024-04-26T23:58:04.573 DEBUG:teuthology.orchestra.run.smithi040:mds.g> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.g.service 2024-04-26T23:58:04.620 DEBUG:teuthology.orchestra.run.smithi040:mds.j> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.j.service 2024-04-26T23:58:04.666 DEBUG:teuthology.orchestra.run.smithi080:mds.b> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.b.service 2024-04-26T23:58:04.669 DEBUG:teuthology.orchestra.run.smithi080:mds.e> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.e.service 2024-04-26T23:58:04.712 DEBUG:teuthology.orchestra.run.smithi080:mds.h> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.h.service 2024-04-26T23:58:04.755 DEBUG:teuthology.orchestra.run.smithi080:mds.k> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.k.service 2024-04-26T23:58:04.798 DEBUG:teuthology.orchestra.run.smithi103:mds.c> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.c.service 2024-04-26T23:58:04.800 DEBUG:teuthology.orchestra.run.smithi103:mds.f> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.f.service 2024-04-26T23:58:04.843 DEBUG:teuthology.orchestra.run.smithi103:mds.i> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.i.service 2024-04-26T23:58:04.886 DEBUG:teuthology.orchestra.run.smithi103:mds.l> sudo journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.l.service 2024-04-26T23:58:04.929 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-26T23:58:04.929 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:58:04.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:04 smithi040 ceph-mon[28095]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi080=b;smithi080=e;smithi080=h;smithi080=k;smithi103=c;smithi103=f;smithi103=i;smithi103=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:58:04.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:04 smithi040 ceph-mon[28095]: Saving service mds.all spec with placement smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi080=b;smithi080=e;smithi080=h;smithi080=k;smithi103=c;smithi103=f;smithi103=i;smithi103=l;count:12 2024-04-26T23:58:04.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:04.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:04 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:58:05.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:04 smithi080 ceph-mon[32362]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi080=b;smithi080=e;smithi080=h;smithi080=k;smithi103=c;smithi103=f;smithi103=i;smithi103=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:58:05.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:04 smithi080 ceph-mon[32362]: Saving service mds.all spec with placement smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi080=b;smithi080=e;smithi080=h;smithi080=k;smithi103=c;smithi103=f;smithi103=i;smithi103=l;count:12 2024-04-26T23:58:05.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:05.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:04 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:58:05.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:04 smithi103 ceph-mon[32952]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi080=b;smithi080=e;smithi080=h;smithi080=k;smithi103=c;smithi103=f;smithi103=i;smithi103=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:58:05.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:04 smithi103 ceph-mon[32952]: Saving service mds.all spec with placement smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi080=b;smithi080=e;smithi080=h;smithi080=k;smithi103=c;smithi103=f;smithi103=i;smithi103=l;count:12 2024-04-26T23:58:05.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:05.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:04 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:58:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.148+0000 7f82b3e00640 1 -- 172.21.15.40:0/3822538124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f82ac103b60 msgr2=0x7f82ac103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.148+0000 7f82b3e00640 1 --2- 172.21.15.40:0/3822538124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f82ac103b60 0x7f82ac103fe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8298009900 tx=0x7f829802a400 comp rx=0 tx=0).stop 2024-04-26T23:58:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.148+0000 7f82b3e00640 1 -- 172.21.15.40:0/3822538124 shutdown_connections 2024-04-26T23:58:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.148+0000 7f82b3e00640 1 --2- 172.21.15.40:0/3822538124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82ac104520 0x7f82ac10adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.148+0000 7f82b3e00640 1 --2- 172.21.15.40:0/3822538124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f82ac103b60 0x7f82ac103fe0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.148+0000 7f82b3e00640 1 --2- 172.21.15.40:0/3822538124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82ac102970 0x7f82ac102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.148+0000 7f82b3e00640 1 -- 172.21.15.40:0/3822538124 >> 172.21.15.40:0/3822538124 conn(0x7f82ac0fdee0 msgr2=0x7f82ac100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.148+0000 7f82b3e00640 1 -- 172.21.15.40:0/3822538124 shutdown_connections 2024-04-26T23:58:05.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.148+0000 7f82b3e00640 1 -- 172.21.15.40:0/3822538124 wait complete. 2024-04-26T23:58:05.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.149+0000 7f82b3e00640 1 Processor -- start 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.149+0000 7f82b3e00640 1 -- start start 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.149+0000 7f82b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82ac102970 0x7f82ac19adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.149+0000 7f82b3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f82ac103b60 0x7f82ac19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.149+0000 7f82b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82ac104520 0x7f82ac1a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.149+0000 7f82b3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f82ac10e4d0 con 0x7f82ac104520 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.149+0000 7f82b3e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f82ac10e350 con 0x7f82ac103b60 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.149+0000 7f82b3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f82ac10e650 con 0x7f82ac102970 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.150+0000 7f82b1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82ac102970 0x7f82ac19adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.150+0000 7f82b1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82ac102970 0x7f82ac19adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36374/0 (socket says 172.21.15.40:36374) 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.150+0000 7f82b1400640 1 -- 172.21.15.40:0/974395025 learned_addr learned my addr 172.21.15.40:0/974395025 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.150+0000 7f82b0a00640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f82ac103b60 0x7f82ac19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.150+0000 7f82b1e00640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82ac104520 0x7f82ac1a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:05.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.150+0000 7f82b1400640 1 -- 172.21.15.40:0/974395025 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f82ac103b60 msgr2=0x7f82ac19b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.150+0000 7f82b1400640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f82ac103b60 0x7f82ac19b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.150+0000 7f82b1400640 1 -- 172.21.15.40:0/974395025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82ac104520 msgr2=0x7f82ac1a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.150+0000 7f82b1400640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82ac104520 0x7f82ac1a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.150+0000 7f82b1400640 1 -- 172.21.15.40:0/974395025 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82ac1a29c0 con 0x7f82ac102970 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.150+0000 7f82b1e00640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82ac104520 0x7f82ac1a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.150+0000 7f82b1400640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82ac102970 0x7f82ac19adb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f82a000e9c0 tx=0x7f82a000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.151+0000 7f82aa000640 1 -- 172.21.15.40:0/974395025 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a000cd80 con 0x7f82ac102970 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.151+0000 7f82b3e00640 1 -- 172.21.15.40:0/974395025 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82ac1a2cb0 con 0x7f82ac102970 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.151+0000 7f82aa000640 1 -- 172.21.15.40:0/974395025 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82a00035f0 con 0x7f82ac102970 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.151+0000 7f82aa000640 1 -- 172.21.15.40:0/974395025 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a0010690 con 0x7f82ac102970 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.151+0000 7f82b3e00640 1 -- 172.21.15.40:0/974395025 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82ac1a31a0 con 0x7f82ac102970 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.152+0000 7f82b3e00640 1 -- 172.21.15.40:0/974395025 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82ac1a3430 con 0x7f82ac102970 2024-04-26T23:58:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.154+0000 7f82aa000640 1 -- 172.21.15.40:0/974395025 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f82a00037a0 con 0x7f82ac102970 2024-04-26T23:58:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.155+0000 7f82aa000640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f82940a3bb0 0x7f82940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.155+0000 7f82aa000640 1 -- 172.21.15.40:0/974395025 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f82a00d0480 con 0x7f82ac102970 2024-04-26T23:58:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.155+0000 7f82b0a00640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f82940a3bb0 0x7f82940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.156+0000 7f82b0a00640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f82940a3bb0 0x7f82940a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8298031500 tx=0x7f8298005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:05.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.159+0000 7f82aa000640 1 -- 172.21.15.40:0/974395025 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f82a009d260 con 0x7f82ac102970 2024-04-26T23:58:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.368+0000 7f82b3e00640 1 -- 172.21.15.40:0/974395025 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82ac1a3600 con 0x7f82ac102970 2024-04-26T23:58:05.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.369+0000 7f82aa000640 1 -- 172.21.15.40:0/974395025 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18418 (secure 0 0 0) 0x7f82a009cb60 con 0x7f82ac102970 2024-04-26T23:58:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:05.372 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:57:52.532046+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:58:05.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.372+0000 7f82b3e00640 1 -- 172.21.15.40:0/974395025 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f82940a3bb0 msgr2=0x7f82940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:05.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.372+0000 7f82b3e00640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f82940a3bb0 0x7f82940a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8298031500 tx=0x7f8298005a20 comp rx=0 tx=0).stop 2024-04-26T23:58:05.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.372+0000 7f82b3e00640 1 -- 172.21.15.40:0/974395025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82ac102970 msgr2=0x7f82ac19adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:05.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.372+0000 7f82b3e00640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82ac102970 0x7f82ac19adb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f82a000e9c0 tx=0x7f82a000ee80 comp rx=0 tx=0).stop 2024-04-26T23:58:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.372+0000 7f82b3e00640 1 -- 172.21.15.40:0/974395025 shutdown_connections 2024-04-26T23:58:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.372+0000 7f82b3e00640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82ac104520 0x7f82ac1a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.372+0000 7f82b3e00640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f82ac103b60 0x7f82ac19b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.372+0000 7f82b3e00640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f82940a3bb0 0x7f82940a6070 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.372+0000 7f82b3e00640 1 --2- 172.21.15.40:0/974395025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82ac102970 0x7f82ac19adb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.372+0000 7f82b3e00640 1 -- 172.21.15.40:0/974395025 >> 172.21.15.40:0/974395025 conn(0x7f82ac0fdee0 msgr2=0x7f82ac074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.372+0000 7f82b3e00640 1 -- 172.21.15.40:0/974395025 shutdown_connections 2024-04-26T23:58:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.372+0000 7f82b3e00640 1 -- 172.21.15.40:0/974395025 wait complete. 2024-04-26T23:58:05.395 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:53:32.708818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:58:05.395 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:58:05.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.660+0000 7f99e7e00640 1 -- 172.21.15.40:0/2246811668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e810a980 msgr2=0x7f99e81112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:05.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.660+0000 7f99e7e00640 1 --2- 172.21.15.40:0/2246811668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e810a980 0x7f99e81112a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009ca0 tx=0x7f99e002a8c0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.660+0000 7f99e7e00640 1 -- 172.21.15.40:0/2246811668 shutdown_connections 2024-04-26T23:58:05.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.660+0000 7f99e7e00640 1 --2- 172.21.15.40:0/2246811668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e810a980 0x7f99e81112a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.660+0000 7f99e7e00640 1 --2- 172.21.15.40:0/2246811668 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99e8109fc0 0x7f99e810a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.660+0000 7f99e7e00640 1 --2- 172.21.15.40:0/2246811668 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f99e8108e70 0x7f99e8109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.660+0000 7f99e7e00640 1 -- 172.21.15.40:0/2246811668 >> 172.21.15.40:0/2246811668 conn(0x7f99e8077e00 msgr2=0x7f99e807a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:05.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.660+0000 7f99e7e00640 1 -- 172.21.15.40:0/2246811668 shutdown_connections 2024-04-26T23:58:05.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.660+0000 7f99e7e00640 1 -- 172.21.15.40:0/2246811668 wait complete. 2024-04-26T23:58:05.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.661+0000 7f99e7e00640 1 Processor -- start 2024-04-26T23:58:05.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.661+0000 7f99e7e00640 1 -- start start 2024-04-26T23:58:05.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.662+0000 7f99e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e8108e70 0x7f99e819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:05.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.662+0000 7f99e7e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f99e8109fc0 0x7f99e819f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:05.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.662+0000 7f99e7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99e810a980 0x7f99e81a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:05.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.662+0000 7f99e7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f99e81149f0 con 0x7f99e8108e70 2024-04-26T23:58:05.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.662+0000 7f99e7e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f99e8114870 con 0x7f99e8109fc0 2024-04-26T23:58:05.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.662+0000 7f99e7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f99e8114b70 con 0x7f99e810a980 2024-04-26T23:58:05.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.662+0000 7f99e6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e8108e70 0x7f99e819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:05.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.662+0000 7f99e6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e8108e70 0x7f99e819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33462/0 (socket says 172.21.15.40:33462) 2024-04-26T23:58:05.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.662+0000 7f99e6a00640 1 -- 172.21.15.40:0/3768075331 learned_addr learned my addr 172.21.15.40:0/3768075331 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:05.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.662+0000 7f99e7400640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99e810a980 0x7f99e81a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:05.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.662+0000 7f99e6000640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f99e8109fc0 0x7f99e819f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.663+0000 7f99e7400640 1 -- 172.21.15.40:0/3768075331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f99e8109fc0 msgr2=0x7f99e819f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.663+0000 7f99e7400640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f99e8109fc0 0x7f99e819f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.663+0000 7f99e7400640 1 -- 172.21.15.40:0/3768075331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e8108e70 msgr2=0x7f99e819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.663+0000 7f99e7400640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e8108e70 0x7f99e819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.663+0000 7f99e7400640 1 -- 172.21.15.40:0/3768075331 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e81a6bd0 con 0x7f99e810a980 2024-04-26T23:58:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.663+0000 7f99e6000640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f99e8109fc0 0x7f99e819f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.663+0000 7f99e7400640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99e810a980 0x7f99e81a6560 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f99e0031a20 tx=0x7f99e00316f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.663+0000 7f99df400640 1 -- 172.21.15.40:0/3768075331 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0003b70 con 0x7f99e810a980 2024-04-26T23:58:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.663+0000 7f99df400640 1 -- 172.21.15.40:0/3768075331 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99e0003d10 con 0x7f99e810a980 2024-04-26T23:58:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.663+0000 7f99e7e00640 1 -- 172.21.15.40:0/3768075331 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99e81a6e60 con 0x7f99e810a980 2024-04-26T23:58:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.664+0000 7f99df400640 1 -- 172.21.15.40:0/3768075331 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e002fae0 con 0x7f99e810a980 2024-04-26T23:58:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.664+0000 7f99e7e00640 1 -- 172.21.15.40:0/3768075331 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99e81a7370 con 0x7f99e810a980 2024-04-26T23:58:05.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.665+0000 7f99e7e00640 1 -- 172.21.15.40:0/3768075331 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99e81100b0 con 0x7f99e810a980 2024-04-26T23:58:05.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.666+0000 7f99df400640 1 -- 172.21.15.40:0/3768075331 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f99e0004610 con 0x7f99e810a980 2024-04-26T23:58:05.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.668+0000 7f99df400640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f99cc0a3c30 0x7f99cc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:05.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.668+0000 7f99df400640 1 -- 172.21.15.40:0/3768075331 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f99e00f1270 con 0x7f99e810a980 2024-04-26T23:58:05.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.668+0000 7f99e6a00640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f99cc0a3c30 0x7f99cc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:05.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.669+0000 7f99e6a00640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f99cc0a3c30 0x7f99cc0a60f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009c90 tx=0x7f99d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:05.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.672+0000 7f99df400640 1 -- 172.21.15.40:0/3768075331 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f99e00be050 con 0x7f99e810a980 2024-04-26T23:58:05.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.877+0000 7f99e7e00640 1 -- 172.21.15.40:0/3768075331 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f99e8063530 con 0x7f99e810a980 2024-04-26T23:58:05.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.878+0000 7f99df400640 1 -- 172.21.15.40:0/3768075331 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f99e00bd950 con 0x7f99e810a980 2024-04-26T23:58:05.880 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:58:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.881+0000 7f99e7e00640 1 -- 172.21.15.40:0/3768075331 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f99cc0a3c30 msgr2=0x7f99cc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.881+0000 7f99e7e00640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f99cc0a3c30 0x7f99cc0a60f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009c90 tx=0x7f99d8005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.881+0000 7f99e7e00640 1 -- 172.21.15.40:0/3768075331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99e810a980 msgr2=0x7f99e81a6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.881+0000 7f99e7e00640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99e810a980 0x7f99e81a6560 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f99e0031a20 tx=0x7f99e00316f0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.881+0000 7f99e7e00640 1 -- 172.21.15.40:0/3768075331 shutdown_connections 2024-04-26T23:58:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.881+0000 7f99e7e00640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f99cc0a3c30 0x7f99cc0a60f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.881+0000 7f99e7e00640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99e810a980 0x7f99e81a6560 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.881+0000 7f99e7e00640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f99e8109fc0 0x7f99e819f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.881+0000 7f99e7e00640 1 --2- 172.21.15.40:0/3768075331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99e8108e70 0x7f99e819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.882+0000 7f99e7e00640 1 -- 172.21.15.40:0/3768075331 >> 172.21.15.40:0/3768075331 conn(0x7f99e8077e00 msgr2=0x7f99e810d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.882+0000 7f99e7e00640 1 -- 172.21.15.40:0/3768075331 shutdown_connections 2024-04-26T23:58:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:05.882+0000 7f99e7e00640 1 -- 172.21.15.40:0/3768075331 wait complete. 2024-04-26T23:58:05.906 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:06.008 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:05 smithi103 ceph-mon[32952]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:06.008 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:05 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:06.008 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:05 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:58:06.009 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:05 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:06.009 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:05 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:06.009 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:05 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:06.009 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:05 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:06.009 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:05 smithi103 ceph-mon[32952]: Deploying daemon mds.l on smithi103 2024-04-26T23:58:06.009 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:05 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/974395025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:06.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:05 smithi080 ceph-mon[32362]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:06.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:05 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:06.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:05 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:58:06.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:05 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:06.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:05 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:06.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:05 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:06.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:05 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:06.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:05 smithi080 ceph-mon[32362]: Deploying daemon mds.l on smithi103 2024-04-26T23:58:06.051 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:05 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/974395025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:06.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.121+0000 7fbf86800640 1 -- 172.21.15.40:0/640210221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbf801039a0 msgr2=0x7fbf80103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:06.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.121+0000 7fbf86800640 1 --2- 172.21.15.40:0/640210221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbf801039a0 0x7fbf80103e20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009860 tx=0x7fbf6c02a340 comp rx=0 tx=0).stop 2024-04-26T23:58:06.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.121+0000 7fbf86800640 1 -- 172.21.15.40:0/640210221 shutdown_connections 2024-04-26T23:58:06.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.121+0000 7fbf86800640 1 --2- 172.21.15.40:0/640210221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf80104360 0x7fbf8010abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:06.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.121+0000 7fbf86800640 1 --2- 172.21.15.40:0/640210221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbf801039a0 0x7fbf80103e20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:06.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.121+0000 7fbf86800640 1 --2- 172.21.15.40:0/640210221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf801027b0 0x7fbf80102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:06.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.121+0000 7fbf86800640 1 -- 172.21.15.40:0/640210221 >> 172.21.15.40:0/640210221 conn(0x7fbf800fdd40 msgr2=0x7fbf80100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:06.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.122+0000 7fbf86800640 1 -- 172.21.15.40:0/640210221 shutdown_connections 2024-04-26T23:58:06.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.122+0000 7fbf86800640 1 -- 172.21.15.40:0/640210221 wait complete. 2024-04-26T23:58:06.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.123+0000 7fbf86800640 1 Processor -- start 2024-04-26T23:58:06.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.123+0000 7fbf86800640 1 -- start start 2024-04-26T23:58:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.123+0000 7fbf86800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbf801027b0 0x7fbf8019ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.123+0000 7fbf86800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf801039a0 0x7fbf8019b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf86800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf80104360 0x7fbf801a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf86800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8010e2c0 con 0x7fbf801039a0 2024-04-26T23:58:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf86800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8010e140 con 0x7fbf801027b0 2024-04-26T23:58:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf86800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8010e440 con 0x7fbf80104360 2024-04-26T23:58:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf84a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf801039a0 0x7fbf8019b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf85400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbf801027b0 0x7fbf8019ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf85400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbf801027b0 0x7fbf8019ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:41568/0 (socket says 172.21.15.40:41568) 2024-04-26T23:58:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf85400640 1 -- 172.21.15.40:0/444319978 learned_addr learned my addr 172.21.15.40:0/444319978 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf85e00640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf80104360 0x7fbf801a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf85400640 1 -- 172.21.15.40:0/444319978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf80104360 msgr2=0x7fbf801a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf85400640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf80104360 0x7fbf801a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf85400640 1 -- 172.21.15.40:0/444319978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf801039a0 msgr2=0x7fbf8019b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf85400640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf801039a0 0x7fbf8019b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.124+0000 7fbf85400640 1 -- 172.21.15.40:0/444319978 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf801a27c0 con 0x7fbf801027b0 2024-04-26T23:58:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.125+0000 7fbf85400640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbf801027b0 0x7fbf8019ac40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf7400e9c0 tx=0x7fbf7400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:06.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.125+0000 7fbf7e000640 1 -- 172.21.15.40:0/444319978 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7400cd80 con 0x7fbf801027b0 2024-04-26T23:58:06.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.125+0000 7fbf7e000640 1 -- 172.21.15.40:0/444319978 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf740035f0 con 0x7fbf801027b0 2024-04-26T23:58:06.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.125+0000 7fbf86800640 1 -- 172.21.15.40:0/444319978 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf801a2ab0 con 0x7fbf801027b0 2024-04-26T23:58:06.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.125+0000 7fbf7e000640 1 -- 172.21.15.40:0/444319978 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf74010690 con 0x7fbf801027b0 2024-04-26T23:58:06.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.125+0000 7fbf86800640 1 -- 172.21.15.40:0/444319978 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf801a3020 con 0x7fbf801027b0 2024-04-26T23:58:06.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.126+0000 7fbf86800640 1 -- 172.21.15.40:0/444319978 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf801a31f0 con 0x7fbf801027b0 2024-04-26T23:58:06.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.128+0000 7fbf7e000640 1 -- 172.21.15.40:0/444319978 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbf740037a0 con 0x7fbf801027b0 2024-04-26T23:58:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.130+0000 7fbf7e000640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbf680a3c30 0x7fbf680a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:06.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.130+0000 7fbf7e000640 1 -- 172.21.15.40:0/444319978 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbf740d0300 con 0x7fbf801027b0 2024-04-26T23:58:06.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.130+0000 7fbf84a00640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbf680a3c30 0x7fbf680a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:06.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.131+0000 7fbf84a00640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbf680a3c30 0x7fbf680a60f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c02a930 tx=0x7fbf6c030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:06.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.134+0000 7fbf7e000640 1 -- 172.21.15.40:0/444319978 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbf7409d0e0 con 0x7fbf801027b0 2024-04-26T23:58:06.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:05 smithi040 ceph-mon[28095]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:06.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:05 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:06.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:05 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:58:06.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:05 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:06.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:05 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:06.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:05 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:06.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:05 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:06.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:05 smithi040 ceph-mon[28095]: Deploying daemon mds.l on smithi103 2024-04-26T23:58:06.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:05 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/974395025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:06.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.386+0000 7fbf86800640 1 -- 172.21.15.40:0/444319978 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf801a33c0 con 0x7fbf801027b0 2024-04-26T23:58:06.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.387+0000 7fbf7e000640 1 -- 172.21.15.40:0/444319978 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fbf7409c9e0 con 0x7fbf801027b0 2024-04-26T23:58:06.389 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:06.389 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-26T23:58:06.390 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 1 2024-04-26T23:58:06.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.390+0000 7fbf86800640 1 -- 172.21.15.40:0/444319978 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbf680a3c30 msgr2=0x7fbf680a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.390+0000 7fbf86800640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbf680a3c30 0x7fbf680a60f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c02a930 tx=0x7fbf6c030450 comp rx=0 tx=0).stop 2024-04-26T23:58:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.390+0000 7fbf86800640 1 -- 172.21.15.40:0/444319978 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbf801027b0 msgr2=0x7fbf8019ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.390+0000 7fbf86800640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbf801027b0 0x7fbf8019ac40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf7400e9c0 tx=0x7fbf7400ee80 comp rx=0 tx=0).stop 2024-04-26T23:58:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.390+0000 7fbf86800640 1 -- 172.21.15.40:0/444319978 shutdown_connections 2024-04-26T23:58:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.390+0000 7fbf86800640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbf680a3c30 0x7fbf680a60f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.390+0000 7fbf86800640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf80104360 0x7fbf801a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.390+0000 7fbf86800640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf801039a0 0x7fbf8019b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.390+0000 7fbf86800640 1 --2- 172.21.15.40:0/444319978 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbf801027b0 0x7fbf8019ac40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.390+0000 7fbf86800640 1 -- 172.21.15.40:0/444319978 >> 172.21.15.40:0/444319978 conn(0x7fbf800fdd40 msgr2=0x7fbf80074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.390+0000 7fbf86800640 1 -- 172.21.15.40:0/444319978 shutdown_connections 2024-04-26T23:58:06.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:06.390+0000 7fbf86800640 1 -- 172.21.15.40:0/444319978 wait complete. 2024-04-26T23:58:07.056 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:06 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3768075331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:58:07.057 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:06 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/444319978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:07.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:06 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3768075331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:58:07.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:06 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/444319978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:07.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:06 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3768075331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:58:07.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:06 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/444319978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:07.348 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 26 23:58:07 smithi103 systemd[1]: Starting Ceph mds.l for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:58:07.744 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 26 23:58:07 smithi103 podman[62835]: 2024-04-26 23:58:07.349948224 +0000 UTC m=+0.017882795 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:58:07.744 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 26 23:58:07 smithi103 podman[62835]: 2024-04-26 23:58:07.465448056 +0000 UTC m=+0.133382656 container create 646bac59a43542dcc10839dba24ca592b95be584b5cc32daf80f11d852ca7aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T23:58:07.744 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 26 23:58:07 smithi103 podman[62835]: 2024-04-26 23:58:07.743256287 +0000 UTC m=+0.411190854 container init 646bac59a43542dcc10839dba24ca592b95be584b5cc32daf80f11d852ca7aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-26T23:58:08.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:07 smithi103 ceph-mon[32952]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:08.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:07 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.103:6832/970790712,v1:172.21.15.103:6833/970790712] up:boot 2024-04-26T23:58:08.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:07 smithi103 ceph-mon[32952]: fsmap 1 up:standby 2024-04-26T23:58:08.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:07 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T23:58:08.099 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 26 23:58:07 smithi103 podman[62835]: 2024-04-26 23:58:07.745983312 +0000 UTC m=+0.413917868 container start 646bac59a43542dcc10839dba24ca592b95be584b5cc32daf80f11d852ca7aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-04-26T23:58:08.099 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 26 23:58:07 smithi103 ceph-mds[62854]: -- 172.21.15.103:0/1617366522 <== mon.1 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bfd71fda40 con 0x55bfd71d0800 2024-04-26T23:58:08.099 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 26 23:58:07 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-l[62850]: starting mds.l at 2024-04-26T23:58:08.099 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 26 23:58:07 smithi103 bash[62835]: 646bac59a43542dcc10839dba24ca592b95be584b5cc32daf80f11d852ca7aed 2024-04-26T23:58:08.099 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 26 23:58:07 smithi103 systemd[1]: Started Ceph mds.l for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:58:08.201 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:07 smithi080 ceph-mon[32362]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:08.201 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:07 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.103:6832/970790712,v1:172.21.15.103:6833/970790712] up:boot 2024-04-26T23:58:08.201 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:07 smithi080 ceph-mon[32362]: fsmap 1 up:standby 2024-04-26T23:58:08.201 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:07 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T23:58:08.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:07 smithi040 ceph-mon[28095]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:08.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:07 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.103:6832/970790712,v1:172.21.15.103:6833/970790712] up:boot 2024-04-26T23:58:08.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:07 smithi040 ceph-mon[28095]: fsmap 1 up:standby 2024-04-26T23:58:08.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:07 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T23:58:08.413 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:08.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.641+0000 7fd95cc00640 1 -- 172.21.15.40:0/3528431110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9581022e0 msgr2=0x7fd958102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:08.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.641+0000 7fd95cc00640 1 --2- 172.21.15.40:0/3528431110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9581022e0 0x7fd958102760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd950009ca0 tx=0x7fd95002a8c0 comp rx=0 tx=0).stop 2024-04-26T23:58:08.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.641+0000 7fd95cc00640 1 -- 172.21.15.40:0/3528431110 shutdown_connections 2024-04-26T23:58:08.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.641+0000 7fd95cc00640 1 --2- 172.21.15.40:0/3528431110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9581022e0 0x7fd958102760 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:08.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.641+0000 7fd95cc00640 1 --2- 172.21.15.40:0/3528431110 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd958101940 0x7fd958101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:08.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.641+0000 7fd95cc00640 1 --2- 172.21.15.40:0/3528431110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd958100fe0 0x7fd958101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:08.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.641+0000 7fd95cc00640 1 -- 172.21.15.40:0/3528431110 >> 172.21.15.40:0/3528431110 conn(0x7fd9580fcbc0 msgr2=0x7fd9580ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:08.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.641+0000 7fd95cc00640 1 -- 172.21.15.40:0/3528431110 shutdown_connections 2024-04-26T23:58:08.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.641+0000 7fd95cc00640 1 -- 172.21.15.40:0/3528431110 wait complete. 2024-04-26T23:58:08.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.642+0000 7fd95cc00640 1 Processor -- start 2024-04-26T23:58:08.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.642+0000 7fd95cc00640 1 -- start start 2024-04-26T23:58:08.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd95cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd958100fe0 0x7fd95819ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd95cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd958101940 0x7fd95819b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd95cc00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd9581022e0 0x7fd9581a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd95cc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd958110490 con 0x7fd958101940 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd95cc00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fd958110310 con 0x7fd9581022e0 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd95cc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd958110610 con 0x7fd958100fe0 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd94ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd958101940 0x7fd95819b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd94ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd958101940 0x7fd95819b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33498/0 (socket says 172.21.15.40:33498) 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd94ea00640 1 -- 172.21.15.40:0/961455835 learned_addr learned my addr 172.21.15.40:0/961455835 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd957400640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd958100fe0 0x7fd95819ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd957e00640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd9581022e0 0x7fd9581a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd94ea00640 1 -- 172.21.15.40:0/961455835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd958100fe0 msgr2=0x7fd95819ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd94ea00640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd958100fe0 0x7fd95819ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd94ea00640 1 -- 172.21.15.40:0/961455835 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd9581022e0 msgr2=0x7fd9581a20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:08.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd94ea00640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd9581022e0 0x7fd9581a20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:08.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.643+0000 7fd94ea00640 1 -- 172.21.15.40:0/961455835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9581a2730 con 0x7fd958101940 2024-04-26T23:58:08.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.644+0000 7fd94ea00640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd958101940 0x7fd95819b050 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd94000e9f0 tx=0x7fd94000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:08.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.644+0000 7fd954c00640 1 -- 172.21.15.40:0/961455835 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94000cd80 con 0x7fd958101940 2024-04-26T23:58:08.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.644+0000 7fd954c00640 1 -- 172.21.15.40:0/961455835 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9400108e0 con 0x7fd958101940 2024-04-26T23:58:08.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.644+0000 7fd954c00640 1 -- 172.21.15.40:0/961455835 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940018980 con 0x7fd958101940 2024-04-26T23:58:08.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.644+0000 7fd95cc00640 1 -- 172.21.15.40:0/961455835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9581a2a20 con 0x7fd958101940 2024-04-26T23:58:08.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.644+0000 7fd95cc00640 1 -- 172.21.15.40:0/961455835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9581a2fc0 con 0x7fd958101940 2024-04-26T23:58:08.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.645+0000 7fd95cc00640 1 -- 172.21.15.40:0/961455835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd95810bee0 con 0x7fd958101940 2024-04-26T23:58:08.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.647+0000 7fd954c00640 1 -- 172.21.15.40:0/961455835 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd940010430 con 0x7fd958101940 2024-04-26T23:58:08.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.649+0000 7fd954c00640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd93c0a3c30 0x7fd93c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:08.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.649+0000 7fd954c00640 1 -- 172.21.15.40:0/961455835 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd9400d0260 con 0x7fd958101940 2024-04-26T23:58:08.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.649+0000 7fd957400640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd93c0a3c30 0x7fd93c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:08.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.650+0000 7fd957400640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd93c0a3c30 0x7fd93c0a60f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd948009bd0 tx=0x7fd948005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:08.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.653+0000 7fd954c00640 1 -- 172.21.15.40:0/961455835 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd94009c9f0 con 0x7fd958101940 2024-04-26T23:58:08.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.902+0000 7fd95cc00640 1 -- 172.21.15.40:0/961455835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd95802dad0 con 0x7fd958101940 2024-04-26T23:58:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.903+0000 7fd954c00640 1 -- 172.21.15.40:0/961455835 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7fd9400a0cf0 con 0x7fd958101940 2024-04-26T23:58:08.905 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:08.905 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T23:58:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 2 2024-04-26T23:58:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.906+0000 7fd95cc00640 1 -- 172.21.15.40:0/961455835 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd93c0a3c30 msgr2=0x7fd93c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.906+0000 7fd95cc00640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd93c0a3c30 0x7fd93c0a60f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd948009bd0 tx=0x7fd948005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.906+0000 7fd95cc00640 1 -- 172.21.15.40:0/961455835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd958101940 msgr2=0x7fd95819b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.906+0000 7fd95cc00640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd958101940 0x7fd95819b050 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd94000e9f0 tx=0x7fd94000eeb0 comp rx=0 tx=0).stop 2024-04-26T23:58:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.906+0000 7fd95cc00640 1 -- 172.21.15.40:0/961455835 shutdown_connections 2024-04-26T23:58:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.906+0000 7fd95cc00640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd9581022e0 0x7fd9581a20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.906+0000 7fd95cc00640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd958101940 0x7fd95819b050 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.906+0000 7fd95cc00640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd93c0a3c30 0x7fd93c0a60f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.906+0000 7fd95cc00640 1 --2- 172.21.15.40:0/961455835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd958100fe0 0x7fd95819ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.907+0000 7fd95cc00640 1 -- 172.21.15.40:0/961455835 >> 172.21.15.40:0/961455835 conn(0x7fd9580fcbc0 msgr2=0x7fd9580fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:08.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.907+0000 7fd95cc00640 1 -- 172.21.15.40:0/961455835 shutdown_connections 2024-04-26T23:58:08.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:08.907+0000 7fd95cc00640 1 -- 172.21.15.40:0/961455835 wait complete. 2024-04-26T23:58:09.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:08 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:09.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:08 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:09.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:08 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:09.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:08 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:09.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:08 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:09.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:08 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:09.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:08 smithi040 ceph-mon[28095]: Deploying daemon mds.k on smithi080 2024-04-26T23:58:09.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:08 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/961455835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:09.252 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:08 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:09.252 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:08 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:09.252 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:08 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:09.252 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:08 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:09.252 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:08 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:09.252 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:08 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:09.252 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:08 smithi080 ceph-mon[32362]: Deploying daemon mds.k on smithi080 2024-04-26T23:58:09.252 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:08 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/961455835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:09.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:08 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:09.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:08 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:09.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:08 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:09.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:08 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:09.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:08 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:09.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:08 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:09.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:08 smithi103 ceph-mon[32952]: Deploying daemon mds.k on smithi080 2024-04-26T23:58:09.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:08 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/961455835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:10.295 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:09 smithi080 ceph-mon[32362]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:10.295 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 26 23:58:10 smithi080 systemd[1]: Starting Ceph mds.k for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:58:10.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:09 smithi103 ceph-mon[32952]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:10.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:09 smithi040 ceph-mon[28095]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:10.550 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 26 23:58:10 smithi080 podman[62198]: 2024-04-26 23:58:10.4130435 +0000 UTC m=+0.017843828 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:58:10.550 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 26 23:58:10 smithi080 podman[62198]: 2024-04-26 23:58:10.533069702 +0000 UTC m=+0.137870029 container create bf6655a67f9f1e8ffea16a5a78106e5090e36eed1d73cfd5d354b40d7db92077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-26T23:58:10.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:11.050 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 26 23:58:10 smithi080 podman[62198]: 2024-04-26 23:58:10.785854438 +0000 UTC m=+0.390654769 container init bf6655a67f9f1e8ffea16a5a78106e5090e36eed1d73cfd5d354b40d7db92077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-26T23:58:11.050 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 26 23:58:10 smithi080 podman[62198]: 2024-04-26 23:58:10.788451109 +0000 UTC m=+0.393251447 container start bf6655a67f9f1e8ffea16a5a78106e5090e36eed1d73cfd5d354b40d7db92077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T23:58:11.050 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 26 23:58:10 smithi080 ceph-mds[62217]: -- 172.21.15.80:0/515276158 <== mon.2 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560058045a40 con 0x560058019000 2024-04-26T23:58:11.050 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 26 23:58:10 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-k[62213]: starting mds.k at 2024-04-26T23:58:11.051 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 26 23:58:10 smithi080 bash[62198]: bf6655a67f9f1e8ffea16a5a78106e5090e36eed1d73cfd5d354b40d7db92077 2024-04-26T23:58:11.051 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 26 23:58:10 smithi080 systemd[1]: Started Ceph mds.k for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:58:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.146+0000 7ff9b8e00640 1 -- 172.21.15.40:0/1274711709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9b41033e0 msgr2=0x7ff9b41057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.146+0000 7ff9b8e00640 1 --2- 172.21.15.40:0/1274711709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9b41033e0 0x7ff9b41057d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009ca0 tx=0x7ff9a402a8c0 comp rx=0 tx=0).stop 2024-04-26T23:58:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.146+0000 7ff9b8e00640 1 -- 172.21.15.40:0/1274711709 shutdown_connections 2024-04-26T23:58:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.146+0000 7ff9b8e00640 1 --2- 172.21.15.40:0/1274711709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b4108640 0x7ff9b410aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.146+0000 7ff9b8e00640 1 --2- 172.21.15.40:0/1274711709 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff9b4105d10 0x7ff9b4108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.146+0000 7ff9b8e00640 1 --2- 172.21.15.40:0/1274711709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9b41033e0 0x7ff9b41057d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.146+0000 7ff9b8e00640 1 -- 172.21.15.40:0/1274711709 >> 172.21.15.40:0/1274711709 conn(0x7ff9b40fcdf0 msgr2=0x7ff9b40ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.146+0000 7ff9b8e00640 1 -- 172.21.15.40:0/1274711709 shutdown_connections 2024-04-26T23:58:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.146+0000 7ff9b8e00640 1 -- 172.21.15.40:0/1274711709 wait complete. 2024-04-26T23:58:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.147+0000 7ff9b8e00640 1 Processor -- start 2024-04-26T23:58:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.147+0000 7ff9b8e00640 1 -- start start 2024-04-26T23:58:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9b8e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff9b41033e0 0x7ff9b419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9b8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b4105d10 0x7ff9b419b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9b8e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9b4108640 0x7ff9b41a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9b8e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b410e4e0 con 0x7ff9b4105d10 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9b8e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b410e360 con 0x7ff9b41033e0 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9b8e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b410e660 con 0x7ff9b4108640 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9a9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b4105d10 0x7ff9b419b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9b1e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff9b41033e0 0x7ff9b419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9a9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b4105d10 0x7ff9b419b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59636/0 (socket says 172.21.15.40:59636) 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9a9400640 1 -- 172.21.15.40:0/157873971 learned_addr learned my addr 172.21.15.40:0/157873971 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9b2800640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9b4108640 0x7ff9b41a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9a9400640 1 -- 172.21.15.40:0/157873971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9b4108640 msgr2=0x7ff9b41a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9a9400640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9b4108640 0x7ff9b41a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9a9400640 1 -- 172.21.15.40:0/157873971 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff9b41033e0 msgr2=0x7ff9b419ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9a9400640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff9b41033e0 0x7ff9b419ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9a9400640 1 -- 172.21.15.40:0/157873971 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b41a2a20 con 0x7ff9b4105d10 2024-04-26T23:58:11.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.148+0000 7ff9b2800640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9b4108640 0x7ff9b41a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:58:11.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.149+0000 7ff9a9400640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b4105d10 0x7ff9b419b340 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00e9c0 tx=0x7ff99c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:11.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.149+0000 7ff9ab400640 1 -- 172.21.15.40:0/157873971 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99c00cd80 con 0x7ff9b4105d10 2024-04-26T23:58:11.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.149+0000 7ff9b8e00640 1 -- 172.21.15.40:0/157873971 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b41a2d10 con 0x7ff9b4105d10 2024-04-26T23:58:11.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.149+0000 7ff9ab400640 1 -- 172.21.15.40:0/157873971 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff99c0054d0 con 0x7ff9b4105d10 2024-04-26T23:58:11.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.149+0000 7ff9ab400640 1 -- 172.21.15.40:0/157873971 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99c010430 con 0x7ff9b4105d10 2024-04-26T23:58:11.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.149+0000 7ff9b8e00640 1 -- 172.21.15.40:0/157873971 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b41a3200 con 0x7ff9b4105d10 2024-04-26T23:58:11.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.150+0000 7ff9b8e00640 1 -- 172.21.15.40:0/157873971 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9b410a190 con 0x7ff9b4105d10 2024-04-26T23:58:11.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.153+0000 7ff9ab400640 1 -- 172.21.15.40:0/157873971 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff99c004fd0 con 0x7ff9b4105d10 2024-04-26T23:58:11.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.154+0000 7ff9ab400640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff9980a3bb0 0x7ff9980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:11.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.154+0000 7ff9ab400640 1 -- 172.21.15.40:0/157873971 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff99c0d0400 con 0x7ff9b4105d10 2024-04-26T23:58:11.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.155+0000 7ff9b1e00640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff9980a3bb0 0x7ff9980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:11.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.155+0000 7ff9b1e00640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff9980a3bb0 0x7ff9980a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4005e30 tx=0x7ff9a4005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:11.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.159+0000 7ff9ab400640 1 -- 172.21.15.40:0/157873971 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff99c09d280 con 0x7ff9b4105d10 2024-04-26T23:58:11.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.413+0000 7ff9b8e00640 1 -- 172.21.15.40:0/157873971 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9b402dad0 con 0x7ff9b4105d10 2024-04-26T23:58:11.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.414+0000 7ff9ab400640 1 -- 172.21.15.40:0/157873971 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7ff99c09cb80 con 0x7ff9b4105d10 2024-04-26T23:58:11.416 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:11.416 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T23:58:11.416 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 3 2024-04-26T23:58:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.417+0000 7ff9b8e00640 1 -- 172.21.15.40:0/157873971 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff9980a3bb0 msgr2=0x7ff9980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.417+0000 7ff9b8e00640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff9980a3bb0 0x7ff9980a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4005e30 tx=0x7ff9a4005f50 comp rx=0 tx=0).stop 2024-04-26T23:58:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.417+0000 7ff9b8e00640 1 -- 172.21.15.40:0/157873971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b4105d10 msgr2=0x7ff9b419b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.417+0000 7ff9b8e00640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b4105d10 0x7ff9b419b340 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00e9c0 tx=0x7ff99c00ee80 comp rx=0 tx=0).stop 2024-04-26T23:58:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.417+0000 7ff9b8e00640 1 -- 172.21.15.40:0/157873971 shutdown_connections 2024-04-26T23:58:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.417+0000 7ff9b8e00640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff9980a3bb0 0x7ff9980a6070 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.417+0000 7ff9b8e00640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9b4108640 0x7ff9b41a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.417+0000 7ff9b8e00640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b4105d10 0x7ff9b419b340 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.417+0000 7ff9b8e00640 1 --2- 172.21.15.40:0/157873971 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff9b41033e0 0x7ff9b419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.417+0000 7ff9b8e00640 1 -- 172.21.15.40:0/157873971 >> 172.21.15.40:0/157873971 conn(0x7ff9b40fcdf0 msgr2=0x7ff9b4102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.417+0000 7ff9b8e00640 1 -- 172.21.15.40:0/157873971 shutdown_connections 2024-04-26T23:58:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:11.417+0000 7ff9b8e00640 1 -- 172.21.15.40:0/157873971 wait complete. 2024-04-26T23:58:12.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:11 smithi040 ceph-mon[28095]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:12.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:11 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:12.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:11 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:12.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:11 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:12.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:11 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:12.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:11 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:12.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:11 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:12.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:11 smithi040 ceph-mon[28095]: Deploying daemon mds.j on smithi040 2024-04-26T23:58:12.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:11 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.80:6832/3975801652,v1:172.21.15.80:6833/3975801652] up:boot 2024-04-26T23:58:12.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:11 smithi040 ceph-mon[28095]: fsmap 2 up:standby 2024-04-26T23:58:12.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:11 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T23:58:12.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:11 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/157873971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:12.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:11 smithi080 ceph-mon[32362]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:12.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:11 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:12.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:11 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:12.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:11 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:12.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:11 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:12.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:11 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:12.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:11 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:12.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:11 smithi080 ceph-mon[32362]: Deploying daemon mds.j on smithi040 2024-04-26T23:58:12.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:11 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.80:6832/3975801652,v1:172.21.15.80:6833/3975801652] up:boot 2024-04-26T23:58:12.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:11 smithi080 ceph-mon[32362]: fsmap 2 up:standby 2024-04-26T23:58:12.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:11 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T23:58:12.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:11 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/157873971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:12.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:11 smithi103 ceph-mon[32952]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:12.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:11 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:12.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:11 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:12.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:11 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:12.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:11 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:12.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:11 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:12.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:11 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:12.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:11 smithi103 ceph-mon[32952]: Deploying daemon mds.j on smithi040 2024-04-26T23:58:12.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:11 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.80:6832/3975801652,v1:172.21.15.80:6833/3975801652] up:boot 2024-04-26T23:58:12.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:11 smithi103 ceph-mon[32952]: fsmap 2 up:standby 2024-04-26T23:58:12.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:11 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T23:58:12.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:11 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/157873971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:13.439 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.804+0000 7f0b51c00640 1 -- 172.21.15.40:0/2957935795 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b4c074c40 msgr2=0x7f0b4c0776d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.804+0000 7f0b51c00640 1 --2- 172.21.15.40:0/2957935795 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b4c074c40 0x7f0b4c0776d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0b44009ca0 tx=0x7f0b4402a8d0 comp rx=0 tx=0).stop 2024-04-26T23:58:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.805+0000 7f0b51c00640 1 -- 172.21.15.40:0/2957935795 shutdown_connections 2024-04-26T23:58:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.805+0000 7f0b51c00640 1 --2- 172.21.15.40:0/2957935795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b4c07a240 0x7f0b4c07c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.805+0000 7f0b51c00640 1 --2- 172.21.15.40:0/2957935795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b4c0750f0 0x7f0b4c079c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.805+0000 7f0b51c00640 1 --2- 172.21.15.40:0/2957935795 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b4c074c40 0x7f0b4c0776d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.805+0000 7f0b51c00640 1 -- 172.21.15.40:0/2957935795 >> 172.21.15.40:0/2957935795 conn(0x7f0b4c06f7c0 msgr2=0x7f0b4c071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.805+0000 7f0b51c00640 1 -- 172.21.15.40:0/2957935795 shutdown_connections 2024-04-26T23:58:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.805+0000 7f0b51c00640 1 -- 172.21.15.40:0/2957935795 wait complete. 2024-04-26T23:58:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.806+0000 7f0b51c00640 1 Processor -- start 2024-04-26T23:58:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.806+0000 7f0b51c00640 1 -- start start 2024-04-26T23:58:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.806+0000 7f0b51c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b4c074c40 0x7f0b4c1b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.806+0000 7f0b51c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b4c0750f0 0x7f0b4c1b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.806+0000 7f0b51c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b4c07a240 0x7f0b4c1bbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.806+0000 7f0b51c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4c07fde0 con 0x7f0b4c0750f0 2024-04-26T23:58:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.806+0000 7f0b51c00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4c07fc60 con 0x7f0b4c07a240 2024-04-26T23:58:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.806+0000 7f0b51c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4c07ff60 con 0x7f0b4c074c40 2024-04-26T23:58:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.807+0000 7f0b4aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b4c074c40 0x7f0b4c1b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.807+0000 7f0b4aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b4c074c40 0x7f0b4c1b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49086/0 (socket says 172.21.15.40:49086) 2024-04-26T23:58:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.807+0000 7f0b4aa00640 1 -- 172.21.15.40:0/1840701724 learned_addr learned my addr 172.21.15.40:0/1840701724 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.807+0000 7f0b4a000640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b4c0750f0 0x7f0b4c1b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.807+0000 7f0b4b400640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b4c07a240 0x7f0b4c1bbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:13.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.807+0000 7f0b4aa00640 1 -- 172.21.15.40:0/1840701724 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b4c07a240 msgr2=0x7f0b4c1bbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:13.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.807+0000 7f0b4aa00640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b4c07a240 0x7f0b4c1bbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:13.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.807+0000 7f0b4aa00640 1 -- 172.21.15.40:0/1840701724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b4c0750f0 msgr2=0x7f0b4c1b4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:13.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.807+0000 7f0b4aa00640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b4c0750f0 0x7f0b4c1b4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:13.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.807+0000 7f0b4aa00640 1 -- 172.21.15.40:0/1840701724 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4c1bc450 con 0x7f0b4c074c40 2024-04-26T23:58:13.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.808+0000 7f0b4aa00640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b4c074c40 0x7f0b4c1b4830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b4402aec0 tx=0x7f0b44003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:13.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.808+0000 7f0b43400640 1 -- 172.21.15.40:0/1840701724 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b44004240 con 0x7f0b4c074c40 2024-04-26T23:58:13.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.808+0000 7f0b51c00640 1 -- 172.21.15.40:0/1840701724 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4c1bc6e0 con 0x7f0b4c074c40 2024-04-26T23:58:13.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.808+0000 7f0b43400640 1 -- 172.21.15.40:0/1840701724 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b44003e10 con 0x7f0b4c074c40 2024-04-26T23:58:13.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.808+0000 7f0b43400640 1 -- 172.21.15.40:0/1840701724 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4402fd60 con 0x7f0b4c074c40 2024-04-26T23:58:13.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.808+0000 7f0b51c00640 1 -- 172.21.15.40:0/1840701724 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b4c1bcc80 con 0x7f0b4c074c40 2024-04-26T23:58:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.810+0000 7f0b51c00640 1 -- 172.21.15.40:0/1840701724 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b4c0835e0 con 0x7f0b4c074c40 2024-04-26T23:58:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.812+0000 7f0b43400640 1 -- 172.21.15.40:0/1840701724 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0b440043e0 con 0x7f0b4c074c40 2024-04-26T23:58:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.813+0000 7f0b43400640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0b300a3ba0 0x7f0b300a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.813+0000 7f0b43400640 1 -- 172.21.15.40:0/1840701724 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0b440f1720 con 0x7f0b4c074c40 2024-04-26T23:58:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.813+0000 7f0b4a000640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0b300a3ba0 0x7f0b300a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.814+0000 7f0b4a000640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0b300a3ba0 0x7f0b300a6060 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c1b5d40 tx=0x7f0b38009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:13.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:13.817+0000 7f0b43400640 1 -- 172.21.15.40:0/1840701724 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0b440be500 con 0x7f0b4c074c40 2024-04-26T23:58:14.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.078+0000 7f0b51c00640 1 -- 172.21.15.40:0/1840701724 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b4c063530 con 0x7f0b4c074c40 2024-04-26T23:58:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.079+0000 7f0b43400640 1 -- 172.21.15.40:0/1840701724 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f0b440bde00 con 0x7f0b4c074c40 2024-04-26T23:58:14.081 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:14.081 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T23:58:14.081 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 3 2024-04-26T23:58:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.081+0000 7f0b51c00640 1 -- 172.21.15.40:0/1840701724 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0b300a3ba0 msgr2=0x7f0b300a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.081+0000 7f0b51c00640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0b300a3ba0 0x7f0b300a6060 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c1b5d40 tx=0x7f0b38009110 comp rx=0 tx=0).stop 2024-04-26T23:58:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.081+0000 7f0b51c00640 1 -- 172.21.15.40:0/1840701724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b4c074c40 msgr2=0x7f0b4c1b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.081+0000 7f0b51c00640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b4c074c40 0x7f0b4c1b4830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b4402aec0 tx=0x7f0b44003c20 comp rx=0 tx=0).stop 2024-04-26T23:58:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.081+0000 7f0b51c00640 1 -- 172.21.15.40:0/1840701724 shutdown_connections 2024-04-26T23:58:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.081+0000 7f0b51c00640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b4c07a240 0x7f0b4c1bbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.081+0000 7f0b51c00640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b4c0750f0 0x7f0b4c1b4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.081+0000 7f0b51c00640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0b300a3ba0 0x7f0b300a6060 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.081+0000 7f0b51c00640 1 --2- 172.21.15.40:0/1840701724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b4c074c40 0x7f0b4c1b4830 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.081+0000 7f0b51c00640 1 -- 172.21.15.40:0/1840701724 >> 172.21.15.40:0/1840701724 conn(0x7f0b4c06f7c0 msgr2=0x7f0b4c0807b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.081+0000 7f0b51c00640 1 -- 172.21.15.40:0/1840701724 shutdown_connections 2024-04-26T23:58:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:14.082+0000 7f0b51c00640 1 -- 172.21.15.40:0/1840701724 wait complete. 2024-04-26T23:58:14.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:13 smithi040 ceph-mon[28095]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:14.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:13 smithi080 ceph-mon[32362]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:14.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:13 smithi103 ceph-mon[32952]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:14.692 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 26 23:58:14 smithi040 podman[66123]: 2024-04-26 23:58:14.678303988 +0000 UTC m=+0.383922126 container init 9261d7109e4aa434376070eae0cd36bf828d9ceda3d89ca863be57389340157e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-26T23:58:14.692 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 26 23:58:14 smithi040 podman[66123]: 2024-04-26 23:58:14.690065742 +0000 UTC m=+0.395683899 container start 9261d7109e4aa434376070eae0cd36bf828d9ceda3d89ca863be57389340157e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:58:14.952 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 26 23:58:14 smithi040 ceph-mds[66141]: -- 172.21.15.40:0/1762385886 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d668c3da40 con 0x55d668c10c00 2024-04-26T23:58:14.952 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 26 23:58:14 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-j[66137]: starting mds.j at 2024-04-26T23:58:14.953 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 26 23:58:14 smithi040 bash[66123]: 9261d7109e4aa434376070eae0cd36bf828d9ceda3d89ca863be57389340157e 2024-04-26T23:58:14.953 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 26 23:58:14 smithi040 systemd[1]: Started Ceph mds.j for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:58:15.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:14 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1840701724' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:15.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:15.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:15.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:14 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1840701724' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:15.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:15.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:15.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:14 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1840701724' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:15.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:15.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:16.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:16.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:15 smithi040 ceph-mon[28095]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:16.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:15 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.40:6834/1384127498,v1:172.21.15.40:6835/1384127498] up:boot 2024-04-26T23:58:16.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:15 smithi040 ceph-mon[28095]: fsmap 3 up:standby 2024-04-26T23:58:16.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:15 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T23:58:16.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:15 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:16.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:15 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:16.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:15 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:16.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:15 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:16.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:15 smithi040 ceph-mon[28095]: Deploying daemon mds.d on smithi040 2024-04-26T23:58:16.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:15 smithi080 ceph-mon[32362]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:16.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:15 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.40:6834/1384127498,v1:172.21.15.40:6835/1384127498] up:boot 2024-04-26T23:58:16.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:15 smithi080 ceph-mon[32362]: fsmap 3 up:standby 2024-04-26T23:58:16.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:15 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T23:58:16.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:15 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:16.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:15 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:16.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:15 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:16.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:15 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:16.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:15 smithi080 ceph-mon[32362]: Deploying daemon mds.d on smithi040 2024-04-26T23:58:16.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.329+0000 7f41eb200640 1 -- 172.21.15.40:0/2071986859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec1031d0 msgr2=0x7f41ec1055c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:16.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.329+0000 7f41eb200640 1 --2- 172.21.15.40:0/2071986859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec1031d0 0x7f41ec1055c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009900 tx=0x7f41dc02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:58:16.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.329+0000 7f41eb200640 1 -- 172.21.15.40:0/2071986859 shutdown_connections 2024-04-26T23:58:16.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.329+0000 7f41eb200640 1 --2- 172.21.15.40:0/2071986859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41ec108430 0x7f41ec10a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:16.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.329+0000 7f41eb200640 1 --2- 172.21.15.40:0/2071986859 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f41ec105b00 0x7f41ec107ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:16.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.329+0000 7f41eb200640 1 --2- 172.21.15.40:0/2071986859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec1031d0 0x7f41ec1055c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:16.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.329+0000 7f41eb200640 1 -- 172.21.15.40:0/2071986859 >> 172.21.15.40:0/2071986859 conn(0x7f41ec0fcbe0 msgr2=0x7f41ec0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:16.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.329+0000 7f41eb200640 1 -- 172.21.15.40:0/2071986859 shutdown_connections 2024-04-26T23:58:16.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.329+0000 7f41eb200640 1 -- 172.21.15.40:0/2071986859 wait complete. 2024-04-26T23:58:16.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.330+0000 7f41eb200640 1 Processor -- start 2024-04-26T23:58:16.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.330+0000 7f41eb200640 1 -- start start 2024-04-26T23:58:16.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.330+0000 7f41eb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41ec1031d0 0x7f41ec19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:16.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.330+0000 7f41eb200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f41ec105b00 0x7f41ec19b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:16.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.330+0000 7f41eb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec108430 0x7f41ec1a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:16.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.330+0000 7f41eb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec101720 con 0x7f41ec1031d0 2024-04-26T23:58:16.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.330+0000 7f41eb200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec1015a0 con 0x7f41ec105b00 2024-04-26T23:58:16.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.330+0000 7f41eb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec1018a0 con 0x7f41ec108430 2024-04-26T23:58:16.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.331+0000 7f41e9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41ec1031d0 0x7f41ec19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:16.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.331+0000 7f41e9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41ec1031d0 0x7f41ec19ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59680/0 (socket says 172.21.15.40:59680) 2024-04-26T23:58:16.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.331+0000 7f41e9e00640 1 -- 172.21.15.40:0/1861677922 learned_addr learned my addr 172.21.15.40:0/1861677922 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:16.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.331+0000 7f41ea800640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec108430 0x7f41ec1a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:16.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.331+0000 7f41e3e00640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f41ec105b00 0x7f41ec19b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:16.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.331+0000 7f41e9e00640 1 -- 172.21.15.40:0/1861677922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec108430 msgr2=0x7f41ec1a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:16.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.331+0000 7f41e9e00640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec108430 0x7f41ec1a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:16.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.331+0000 7f41e9e00640 1 -- 172.21.15.40:0/1861677922 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f41ec105b00 msgr2=0x7f41ec19b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:16.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.331+0000 7f41e9e00640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f41ec105b00 0x7f41ec19b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:16.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.331+0000 7f41e9e00640 1 -- 172.21.15.40:0/1861677922 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec1a27a0 con 0x7f41ec1031d0 2024-04-26T23:58:16.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.331+0000 7f41ea800640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec108430 0x7f41ec1a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:16.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.332+0000 7f41e9e00640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41ec1031d0 0x7f41ec19ab80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f41dc005ab0 tx=0x7f41dc003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:16.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.332+0000 7f41e2a00640 1 -- 172.21.15.40:0/1861677922 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc03f070 con 0x7f41ec1031d0 2024-04-26T23:58:16.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.332+0000 7f41eb200640 1 -- 172.21.15.40:0/1861677922 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ec1a2a30 con 0x7f41ec1031d0 2024-04-26T23:58:16.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.332+0000 7f41e2a00640 1 -- 172.21.15.40:0/1861677922 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41dc031c50 con 0x7f41ec1031d0 2024-04-26T23:58:16.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.332+0000 7f41eb200640 1 -- 172.21.15.40:0/1861677922 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ec1a2f70 con 0x7f41ec1031d0 2024-04-26T23:58:16.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.332+0000 7f41e2a00640 1 -- 172.21.15.40:0/1861677922 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc049060 con 0x7f41ec1031d0 2024-04-26T23:58:16.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.334+0000 7f41eb200640 1 -- 172.21.15.40:0/1861677922 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41ec1a31a0 con 0x7f41ec1031d0 2024-04-26T23:58:16.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.335+0000 7f41e2a00640 1 -- 172.21.15.40:0/1861677922 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f41dc042070 con 0x7f41ec1031d0 2024-04-26T23:58:16.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.337+0000 7f41e2a00640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f41d00a3bb0 0x7f41d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:16.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.337+0000 7f41e2a00640 1 -- 172.21.15.40:0/1861677922 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f41dc0f0a50 con 0x7f41ec1031d0 2024-04-26T23:58:16.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.337+0000 7f41e3e00640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f41d00a3bb0 0x7f41d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:16.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.338+0000 7f41e3e00640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f41d00a3bb0 0x7f41d00a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f41ec19c090 tx=0x7f41d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:16.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.342+0000 7f41e2a00640 1 -- 172.21.15.40:0/1861677922 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f41dc0bd1e0 con 0x7f41ec1031d0 2024-04-26T23:58:16.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:15 smithi103 ceph-mon[32952]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:16.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:15 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.40:6834/1384127498,v1:172.21.15.40:6835/1384127498] up:boot 2024-04-26T23:58:16.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:15 smithi103 ceph-mon[32952]: fsmap 3 up:standby 2024-04-26T23:58:16.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:15 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T23:58:16.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:15 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:16.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:15 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:16.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:15 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:16.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:15 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:16.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:15 smithi103 ceph-mon[32952]: Deploying daemon mds.d on smithi040 2024-04-26T23:58:16.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.607+0000 7f41eb200640 1 -- 172.21.15.40:0/1861677922 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41ec1a3370 con 0x7f41ec1031d0 2024-04-26T23:58:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.608+0000 7f41e2a00640 1 -- 172.21.15.40:0/1861677922 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f41dc0c14e0 con 0x7f41ec1031d0 2024-04-26T23:58:16.610 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:16.610 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T23:58:16.610 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 4 2024-04-26T23:58:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.611+0000 7f41eb200640 1 -- 172.21.15.40:0/1861677922 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f41d00a3bb0 msgr2=0x7f41d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.611+0000 7f41eb200640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f41d00a3bb0 0x7f41d00a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f41ec19c090 tx=0x7f41d4005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.611+0000 7f41eb200640 1 -- 172.21.15.40:0/1861677922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41ec1031d0 msgr2=0x7f41ec19ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:16.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.611+0000 7f41eb200640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41ec1031d0 0x7f41ec19ab80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f41dc005ab0 tx=0x7f41dc003b90 comp rx=0 tx=0).stop 2024-04-26T23:58:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.611+0000 7f41eb200640 1 -- 172.21.15.40:0/1861677922 shutdown_connections 2024-04-26T23:58:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.611+0000 7f41eb200640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f41d00a3bb0 0x7f41d00a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.611+0000 7f41eb200640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec108430 0x7f41ec1a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.611+0000 7f41eb200640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f41ec105b00 0x7f41ec19b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:16.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.611+0000 7f41eb200640 1 --2- 172.21.15.40:0/1861677922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41ec1031d0 0x7f41ec19ab80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:16.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.611+0000 7f41eb200640 1 -- 172.21.15.40:0/1861677922 >> 172.21.15.40:0/1861677922 conn(0x7f41ec0fcbe0 msgr2=0x7f41ec1071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:16.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.611+0000 7f41eb200640 1 -- 172.21.15.40:0/1861677922 shutdown_connections 2024-04-26T23:58:16.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:16.611+0000 7f41eb200640 1 -- 172.21.15.40:0/1861677922 wait complete. 2024-04-26T23:58:17.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:16 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1861677922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:17.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:16 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1861677922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:17.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:16 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1861677922' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:18.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:17 smithi040 ceph-mon[28095]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:18.218 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 26 23:58:18 smithi040 systemd[1]: Starting Ceph mds.d for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:58:18.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:17 smithi080 ceph-mon[32362]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:18.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:17 smithi103 ceph-mon[32952]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:18.573 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 26 23:58:18 smithi040 podman[66465]: 2024-04-26 23:58:18.291789705 +0000 UTC m=+0.036226452 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:58:18.573 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 26 23:58:18 smithi040 podman[66465]: 2024-04-26 23:58:18.412764271 +0000 UTC m=+0.157201046 container create 885a4a0934399dea542a5acda7365c7a6cc528e0aba4ca0ea0c36e181ac9a70c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:58:18.636 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:18.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.858+0000 7ffb1ac00640 1 -- 172.21.15.40:0/62196552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb14104b90 msgr2=0x7ffb14104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:18.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.858+0000 7ffb1ac00640 1 --2- 172.21.15.40:0/62196552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb14104b90 0x7ffb14104fb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009860 tx=0x7ffb0402a340 comp rx=0 tx=0).stop 2024-04-26T23:58:18.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.858+0000 7ffb1ac00640 1 -- 172.21.15.40:0/62196552 shutdown_connections 2024-04-26T23:58:18.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.858+0000 7ffb1ac00640 1 --2- 172.21.15.40:0/62196552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb14106740 0x7ffb1410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:18.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.858+0000 7ffb1ac00640 1 --2- 172.21.15.40:0/62196552 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ffb14105d80 0x7ffb14106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:18.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.858+0000 7ffb1ac00640 1 --2- 172.21.15.40:0/62196552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb14104b90 0x7ffb14104fb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:18.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.858+0000 7ffb1ac00640 1 -- 172.21.15.40:0/62196552 >> 172.21.15.40:0/62196552 conn(0x7ffb14100120 msgr2=0x7ffb14102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:18.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.858+0000 7ffb1ac00640 1 -- 172.21.15.40:0/62196552 shutdown_connections 2024-04-26T23:58:18.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.858+0000 7ffb1ac00640 1 -- 172.21.15.40:0/62196552 wait complete. 2024-04-26T23:58:18.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.859+0000 7ffb1ac00640 1 Processor -- start 2024-04-26T23:58:18.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.859+0000 7ffb1ac00640 1 -- start start 2024-04-26T23:58:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.860+0000 7ffb1ac00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ffb14104b90 0x7ffb1419adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:18.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.860+0000 7ffb1ac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb14105d80 0x7ffb1419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.860+0000 7ffb1ac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb14106740 0x7ffb141a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.860+0000 7ffb1ac00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffb14110690 con 0x7ffb14106740 2024-04-26T23:58:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.860+0000 7ffb1ac00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7ffb14110510 con 0x7ffb14104b90 2024-04-26T23:58:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.860+0000 7ffb1ac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffb14110810 con 0x7ffb14105d80 2024-04-26T23:58:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.860+0000 7ffb18c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb14106740 0x7ffb141a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.860+0000 7ffb13e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ffb14104b90 0x7ffb1419adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.860+0000 7ffb18c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb14106740 0x7ffb141a2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59698/0 (socket says 172.21.15.40:59698) 2024-04-26T23:58:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.860+0000 7ffb18c00640 1 -- 172.21.15.40:0/1960507067 learned_addr learned my addr 172.21.15.40:0/1960507067 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.860+0000 7ffb0be00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb14105d80 0x7ffb1419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.861+0000 7ffb13e00640 1 -- 172.21.15.40:0/1960507067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb14105d80 msgr2=0x7ffb1419b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.861+0000 7ffb13e00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb14105d80 0x7ffb1419b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:18.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.861+0000 7ffb13e00640 1 -- 172.21.15.40:0/1960507067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb14106740 msgr2=0x7ffb141a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:18.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.861+0000 7ffb13e00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb14106740 0x7ffb141a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:18.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.861+0000 7ffb13e00640 1 -- 172.21.15.40:0/1960507067 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb141a29e0 con 0x7ffb14104b90 2024-04-26T23:58:18.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.861+0000 7ffb18c00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb14106740 0x7ffb141a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:18.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.861+0000 7ffb13e00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ffb14104b90 0x7ffb1419adc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb0402a860 tx=0x7ffb04032dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:18.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.862+0000 7ffb11600640 1 -- 172.21.15.40:0/1960507067 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04037040 con 0x7ffb14104b90 2024-04-26T23:58:18.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.862+0000 7ffb11600640 1 -- 172.21.15.40:0/1960507067 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb04030bd0 con 0x7ffb14104b90 2024-04-26T23:58:18.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.862+0000 7ffb11600640 1 -- 172.21.15.40:0/1960507067 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04003850 con 0x7ffb14104b90 2024-04-26T23:58:18.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.862+0000 7ffb1ac00640 1 -- 172.21.15.40:0/1960507067 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb141a2c70 con 0x7ffb14104b90 2024-04-26T23:58:18.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.862+0000 7ffb1ac00640 1 -- 172.21.15.40:0/1960507067 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb141a30b0 con 0x7ffb14104b90 2024-04-26T23:58:18.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.863+0000 7ffb1ac00640 1 -- 172.21.15.40:0/1960507067 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb1410bf30 con 0x7ffb14104b90 2024-04-26T23:58:18.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.865+0000 7ffb11600640 1 -- 172.21.15.40:0/1960507067 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffb04043020 con 0x7ffb14104b90 2024-04-26T23:58:18.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.867+0000 7ffb11600640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ffaf80a3c30 0x7ffaf80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:18.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.867+0000 7ffb11600640 1 -- 172.21.15.40:0/1960507067 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ffb040ef480 con 0x7ffb14104b90 2024-04-26T23:58:18.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.867+0000 7ffb0be00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ffaf80a3c30 0x7ffaf80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:18.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.868+0000 7ffb0be00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ffaf80a3c30 0x7ffaf80a60f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffb1419c2d0 tx=0x7ffafc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:18.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:18.869+0000 7ffb11600640 1 -- 172.21.15.40:0/1960507067 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffb0404a060 con 0x7ffb14104b90 2024-04-26T23:58:18.968 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 26 23:58:18 smithi040 podman[66465]: 2024-04-26 23:58:18.682281039 +0000 UTC m=+0.426717799 container init 885a4a0934399dea542a5acda7365c7a6cc528e0aba4ca0ea0c36e181ac9a70c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:58:18.968 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 26 23:58:18 smithi040 podman[66465]: 2024-04-26 23:58:18.69508114 +0000 UTC m=+0.439517888 container start 885a4a0934399dea542a5acda7365c7a6cc528e0aba4ca0ea0c36e181ac9a70c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:58:18.968 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 26 23:58:18 smithi040 ceph-mds[66500]: -- 172.21.15.40:0/2474297539 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c39f7fba40 con 0x55c39f7ce800 2024-04-26T23:58:18.968 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 26 23:58:18 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-d[66480]: starting mds.d at 2024-04-26T23:58:18.969 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 26 23:58:18 smithi040 bash[66465]: 885a4a0934399dea542a5acda7365c7a6cc528e0aba4ca0ea0c36e181ac9a70c 2024-04-26T23:58:18.969 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 26 23:58:18 smithi040 systemd[1]: Started Ceph mds.d for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:58:19.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.122+0000 7ffb1ac00640 1 -- 172.21.15.40:0/1960507067 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb1402dc90 con 0x7ffb14104b90 2024-04-26T23:58:19.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.123+0000 7ffb11600640 1 -- 172.21.15.40:0/1960507067 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7ffb040bc360 con 0x7ffb14104b90 2024-04-26T23:58:19.125 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:19.125 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-26T23:58:19.125 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 5 2024-04-26T23:58:19.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.126+0000 7ffb1ac00640 1 -- 172.21.15.40:0/1960507067 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ffaf80a3c30 msgr2=0x7ffaf80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:19.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.126+0000 7ffb1ac00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ffaf80a3c30 0x7ffaf80a60f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffb1419c2d0 tx=0x7ffafc009260 comp rx=0 tx=0).stop 2024-04-26T23:58:19.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.126+0000 7ffb1ac00640 1 -- 172.21.15.40:0/1960507067 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ffb14104b90 msgr2=0x7ffb1419adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:19.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.126+0000 7ffb1ac00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ffb14104b90 0x7ffb1419adc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb0402a860 tx=0x7ffb04032dc0 comp rx=0 tx=0).stop 2024-04-26T23:58:19.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.126+0000 7ffb1ac00640 1 -- 172.21.15.40:0/1960507067 shutdown_connections 2024-04-26T23:58:19.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.126+0000 7ffb1ac00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb14106740 0x7ffb141a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:19.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.126+0000 7ffb1ac00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ffaf80a3c30 0x7ffaf80a60f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:19.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.126+0000 7ffb1ac00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb14105d80 0x7ffb1419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:19.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.126+0000 7ffb1ac00640 1 --2- 172.21.15.40:0/1960507067 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ffb14104b90 0x7ffb1419adc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:19.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.126+0000 7ffb1ac00640 1 -- 172.21.15.40:0/1960507067 >> 172.21.15.40:0/1960507067 conn(0x7ffb14100120 msgr2=0x7ffb14109100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:19.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.126+0000 7ffb1ac00640 1 -- 172.21.15.40:0/1960507067 shutdown_connections 2024-04-26T23:58:19.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:19.126+0000 7ffb1ac00640 1 -- 172.21.15.40:0/1960507067 wait complete. 2024-04-26T23:58:20.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:19 smithi040 ceph-mon[28095]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:19 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.40:6836/3497909120,v1:172.21.15.40:6837/3497909120] up:boot 2024-04-26T23:58:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:19 smithi040 ceph-mon[28095]: fsmap 4 up:standby 2024-04-26T23:58:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T23:58:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:19 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:19 smithi040 ceph-mon[28095]: Deploying daemon mds.e on smithi080 2024-04-26T23:58:20.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:19 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1960507067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:19 smithi080 ceph-mon[32362]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:20.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:19 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.40:6836/3497909120,v1:172.21.15.40:6837/3497909120] up:boot 2024-04-26T23:58:20.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:19 smithi080 ceph-mon[32362]: fsmap 4 up:standby 2024-04-26T23:58:20.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T23:58:20.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:20.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:20.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:20.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:19 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:20.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:19 smithi080 ceph-mon[32362]: Deploying daemon mds.e on smithi080 2024-04-26T23:58:20.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:19 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1960507067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:20.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:19 smithi103 ceph-mon[32952]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:20.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:20.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:20.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:19 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.40:6836/3497909120,v1:172.21.15.40:6837/3497909120] up:boot 2024-04-26T23:58:20.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:19 smithi103 ceph-mon[32952]: fsmap 4 up:standby 2024-04-26T23:58:20.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T23:58:20.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:20.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:20.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:20.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:19 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:20.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:19 smithi103 ceph-mon[32952]: Deploying daemon mds.e on smithi080 2024-04-26T23:58:20.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:19 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1960507067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:21.148 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:21.300 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 26 23:58:21 smithi080 systemd[1]: Starting Ceph mds.e for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:58:21.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.367+0000 7f250be00640 1 -- 172.21.15.40:0/2274746263 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f250c106540 msgr2=0x7f250c10cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.367+0000 7f250be00640 1 --2- 172.21.15.40:0/2274746263 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f250c106540 0x7f250c10cdd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2504009860 tx=0x7f250402a340 comp rx=0 tx=0).stop 2024-04-26T23:58:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.367+0000 7f250be00640 1 -- 172.21.15.40:0/2274746263 shutdown_connections 2024-04-26T23:58:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.367+0000 7f250be00640 1 --2- 172.21.15.40:0/2274746263 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f250c106540 0x7f250c10cdd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.367+0000 7f250be00640 1 --2- 172.21.15.40:0/2274746263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c105b80 0x7f250c106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.367+0000 7f250be00640 1 --2- 172.21.15.40:0/2274746263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250c104990 0x7f250c104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.367+0000 7f250be00640 1 -- 172.21.15.40:0/2274746263 >> 172.21.15.40:0/2274746263 conn(0x7f250c0fff00 msgr2=0x7f250c102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.367+0000 7f250be00640 1 -- 172.21.15.40:0/2274746263 shutdown_connections 2024-04-26T23:58:21.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.368+0000 7f250be00640 1 -- 172.21.15.40:0/2274746263 wait complete. 2024-04-26T23:58:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.369+0000 7f250be00640 1 Processor -- start 2024-04-26T23:58:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.369+0000 7f250be00640 1 -- start start 2024-04-26T23:58:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.369+0000 7f250be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f250c104990 0x7f250c19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.369+0000 7f250be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c105b80 0x7f250c19b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.369+0000 7f250be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250c106540 0x7f250c1a20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.369+0000 7f250be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f250c1104a0 con 0x7f250c106540 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.369+0000 7f250be00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f250c110320 con 0x7f250c104990 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.369+0000 7f250be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f250c110620 con 0x7f250c105b80 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.370+0000 7f250b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250c106540 0x7f250c1a20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.370+0000 7f250b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250c106540 0x7f250c1a20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41466/0 (socket says 172.21.15.40:41466) 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.370+0000 7f250b400640 1 -- 172.21.15.40:0/3027972503 learned_addr learned my addr 172.21.15.40:0/3027972503 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.370+0000 7f250a000640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c105b80 0x7f250c19b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.370+0000 7f250aa00640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f250c104990 0x7f250c19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.370+0000 7f250b400640 1 -- 172.21.15.40:0/3027972503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c105b80 msgr2=0x7f250c19b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.370+0000 7f250b400640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c105b80 0x7f250c19b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.370+0000 7f250b400640 1 -- 172.21.15.40:0/3027972503 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f250c104990 msgr2=0x7f250c19aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.370+0000 7f250b400640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f250c104990 0x7f250c19aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.370+0000 7f250b400640 1 -- 172.21.15.40:0/3027972503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f250c1a2710 con 0x7f250c106540 2024-04-26T23:58:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.371+0000 7f250b400640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250c106540 0x7f250c1a20a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f25040057c0 tx=0x7f2504032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.371+0000 7f2503400640 1 -- 172.21.15.40:0/3027972503 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2504040020 con 0x7f250c106540 2024-04-26T23:58:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.371+0000 7f250be00640 1 -- 172.21.15.40:0/3027972503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f250c1a29a0 con 0x7f250c106540 2024-04-26T23:58:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.371+0000 7f2503400640 1 -- 172.21.15.40:0/3027972503 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2504004040 con 0x7f250c106540 2024-04-26T23:58:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.371+0000 7f2503400640 1 -- 172.21.15.40:0/3027972503 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2504037040 con 0x7f250c106540 2024-04-26T23:58:21.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.371+0000 7f250be00640 1 -- 172.21.15.40:0/3027972503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f250c1a2e90 con 0x7f250c106540 2024-04-26T23:58:21.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.372+0000 7f250be00640 1 -- 172.21.15.40:0/3027972503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f250c10bc50 con 0x7f250c106540 2024-04-26T23:58:21.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.374+0000 7f2503400640 1 -- 172.21.15.40:0/3027972503 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2504043020 con 0x7f250c106540 2024-04-26T23:58:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.375+0000 7f2503400640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f24f00a3bb0 0x7f24f00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:21.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.376+0000 7f2503400640 1 -- 172.21.15.40:0/3027972503 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f25040efef0 con 0x7f250c106540 2024-04-26T23:58:21.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.376+0000 7f250aa00640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f24f00a3bb0 0x7f24f00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:21.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.377+0000 7f250aa00640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f24f00a3bb0 0x7f24f00a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f250c1059e0 tx=0x7f24fc010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.380+0000 7f2503400640 1 -- 172.21.15.40:0/3027972503 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f250404a020 con 0x7f250c106540 2024-04-26T23:58:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.624+0000 7f250be00640 1 -- 172.21.15.40:0/3027972503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f250c02dad0 con 0x7f250c106540 2024-04-26T23:58:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.625+0000 7f2503400640 1 -- 172.21.15.40:0/3027972503 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f25040bccd0 con 0x7f250c106540 2024-04-26T23:58:21.627 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:21.627 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-26T23:58:21.627 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 5 2024-04-26T23:58:21.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.628+0000 7f250be00640 1 -- 172.21.15.40:0/3027972503 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f24f00a3bb0 msgr2=0x7f24f00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:21.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.628+0000 7f250be00640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f24f00a3bb0 0x7f24f00a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f250c1059e0 tx=0x7f24fc010040 comp rx=0 tx=0).stop 2024-04-26T23:58:21.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.628+0000 7f250be00640 1 -- 172.21.15.40:0/3027972503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250c106540 msgr2=0x7f250c1a20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:21.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.628+0000 7f250be00640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250c106540 0x7f250c1a20a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f25040057c0 tx=0x7f2504032eb0 comp rx=0 tx=0).stop 2024-04-26T23:58:21.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.628+0000 7f250be00640 1 -- 172.21.15.40:0/3027972503 shutdown_connections 2024-04-26T23:58:21.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.628+0000 7f250be00640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f250c106540 0x7f250c1a20a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:21.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.628+0000 7f250be00640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f24f00a3bb0 0x7f24f00a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:21.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.628+0000 7f250be00640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f250c105b80 0x7f250c19b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:21.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.628+0000 7f250be00640 1 --2- 172.21.15.40:0/3027972503 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f250c104990 0x7f250c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:21.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.628+0000 7f250be00640 1 -- 172.21.15.40:0/3027972503 >> 172.21.15.40:0/3027972503 conn(0x7f250c0fff00 msgr2=0x7f250c108e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:21.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.628+0000 7f250be00640 1 -- 172.21.15.40:0/3027972503 shutdown_connections 2024-04-26T23:58:21.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:21.628+0000 7f250be00640 1 -- 172.21.15.40:0/3027972503 wait complete. 2024-04-26T23:58:21.659 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 26 23:58:21 smithi080 podman[62504]: 2024-04-26 23:58:21.266416055 +0000 UTC m=+0.017892485 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:58:21.659 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 26 23:58:21 smithi080 podman[62504]: 2024-04-26 23:58:21.397184584 +0000 UTC m=+0.148661016 container create 908d3b1b27f62ce0f95f6b2d869147527a92190da66ad9da56692f24dd6756bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:58:21.659 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 26 23:58:21 smithi080 podman[62504]: 2024-04-26 23:58:21.658289388 +0000 UTC m=+0.409765818 container init 908d3b1b27f62ce0f95f6b2d869147527a92190da66ad9da56692f24dd6756bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-26T23:58:21.924 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 26 23:58:21 smithi080 podman[62504]: 2024-04-26 23:58:21.661061351 +0000 UTC m=+0.412537787 container start 908d3b1b27f62ce0f95f6b2d869147527a92190da66ad9da56692f24dd6756bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-26T23:58:21.925 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 26 23:58:21 smithi080 ceph-mds[62523]: -- 172.21.15.80:0/2383319736 <== mon.2 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5632dd67ba40 con 0x5632dd64ec00 2024-04-26T23:58:21.925 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 26 23:58:21 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-e[62519]: starting mds.e at 2024-04-26T23:58:21.925 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 26 23:58:21 smithi080 bash[62504]: 908d3b1b27f62ce0f95f6b2d869147527a92190da66ad9da56692f24dd6756bc 2024-04-26T23:58:21.925 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 26 23:58:21 smithi080 systemd[1]: Started Ceph mds.e for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:58:22.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:21 smithi040 ceph-mon[28095]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:22.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:21 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3027972503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:22.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:21 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:22.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:21 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:22.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:21 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:22.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:21 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:22.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:21 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:22.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:21 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:22.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:21 smithi080 ceph-mon[32362]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:22.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:21 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3027972503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:22.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:21 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:22.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:21 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:22.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:21 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:22.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:21 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:22.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:21 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:22.301 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:21 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:22.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:21 smithi103 ceph-mon[32952]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:22.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:21 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3027972503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:22.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:21 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:22.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:21 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:22.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:21 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:22.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:21 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:22.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:21 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:22.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:21 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:23.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:22 smithi103 ceph-mon[32952]: Deploying daemon mds.f on smithi103 2024-04-26T23:58:23.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:22 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.80:6834/1031714719,v1:172.21.15.80:6835/1031714719] up:boot 2024-04-26T23:58:23.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:22 smithi103 ceph-mon[32952]: fsmap 5 up:standby 2024-04-26T23:58:23.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:22 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T23:58:23.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:22 smithi040 ceph-mon[28095]: Deploying daemon mds.f on smithi103 2024-04-26T23:58:23.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:22 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.80:6834/1031714719,v1:172.21.15.80:6835/1031714719] up:boot 2024-04-26T23:58:23.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:22 smithi040 ceph-mon[28095]: fsmap 5 up:standby 2024-04-26T23:58:23.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:22 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T23:58:23.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:22 smithi080 ceph-mon[32362]: Deploying daemon mds.f on smithi103 2024-04-26T23:58:23.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:22 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.80:6834/1031714719,v1:172.21.15.80:6835/1031714719] up:boot 2024-04-26T23:58:23.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:22 smithi080 ceph-mon[32362]: fsmap 5 up:standby 2024-04-26T23:58:23.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:22 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T23:58:23.649 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:23.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.863+0000 7fa88a400640 1 -- 172.21.15.40:0/609984873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8841031d0 msgr2=0x7fa8841055c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.863+0000 7fa88a400640 1 --2- 172.21.15.40:0/609984873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8841031d0 0x7fa8841055c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa878009900 tx=0x7fa87802a3f0 comp rx=0 tx=0).stop 2024-04-26T23:58:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.863+0000 7fa88a400640 1 -- 172.21.15.40:0/609984873 shutdown_connections 2024-04-26T23:58:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.863+0000 7fa88a400640 1 --2- 172.21.15.40:0/609984873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa884108430 0x7fa88410a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.863+0000 7fa88a400640 1 --2- 172.21.15.40:0/609984873 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa884105b00 0x7fa884107ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.863+0000 7fa88a400640 1 --2- 172.21.15.40:0/609984873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8841031d0 0x7fa8841055c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.863+0000 7fa88a400640 1 -- 172.21.15.40:0/609984873 >> 172.21.15.40:0/609984873 conn(0x7fa8840fcbe0 msgr2=0x7fa8840ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.863+0000 7fa88a400640 1 -- 172.21.15.40:0/609984873 shutdown_connections 2024-04-26T23:58:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.864+0000 7fa88a400640 1 -- 172.21.15.40:0/609984873 wait complete. 2024-04-26T23:58:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.865+0000 7fa88a400640 1 Processor -- start 2024-04-26T23:58:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.865+0000 7fa88a400640 1 -- start start 2024-04-26T23:58:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa88a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8841031d0 0x7fa88419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa88a400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa884105b00 0x7fa88419b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa88a400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa884108430 0x7fa8841a2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa88a400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa88410e2d0 con 0x7fa8841031d0 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa88a400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa88410e150 con 0x7fa884105b00 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa88a400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa88410e450 con 0x7fa884108430 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa889000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8841031d0 0x7fa88419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa889000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8841031d0 0x7fa88419abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41480/0 (socket says 172.21.15.40:41480) 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa889000640 1 -- 172.21.15.40:0/3740178008 learned_addr learned my addr 172.21.15.40:0/3740178008 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa883e00640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa884105b00 0x7fa88419b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa889a00640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa884108430 0x7fa8841a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa889000640 1 -- 172.21.15.40:0/3740178008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa884108430 msgr2=0x7fa8841a2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa889000640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa884108430 0x7fa8841a2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa889000640 1 -- 172.21.15.40:0/3740178008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa884105b00 msgr2=0x7fa88419b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:23.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa889000640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa884105b00 0x7fa88419b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa889000640 1 -- 172.21.15.40:0/3740178008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8841a27f0 con 0x7fa8841031d0 2024-04-26T23:58:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.866+0000 7fa883e00640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa884105b00 0x7fa88419b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.867+0000 7fa889000640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8841031d0 0x7fa88419abd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa878005ab0 tx=0x7fa878003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:23.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.867+0000 7fa881600640 1 -- 172.21.15.40:0/3740178008 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87803f070 con 0x7fa8841031d0 2024-04-26T23:58:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.867+0000 7fa881600640 1 -- 172.21.15.40:0/3740178008 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa878031c50 con 0x7fa8841031d0 2024-04-26T23:58:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.867+0000 7fa88a400640 1 -- 172.21.15.40:0/3740178008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8841a2a80 con 0x7fa8841031d0 2024-04-26T23:58:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.867+0000 7fa881600640 1 -- 172.21.15.40:0/3740178008 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878035030 con 0x7fa8841031d0 2024-04-26T23:58:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.867+0000 7fa88a400640 1 -- 172.21.15.40:0/3740178008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8841a2fc0 con 0x7fa8841031d0 2024-04-26T23:58:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.869+0000 7fa88a400640 1 -- 172.21.15.40:0/3740178008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa884109f80 con 0x7fa8841031d0 2024-04-26T23:58:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.871+0000 7fa881600640 1 -- 172.21.15.40:0/3740178008 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa878042070 con 0x7fa8841031d0 2024-04-26T23:58:23.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.872+0000 7fa881600640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa86c0a3c30 0x7fa86c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:23.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.872+0000 7fa881600640 1 -- 172.21.15.40:0/3740178008 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa8780f0650 con 0x7fa8841031d0 2024-04-26T23:58:23.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.872+0000 7fa883e00640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa86c0a3c30 0x7fa86c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:23.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.873+0000 7fa883e00640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa86c0a3c30 0x7fa86c0a60f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa88419c0e0 tx=0x7fa870005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:23.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:23.876+0000 7fa881600640 1 -- 172.21.15.40:0/3740178008 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa8780bd430 con 0x7fa8841031d0 2024-04-26T23:58:24.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:23 smithi103 ceph-mon[32952]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:24.099 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 26 23:58:23 smithi103 systemd[1]: Starting Ceph mds.f for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:58:24.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.124+0000 7fa88a400640 1 -- 172.21.15.40:0/3740178008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa88402dad0 con 0x7fa8841031d0 2024-04-26T23:58:24.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.126+0000 7fa881600640 1 -- 172.21.15.40:0/3740178008 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7fa8780bcd30 con 0x7fa8841031d0 2024-04-26T23:58:24.128 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:24.128 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T23:58:24.128 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 6 2024-04-26T23:58:24.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.128+0000 7fa88a400640 1 -- 172.21.15.40:0/3740178008 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa86c0a3c30 msgr2=0x7fa86c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:24.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.128+0000 7fa88a400640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa86c0a3c30 0x7fa86c0a60f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa88419c0e0 tx=0x7fa870005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:24.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.128+0000 7fa88a400640 1 -- 172.21.15.40:0/3740178008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8841031d0 msgr2=0x7fa88419abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:24.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.128+0000 7fa88a400640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8841031d0 0x7fa88419abd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa878005ab0 tx=0x7fa878003b90 comp rx=0 tx=0).stop 2024-04-26T23:58:24.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.128+0000 7fa88a400640 1 -- 172.21.15.40:0/3740178008 shutdown_connections 2024-04-26T23:58:24.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.128+0000 7fa88a400640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa86c0a3c30 0x7fa86c0a60f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:24.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.128+0000 7fa88a400640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa884108430 0x7fa8841a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:24.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.129+0000 7fa88a400640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa884105b00 0x7fa88419b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:24.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.129+0000 7fa88a400640 1 --2- 172.21.15.40:0/3740178008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8841031d0 0x7fa88419abd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:24.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.129+0000 7fa88a400640 1 -- 172.21.15.40:0/3740178008 >> 172.21.15.40:0/3740178008 conn(0x7fa8840fcbe0 msgr2=0x7fa884102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:24.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.129+0000 7fa88a400640 1 -- 172.21.15.40:0/3740178008 shutdown_connections 2024-04-26T23:58:24.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:24.129+0000 7fa88a400640 1 -- 172.21.15.40:0/3740178008 wait complete. 2024-04-26T23:58:24.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:23 smithi040 ceph-mon[28095]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:24.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:23 smithi080 ceph-mon[32362]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:24.468 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 26 23:58:24 smithi103 podman[63147]: 2024-04-26 23:58:24.116102246 +0000 UTC m=+0.018604958 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:58:24.468 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 26 23:58:24 smithi103 podman[63147]: 2024-04-26 23:58:24.213096694 +0000 UTC m=+0.115599396 container create 642aacff1d54ce4ff7f4a3671bd0faaebd4f73891e59b69b9354e149fabc53bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-26T23:58:24.468 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 26 23:58:24 smithi103 podman[63147]: 2024-04-26 23:58:24.441060897 +0000 UTC m=+0.343563598 container init 642aacff1d54ce4ff7f4a3671bd0faaebd4f73891e59b69b9354e149fabc53bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, ceph=True) 2024-04-26T23:58:24.468 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 26 23:58:24 smithi103 podman[63147]: 2024-04-26 23:58:24.443727253 +0000 UTC m=+0.346229966 container start 642aacff1d54ce4ff7f4a3671bd0faaebd4f73891e59b69b9354e149fabc53bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-26T23:58:24.848 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 26 23:58:24 smithi103 ceph-mds[63166]: -- 172.21.15.103:0/285945884 <== mon.1 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55feee50fa40 con 0x55feee4e2800 2024-04-26T23:58:24.848 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 26 23:58:24 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-f[63162]: starting mds.f at 2024-04-26T23:58:24.848 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 26 23:58:24 smithi103 bash[63147]: 642aacff1d54ce4ff7f4a3671bd0faaebd4f73891e59b69b9354e149fabc53bd 2024-04-26T23:58:24.848 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 26 23:58:24 smithi103 systemd[1]: Started Ceph mds.f for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:58:25.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:24 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3740178008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:25.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:25.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:25.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:25.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:25.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:25.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:24 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:25.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:24 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3740178008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:25.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:25.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:25.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:25.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:25.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:25.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:24 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:25.325 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:24 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3740178008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:25.326 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:25.326 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:25.326 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:25.326 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:25.326 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:25.326 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:24 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:26.150 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:26.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:25 smithi040 ceph-mon[28095]: Deploying daemon mds.c on smithi103 2024-04-26T23:58:26.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:25 smithi040 ceph-mon[28095]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:26.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:25 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.103:6834/2596993377,v1:172.21.15.103:6835/2596993377] up:boot 2024-04-26T23:58:26.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:25 smithi040 ceph-mon[28095]: fsmap 6 up:standby 2024-04-26T23:58:26.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:25 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T23:58:26.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:25 smithi080 ceph-mon[32362]: Deploying daemon mds.c on smithi103 2024-04-26T23:58:26.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:25 smithi080 ceph-mon[32362]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:26.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:25 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.103:6834/2596993377,v1:172.21.15.103:6835/2596993377] up:boot 2024-04-26T23:58:26.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:25 smithi080 ceph-mon[32362]: fsmap 6 up:standby 2024-04-26T23:58:26.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:25 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T23:58:26.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:25 smithi103 ceph-mon[32952]: Deploying daemon mds.c on smithi103 2024-04-26T23:58:26.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:25 smithi103 ceph-mon[32952]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:26.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:25 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.103:6834/2596993377,v1:172.21.15.103:6835/2596993377] up:boot 2024-04-26T23:58:26.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:25 smithi103 ceph-mon[32952]: fsmap 6 up:standby 2024-04-26T23:58:26.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:25 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T23:58:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.364+0000 7fea52e00640 1 -- 172.21.15.40:0/1730425971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea4c103b60 msgr2=0x7fea4c103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.364+0000 7fea52e00640 1 --2- 172.21.15.40:0/1730425971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea4c103b60 0x7fea4c103fe0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fea38009900 tx=0x7fea3802a3f0 comp rx=0 tx=0).stop 2024-04-26T23:58:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.364+0000 7fea52e00640 1 -- 172.21.15.40:0/1730425971 shutdown_connections 2024-04-26T23:58:26.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.364+0000 7fea52e00640 1 --2- 172.21.15.40:0/1730425971 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea4c104520 0x7fea4c10adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:26.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.364+0000 7fea52e00640 1 --2- 172.21.15.40:0/1730425971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea4c103b60 0x7fea4c103fe0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:26.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.364+0000 7fea52e00640 1 --2- 172.21.15.40:0/1730425971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4c102970 0x7fea4c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:26.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.364+0000 7fea52e00640 1 -- 172.21.15.40:0/1730425971 >> 172.21.15.40:0/1730425971 conn(0x7fea4c0fdee0 msgr2=0x7fea4c100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:26.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.364+0000 7fea52e00640 1 -- 172.21.15.40:0/1730425971 shutdown_connections 2024-04-26T23:58:26.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.364+0000 7fea52e00640 1 -- 172.21.15.40:0/1730425971 wait complete. 2024-04-26T23:58:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.365+0000 7fea52e00640 1 Processor -- start 2024-04-26T23:58:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.365+0000 7fea52e00640 1 -- start start 2024-04-26T23:58:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea52e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea4c102970 0x7fea4c198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea52e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea4c103b60 0x7fea4c199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea52e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4c104520 0x7fea4c1a0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea52e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c10e480 con 0x7fea4c102970 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea52e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c10e300 con 0x7fea4c103b60 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea52e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c10e600 con 0x7fea4c104520 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea4be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea4c102970 0x7fea4c198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea50e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4c104520 0x7fea4c1a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea4be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea4c102970 0x7fea4c198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41508/0 (socket says 172.21.15.40:41508) 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea4be00640 1 -- 172.21.15.40:0/1136532187 learned_addr learned my addr 172.21.15.40:0/1136532187 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea4b400640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea4c103b60 0x7fea4c199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea50e00640 1 -- 172.21.15.40:0/1136532187 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea4c103b60 msgr2=0x7fea4c199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.366+0000 7fea50e00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea4c103b60 0x7fea4c199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.367+0000 7fea50e00640 1 -- 172.21.15.40:0/1136532187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea4c102970 msgr2=0x7fea4c198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.367+0000 7fea50e00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea4c102970 0x7fea4c198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.367+0000 7fea50e00640 1 -- 172.21.15.40:0/1136532187 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4c1a0770 con 0x7fea4c104520 2024-04-26T23:58:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.367+0000 7fea4be00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea4c102970 0x7fea4c198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.367+0000 7fea50e00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4c104520 0x7fea4c1a0100 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fea44009fd0 tx=0x7fea4400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.367+0000 7fea48c00640 1 -- 172.21.15.40:0/1136532187 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4400ce20 con 0x7fea4c104520 2024-04-26T23:58:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.367+0000 7fea48c00640 1 -- 172.21.15.40:0/1136532187 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea44005020 con 0x7fea4c104520 2024-04-26T23:58:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.367+0000 7fea52e00640 1 -- 172.21.15.40:0/1136532187 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4c1a0a60 con 0x7fea4c104520 2024-04-26T23:58:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.367+0000 7fea48c00640 1 -- 172.21.15.40:0/1136532187 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea44010690 con 0x7fea4c104520 2024-04-26T23:58:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.367+0000 7fea52e00640 1 -- 172.21.15.40:0/1136532187 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea4c1a0f60 con 0x7fea4c104520 2024-04-26T23:58:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.368+0000 7fea52e00640 1 -- 172.21.15.40:0/1136532187 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea4c109d10 con 0x7fea4c104520 2024-04-26T23:58:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.370+0000 7fea48c00640 1 -- 172.21.15.40:0/1136532187 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fea440054d0 con 0x7fea4c104520 2024-04-26T23:58:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.372+0000 7fea48c00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fea300a3bb0 0x7fea300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.372+0000 7fea48c00640 1 -- 172.21.15.40:0/1136532187 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fea440d0770 con 0x7fea4c104520 2024-04-26T23:58:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.372+0000 7fea4be00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fea300a3bb0 0x7fea300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.373+0000 7fea4be00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fea300a3bb0 0x7fea300a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fea3c0058c0 tx=0x7fea3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:26.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.376+0000 7fea48c00640 1 -- 172.21.15.40:0/1136532187 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fea4409d550 con 0x7fea4c104520 2024-04-26T23:58:26.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.624+0000 7fea52e00640 1 -- 172.21.15.40:0/1136532187 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea4c02dad0 con 0x7fea4c104520 2024-04-26T23:58:26.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.625+0000 7fea48c00640 1 -- 172.21.15.40:0/1136532187 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7fea4409ce50 con 0x7fea4c104520 2024-04-26T23:58:26.627 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:26.627 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-26T23:58:26.627 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 7 2024-04-26T23:58:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.628+0000 7fea52e00640 1 -- 172.21.15.40:0/1136532187 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fea300a3bb0 msgr2=0x7fea300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.628+0000 7fea52e00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fea300a3bb0 0x7fea300a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fea3c0058c0 tx=0x7fea3c005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:26.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.628+0000 7fea52e00640 1 -- 172.21.15.40:0/1136532187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4c104520 msgr2=0x7fea4c1a0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.628+0000 7fea52e00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4c104520 0x7fea4c1a0100 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fea44009fd0 tx=0x7fea4400ef10 comp rx=0 tx=0).stop 2024-04-26T23:58:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.628+0000 7fea52e00640 1 -- 172.21.15.40:0/1136532187 shutdown_connections 2024-04-26T23:58:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.628+0000 7fea52e00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fea300a3bb0 0x7fea300a6070 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.628+0000 7fea52e00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4c104520 0x7fea4c1a0100 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.628+0000 7fea52e00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea4c103b60 0x7fea4c199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.628+0000 7fea52e00640 1 --2- 172.21.15.40:0/1136532187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea4c102970 0x7fea4c198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.628+0000 7fea52e00640 1 -- 172.21.15.40:0/1136532187 >> 172.21.15.40:0/1136532187 conn(0x7fea4c0fdee0 msgr2=0x7fea4c106ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.628+0000 7fea52e00640 1 -- 172.21.15.40:0/1136532187 shutdown_connections 2024-04-26T23:58:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:26.628+0000 7fea52e00640 1 -- 172.21.15.40:0/1136532187 wait complete. 2024-04-26T23:58:27.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:26 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1136532187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:27.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:26 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1136532187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:27.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:26 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1136532187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:27.848 INFO:journalctl@ceph.mds.c.smithi103.stdout:Apr 26 23:58:27 smithi103 systemd[1]: Started Ceph mds.c for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:58:28.650 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:28.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:28 smithi080 ceph-mon[32362]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:28.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:28.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:28.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:28.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:28.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:28.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:28.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:28 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.103:6836/2206925949,v1:172.21.15.103:6837/2206925949] up:boot 2024-04-26T23:58:28.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:28 smithi080 ceph-mon[32362]: fsmap 7 up:standby 2024-04-26T23:58:28.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:28 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T23:58:28.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:28 smithi103 ceph-mon[32952]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:28.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:28.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:28.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:28.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:28.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:28.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:28.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:28 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.103:6836/2206925949,v1:172.21.15.103:6837/2206925949] up:boot 2024-04-26T23:58:28.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:28 smithi103 ceph-mon[32952]: fsmap 7 up:standby 2024-04-26T23:58:28.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:28 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T23:58:28.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.871+0000 7f396f400640 1 -- 172.21.15.40:0/1969362511 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3968104b70 msgr2=0x7f3968104f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:28.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.871+0000 7f396f400640 1 --2- 172.21.15.40:0/1969362511 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3968104b70 0x7f3968104f90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f395c009900 tx=0x7f395c02a410 comp rx=0 tx=0).stop 2024-04-26T23:58:28.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.871+0000 7f396f400640 1 -- 172.21.15.40:0/1969362511 shutdown_connections 2024-04-26T23:58:28.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.871+0000 7f396f400640 1 --2- 172.21.15.40:0/1969362511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3968106720 0x7f396810cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:28.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.871+0000 7f396f400640 1 --2- 172.21.15.40:0/1969362511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3968105d60 0x7f39681061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:28.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.871+0000 7f396f400640 1 --2- 172.21.15.40:0/1969362511 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3968104b70 0x7f3968104f90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:28.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.871+0000 7f396f400640 1 -- 172.21.15.40:0/1969362511 >> 172.21.15.40:0/1969362511 conn(0x7f3968100120 msgr2=0x7f3968102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.871+0000 7f396f400640 1 -- 172.21.15.40:0/1969362511 shutdown_connections 2024-04-26T23:58:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.871+0000 7f396f400640 1 -- 172.21.15.40:0/1969362511 wait complete. 2024-04-26T23:58:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.872+0000 7f396f400640 1 Processor -- start 2024-04-26T23:58:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f396f400640 1 -- start start 2024-04-26T23:58:28.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f396f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3968104b70 0x7f396819adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f396f400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3968105d60 0x7f396819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f396f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3968106720 0x7f39681a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f396f400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f39681106c0 con 0x7f3968104b70 2024-04-26T23:58:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f396f400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f3968110540 con 0x7f3968105d60 2024-04-26T23:58:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f396f400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3968110840 con 0x7f3968106720 2024-04-26T23:58:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f3967e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3968105d60 0x7f396819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f396d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3968106720 0x7f39681a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f396d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3968106720 0x7f39681a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:57854/0 (socket says 172.21.15.40:57854) 2024-04-26T23:58:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f396d400640 1 -- 172.21.15.40:0/807794662 learned_addr learned my addr 172.21.15.40:0/807794662 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f3967e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3968105d60 0x7f396819b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49644/0 (socket says 172.21.15.40:49644) 2024-04-26T23:58:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.873+0000 7f396ca00640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3968104b70 0x7f396819adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.874+0000 7f396d400640 1 -- 172.21.15.40:0/807794662 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3968105d60 msgr2=0x7f396819b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.874+0000 7f396d400640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3968105d60 0x7f396819b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.874+0000 7f396d400640 1 -- 172.21.15.40:0/807794662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3968104b70 msgr2=0x7f396819adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.874+0000 7f396d400640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3968104b70 0x7f396819adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.874+0000 7f396d400640 1 -- 172.21.15.40:0/807794662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39681a2a10 con 0x7f3968106720 2024-04-26T23:58:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.874+0000 7f3967e00640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3968105d60 0x7f396819b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.874+0000 7f396d400640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3968106720 0x7f39681a23a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3960009fd0 tx=0x7f396000b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.874+0000 7f3965600640 1 -- 172.21.15.40:0/807794662 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960017070 con 0x7f3968106720 2024-04-26T23:58:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.875+0000 7f3965600640 1 -- 172.21.15.40:0/807794662 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3960003e20 con 0x7f3968106720 2024-04-26T23:58:28.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.875+0000 7f396f400640 1 -- 172.21.15.40:0/807794662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39681a2d00 con 0x7f3968106720 2024-04-26T23:58:28.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.875+0000 7f3965600640 1 -- 172.21.15.40:0/807794662 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960010460 con 0x7f3968106720 2024-04-26T23:58:28.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.875+0000 7f396f400640 1 -- 172.21.15.40:0/807794662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39681a31a0 con 0x7f3968106720 2024-04-26T23:58:28.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.876+0000 7f396f400640 1 -- 172.21.15.40:0/807794662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396810bf00 con 0x7f3968106720 2024-04-26T23:58:28.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.878+0000 7f3965600640 1 -- 172.21.15.40:0/807794662 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3960003970 con 0x7f3968106720 2024-04-26T23:58:28.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.880+0000 7f3965600640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f39500a3bb0 0x7f39500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:28.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.880+0000 7f3965600640 1 -- 172.21.15.40:0/807794662 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f39600cfd00 con 0x7f3968106720 2024-04-26T23:58:28.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.880+0000 7f396ca00640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f39500a3bb0 0x7f39500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:28.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.881+0000 7f396ca00640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f39500a3bb0 0x7f39500a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f395c005ab0 tx=0x7f395c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:28.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:28.888+0000 7f3965600640 1 -- 172.21.15.40:0/807794662 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f396009c490 con 0x7f3968106720 2024-04-26T23:58:28.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:28 smithi040 ceph-mon[28095]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:28.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:28.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:28.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:28.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:28.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:28.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:28.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:28 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.103:6836/2206925949,v1:172.21.15.103:6837/2206925949] up:boot 2024-04-26T23:58:28.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:28 smithi040 ceph-mon[28095]: fsmap 7 up:standby 2024-04-26T23:58:28.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:28 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T23:58:29.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.137+0000 7f396f400640 1 -- 172.21.15.40:0/807794662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f396802dad0 con 0x7f3968106720 2024-04-26T23:58:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.138+0000 7f3965600640 1 -- 172.21.15.40:0/807794662 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f39600a0790 con 0x7f3968106720 2024-04-26T23:58:29.140 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:29.140 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T23:58:29.140 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 8 2024-04-26T23:58:29.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.140+0000 7f396f400640 1 -- 172.21.15.40:0/807794662 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f39500a3bb0 msgr2=0x7f39500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:29.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.140+0000 7f396f400640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f39500a3bb0 0x7f39500a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f395c005ab0 tx=0x7f395c005a20 comp rx=0 tx=0).stop 2024-04-26T23:58:29.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.140+0000 7f396f400640 1 -- 172.21.15.40:0/807794662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3968106720 msgr2=0x7f39681a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:29.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.140+0000 7f396f400640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3968106720 0x7f39681a23a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3960009fd0 tx=0x7f396000b260 comp rx=0 tx=0).stop 2024-04-26T23:58:29.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.140+0000 7f396f400640 1 -- 172.21.15.40:0/807794662 shutdown_connections 2024-04-26T23:58:29.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.140+0000 7f396f400640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f39500a3bb0 0x7f39500a6070 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:29.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.140+0000 7f396f400640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3968106720 0x7f39681a23a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:29.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.140+0000 7f396f400640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3968105d60 0x7f396819b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:29.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.140+0000 7f396f400640 1 --2- 172.21.15.40:0/807794662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3968104b70 0x7f396819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:29.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.140+0000 7f396f400640 1 -- 172.21.15.40:0/807794662 >> 172.21.15.40:0/807794662 conn(0x7f3968100120 msgr2=0x7f39681090d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:29.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.141+0000 7f396f400640 1 -- 172.21.15.40:0/807794662 shutdown_connections 2024-04-26T23:58:29.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:29.141+0000 7f396f400640 1 -- 172.21.15.40:0/807794662 wait complete. 2024-04-26T23:58:29.673 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:29 smithi080 ceph-mon[32362]: Deploying daemon mds.h on smithi080 2024-04-26T23:58:29.673 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:29 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/807794662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:29.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:29 smithi103 ceph-mon[32952]: Deploying daemon mds.h on smithi080 2024-04-26T23:58:29.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:29 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/807794662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:29.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:29 smithi040 ceph-mon[28095]: Deploying daemon mds.h on smithi080 2024-04-26T23:58:29.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:29 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/807794662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:30.050 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 26 23:58:29 smithi080 systemd[1]: Starting Ceph mds.h for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:58:30.382 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 26 23:58:30 smithi080 podman[62813]: 2024-04-26 23:58:29.972027711 +0000 UTC m=+0.018029355 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:58:30.382 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 26 23:58:30 smithi080 podman[62813]: 2024-04-26 23:58:30.094862584 +0000 UTC m=+0.140864221 container create 19f051a4dcd9d84cbd4584cec0f3bba323db2f778bd2de9dbf001de18705e7f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-26T23:58:30.382 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 26 23:58:30 smithi080 podman[62813]: 2024-04-26 23:58:30.381135196 +0000 UTC m=+0.427136830 container init 19f051a4dcd9d84cbd4584cec0f3bba323db2f778bd2de9dbf001de18705e7f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T23:58:30.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:30 smithi080 ceph-mon[32362]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:30.800 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 26 23:58:30 smithi080 podman[62813]: 2024-04-26 23:58:30.383734772 +0000 UTC m=+0.429736409 container start 19f051a4dcd9d84cbd4584cec0f3bba323db2f778bd2de9dbf001de18705e7f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-26T23:58:30.800 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 26 23:58:30 smithi080 ceph-mds[62832]: -- 172.21.15.80:0/2526327923 <== mon.2 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56060fabfa40 con 0x56060fa93000 2024-04-26T23:58:30.800 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 26 23:58:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-h[62828]: starting mds.h at 2024-04-26T23:58:30.800 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 26 23:58:30 smithi080 bash[62813]: 19f051a4dcd9d84cbd4584cec0f3bba323db2f778bd2de9dbf001de18705e7f3 2024-04-26T23:58:30.800 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 26 23:58:30 smithi080 systemd[1]: Started Ceph mds.h for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:58:30.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:30 smithi103 ceph-mon[32952]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:30.944 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:30 smithi040 ceph-mon[28095]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:31.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:31.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.386+0000 7f4fcf000640 1 -- 172.21.15.40:0/2645422345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc8101060 msgr2=0x7f4fc8101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:31.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.386+0000 7f4fcf000640 1 --2- 172.21.15.40:0/2645422345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc8101060 0x7f4fc8101480 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009900 tx=0x7f4fb802a480 comp rx=0 tx=0).stop 2024-04-26T23:58:31.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.386+0000 7f4fcf000640 1 -- 172.21.15.40:0/2645422345 shutdown_connections 2024-04-26T23:58:31.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.386+0000 7f4fcf000640 1 --2- 172.21.15.40:0/2645422345 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fc8102360 0x7f4fc81027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:31.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.386+0000 7f4fcf000640 1 --2- 172.21.15.40:0/2645422345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fc81019c0 0x7f4fc8101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:31.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.386+0000 7f4fcf000640 1 --2- 172.21.15.40:0/2645422345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc8101060 0x7f4fc8101480 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:31.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.386+0000 7f4fcf000640 1 -- 172.21.15.40:0/2645422345 >> 172.21.15.40:0/2645422345 conn(0x7f4fc80fcbc0 msgr2=0x7f4fc80ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:31.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.386+0000 7f4fcf000640 1 -- 172.21.15.40:0/2645422345 shutdown_connections 2024-04-26T23:58:31.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.386+0000 7f4fcf000640 1 -- 172.21.15.40:0/2645422345 wait complete. 2024-04-26T23:58:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.387+0000 7f4fcf000640 1 Processor -- start 2024-04-26T23:58:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.387+0000 7f4fcf000640 1 -- start start 2024-04-26T23:58:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcf000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc8101060 0x7f4fc819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcf000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fc81019c0 0x7f4fc819b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcf000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fc8102360 0x7f4fc81a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcf000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc81104c0 con 0x7f4fc81019c0 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcf000640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc8110340 con 0x7f4fc8102360 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcf000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc8110640 con 0x7f4fc8101060 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcdc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc8101060 0x7f4fc819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcdc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc8101060 0x7f4fc819ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43390/0 (socket says 172.21.15.40:43390) 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcdc00640 1 -- 172.21.15.40:0/1770503006 learned_addr learned my addr 172.21.15.40:0/1770503006 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcd200640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fc81019c0 0x7f4fc819b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fce600640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fc8102360 0x7f4fc81a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcdc00640 1 -- 172.21.15.40:0/1770503006 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fc8102360 msgr2=0x7f4fc81a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcdc00640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fc8102360 0x7f4fc81a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcdc00640 1 -- 172.21.15.40:0/1770503006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fc81019c0 msgr2=0x7f4fc819b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.388+0000 7f4fcdc00640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fc81019c0 0x7f4fc819b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.389+0000 7f4fcdc00640 1 -- 172.21.15.40:0/1770503006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc81a27b0 con 0x7f4fc8101060 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.389+0000 7f4fcd200640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fc81019c0 0x7f4fc819b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.389+0000 7f4fcdc00640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc8101060 0x7f4fc819ab90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8005ab0 tx=0x7f4fb8003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.389+0000 7f4fc6000640 1 -- 172.21.15.40:0/1770503006 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb803f070 con 0x7f4fc8101060 2024-04-26T23:58:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.389+0000 7f4fcf000640 1 -- 172.21.15.40:0/1770503006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc81a2a40 con 0x7f4fc8101060 2024-04-26T23:58:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.389+0000 7f4fc6000640 1 -- 172.21.15.40:0/1770503006 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fb8031c50 con 0x7f4fc8101060 2024-04-26T23:58:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.389+0000 7f4fc6000640 1 -- 172.21.15.40:0/1770503006 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8035030 con 0x7f4fc8101060 2024-04-26T23:58:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.389+0000 7f4fcf000640 1 -- 172.21.15.40:0/1770503006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc81a2f80 con 0x7f4fc8101060 2024-04-26T23:58:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.390+0000 7f4fcf000640 1 -- 172.21.15.40:0/1770503006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc810bdb0 con 0x7f4fc8101060 2024-04-26T23:58:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.392+0000 7f4fc6000640 1 -- 172.21.15.40:0/1770503006 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4fb8042070 con 0x7f4fc8101060 2024-04-26T23:58:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.394+0000 7f4fc6000640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4fac0a3c30 0x7f4fac0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.394+0000 7f4fc6000640 1 -- 172.21.15.40:0/1770503006 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4fb80f0640 con 0x7f4fc8101060 2024-04-26T23:58:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.394+0000 7f4fcd200640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4fac0a3c30 0x7f4fac0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.395+0000 7f4fcd200640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4fac0a3c30 0x7f4fac0a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4fc819c0a0 tx=0x7f4fb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.398+0000 7f4fc6000640 1 -- 172.21.15.40:0/1770503006 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4fb80bd420 con 0x7f4fc8101060 2024-04-26T23:58:31.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.649+0000 7f4fcf000640 1 -- 172.21.15.40:0/1770503006 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fc802dad0 con 0x7f4fc8101060 2024-04-26T23:58:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.650+0000 7f4fc6000640 1 -- 172.21.15.40:0/1770503006 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7f4fb80bcd20 con 0x7f4fc8101060 2024-04-26T23:58:31.652 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:31.652 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T23:58:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 9 2024-04-26T23:58:31.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.653+0000 7f4fcf000640 1 -- 172.21.15.40:0/1770503006 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4fac0a3c30 msgr2=0x7f4fac0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:31.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.653+0000 7f4fcf000640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4fac0a3c30 0x7f4fac0a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4fc819c0a0 tx=0x7f4fb4005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:31.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.653+0000 7f4fcf000640 1 -- 172.21.15.40:0/1770503006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc8101060 msgr2=0x7f4fc819ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:31.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.653+0000 7f4fcf000640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc8101060 0x7f4fc819ab90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8005ab0 tx=0x7f4fb8003b90 comp rx=0 tx=0).stop 2024-04-26T23:58:31.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.653+0000 7f4fcf000640 1 -- 172.21.15.40:0/1770503006 shutdown_connections 2024-04-26T23:58:31.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.653+0000 7f4fcf000640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4fc8102360 0x7f4fc81a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:31.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.653+0000 7f4fcf000640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fc81019c0 0x7f4fc819b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:31.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.653+0000 7f4fcf000640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4fac0a3c30 0x7f4fac0a60f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:31.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.653+0000 7f4fcf000640 1 --2- 172.21.15.40:0/1770503006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc8101060 0x7f4fc819ab90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:31.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.653+0000 7f4fcf000640 1 -- 172.21.15.40:0/1770503006 >> 172.21.15.40:0/1770503006 conn(0x7f4fc80fcbc0 msgr2=0x7f4fc80fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:31.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.653+0000 7f4fcf000640 1 -- 172.21.15.40:0/1770503006 shutdown_connections 2024-04-26T23:58:31.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:31.653+0000 7f4fcf000640 1 -- 172.21.15.40:0/1770503006 wait complete. 2024-04-26T23:58:31.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:31 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.80:6836/4116866934,v1:172.21.15.80:6837/4116866934] up:boot 2024-04-26T23:58:31.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:31 smithi080 ceph-mon[32362]: fsmap 8 up:standby 2024-04-26T23:58:31.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T23:58:31.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:31.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:31.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:31.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:31.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:31.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:31 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:31.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:31 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.80:6836/4116866934,v1:172.21.15.80:6837/4116866934] up:boot 2024-04-26T23:58:31.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:31 smithi103 ceph-mon[32952]: fsmap 8 up:standby 2024-04-26T23:58:31.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T23:58:31.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:31.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:31.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:31.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:31.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:31.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:31 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:31.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:31 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.80:6836/4116866934,v1:172.21.15.80:6837/4116866934] up:boot 2024-04-26T23:58:31.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:31 smithi040 ceph-mon[28095]: fsmap 8 up:standby 2024-04-26T23:58:31.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T23:58:31.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:31.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:31.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:31.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:31.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:31.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:31 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:32.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:32 smithi103 ceph-mon[32952]: Deploying daemon mds.i on smithi103 2024-04-26T23:58:32.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:32 smithi103 ceph-mon[32952]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:32.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:32 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1770503006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:32.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:32 smithi080 ceph-mon[32362]: Deploying daemon mds.i on smithi103 2024-04-26T23:58:32.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:32 smithi080 ceph-mon[32362]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:32.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:32 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1770503006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:32.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:32 smithi040 ceph-mon[28095]: Deploying daemon mds.i on smithi103 2024-04-26T23:58:32.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:32 smithi040 ceph-mon[28095]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:32.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:32 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1770503006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:33.093 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 26 23:58:32 smithi103 systemd[1]: Starting Ceph mds.i for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:58:33.348 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 26 23:58:33 smithi103 podman[63771]: 2024-04-26 23:58:32.994391138 +0000 UTC m=+0.019009473 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:58:33.348 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 26 23:58:33 smithi103 podman[63771]: 2024-04-26 23:58:33.093937446 +0000 UTC m=+0.118555781 container create ee1cb2f0787cdc4c8d08be88788e287a02ef4f3cfe438cd2152c6b6ad7c61d0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:58:33.349 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 26 23:58:33 smithi103 podman[63771]: 2024-04-26 23:58:33.331761067 +0000 UTC m=+0.356379402 container init ee1cb2f0787cdc4c8d08be88788e287a02ef4f3cfe438cd2152c6b6ad7c61d0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:58:33.349 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 26 23:58:33 smithi103 podman[63771]: 2024-04-26 23:58:33.33458079 +0000 UTC m=+0.359199131 container start ee1cb2f0787cdc4c8d08be88788e287a02ef4f3cfe438cd2152c6b6ad7c61d0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:58:33.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:33.848 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 26 23:58:33 smithi103 ceph-mds[63790]: -- 172.21.15.103:0/4245227156 <== mon.1 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562488029a40 con 0x562487ffcc00 2024-04-26T23:58:33.848 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 26 23:58:33 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i[63786]: starting mds.i at 2024-04-26T23:58:33.848 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 26 23:58:33 smithi103 bash[63771]: ee1cb2f0787cdc4c8d08be88788e287a02ef4f3cfe438cd2152c6b6ad7c61d0b 2024-04-26T23:58:33.849 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 26 23:58:33 smithi103 systemd[1]: Started Ceph mds.i for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:58:33.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.873+0000 7f2cff000640 1 -- 172.21.15.40:0/3383428101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf8105dc0 msgr2=0x7f2cf81081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:33.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.873+0000 7f2cff000640 1 --2- 172.21.15.40:0/3383428101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf8105dc0 0x7f2cf81081b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009900 tx=0x7f2ce402a410 comp rx=0 tx=0).stop 2024-04-26T23:58:33.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.873+0000 7f2cff000640 1 -- 172.21.15.40:0/3383428101 shutdown_connections 2024-04-26T23:58:33.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.873+0000 7f2cff000640 1 --2- 172.21.15.40:0/3383428101 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cf81086f0 0x7f2cf810aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:33.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.873+0000 7f2cff000640 1 --2- 172.21.15.40:0/3383428101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf8105dc0 0x7f2cf81081b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:33.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.873+0000 7f2cff000640 1 --2- 172.21.15.40:0/3383428101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8103490 0x7f2cf8105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:33.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.873+0000 7f2cff000640 1 -- 172.21.15.40:0/3383428101 >> 172.21.15.40:0/3383428101 conn(0x7f2cf80fce00 msgr2=0x7f2cf80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:33.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.874+0000 7f2cff000640 1 -- 172.21.15.40:0/3383428101 shutdown_connections 2024-04-26T23:58:33.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.874+0000 7f2cff000640 1 -- 172.21.15.40:0/3383428101 wait complete. 2024-04-26T23:58:33.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.875+0000 7f2cff000640 1 Processor -- start 2024-04-26T23:58:33.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.875+0000 7f2cff000640 1 -- start start 2024-04-26T23:58:33.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.875+0000 7f2cff000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8103490 0x7f2cf8102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:33.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.875+0000 7f2cff000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cf8105dc0 0x7f2cf8101440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:33.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.875+0000 7f2cff000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf81086f0 0x7f2cf8101980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:33.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.875+0000 7f2cff000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf810e4b0 con 0x7f2cf81086f0 2024-04-26T23:58:33.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.875+0000 7f2cff000640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf810e330 con 0x7f2cf8105dc0 2024-04-26T23:58:33.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.875+0000 7f2cff000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf810e630 con 0x7f2cf8103490 2024-04-26T23:58:33.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.875+0000 7f2cf7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8103490 0x7f2cf8102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:33.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.875+0000 7f2cf7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8103490 0x7f2cf8102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43412/0 (socket says 172.21.15.40:43412) 2024-04-26T23:58:33.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.875+0000 7f2cf7e00640 1 -- 172.21.15.40:0/242705294 learned_addr learned my addr 172.21.15.40:0/242705294 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:33.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.875+0000 7f2cfd000640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf81086f0 0x7f2cf8101980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:33.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.876+0000 7f2cf7400640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cf8105dc0 0x7f2cf8101440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:33.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.876+0000 7f2cf7e00640 1 -- 172.21.15.40:0/242705294 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cf8105dc0 msgr2=0x7f2cf8101440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:33.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.876+0000 7f2cf7e00640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cf8105dc0 0x7f2cf8101440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:33.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.876+0000 7f2cf7e00640 1 -- 172.21.15.40:0/242705294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf81086f0 msgr2=0x7f2cf8101980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:33.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.876+0000 7f2cf7e00640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf81086f0 0x7f2cf8101980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:33.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.876+0000 7f2cf7e00640 1 -- 172.21.15.40:0/242705294 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf8102210 con 0x7f2cf8103490 2024-04-26T23:58:33.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.876+0000 7f2cf7e00640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8103490 0x7f2cf8102e00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ce800ea60 tx=0x7f2ce800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:33.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.876+0000 7f2cf4c00640 1 -- 172.21.15.40:0/242705294 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce800cde0 con 0x7f2cf8103490 2024-04-26T23:58:33.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.877+0000 7f2cf4c00640 1 -- 172.21.15.40:0/242705294 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ce80035f0 con 0x7f2cf8103490 2024-04-26T23:58:33.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.877+0000 7f2cff000640 1 -- 172.21.15.40:0/242705294 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf81a2d00 con 0x7f2cf8103490 2024-04-26T23:58:33.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.877+0000 7f2cf4c00640 1 -- 172.21.15.40:0/242705294 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8010690 con 0x7f2cf8103490 2024-04-26T23:58:33.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.877+0000 7f2cff000640 1 -- 172.21.15.40:0/242705294 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf81a31c0 con 0x7f2cf8103490 2024-04-26T23:58:33.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.878+0000 7f2cff000640 1 -- 172.21.15.40:0/242705294 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf80fe8f0 con 0x7f2cf8103490 2024-04-26T23:58:33.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.880+0000 7f2cf4c00640 1 -- 172.21.15.40:0/242705294 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2ce80037a0 con 0x7f2cf8103490 2024-04-26T23:58:33.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.881+0000 7f2cf4c00640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2cdc0a3c30 0x7f2cdc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:33.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.881+0000 7f2cf4c00640 1 -- 172.21.15.40:0/242705294 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2ce80d0270 con 0x7f2cf8103490 2024-04-26T23:58:33.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.881+0000 7f2cf7400640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2cdc0a3c30 0x7f2cdc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:33.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.882+0000 7f2cf7400640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2cdc0a3c30 0x7f2cdc0a60f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4005ab0 tx=0x7f2ce4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:33.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:33.885+0000 7f2cf4c00640 1 -- 172.21.15.40:0/242705294 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2ce809d050 con 0x7f2cf8103490 2024-04-26T23:58:34.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.134+0000 7f2cff000640 1 -- 172.21.15.40:0/242705294 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2cf8063530 con 0x7f2cf8103490 2024-04-26T23:58:34.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.135+0000 7f2cf4c00640 1 -- 172.21.15.40:0/242705294 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7f2ce809c950 con 0x7f2cf8103490 2024-04-26T23:58:34.137 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:34.137 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T23:58:34.138 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 10 2024-04-26T23:58:34.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.137+0000 7f2cff000640 1 -- 172.21.15.40:0/242705294 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2cdc0a3c30 msgr2=0x7f2cdc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:34.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.137+0000 7f2cff000640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2cdc0a3c30 0x7f2cdc0a60f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4005ab0 tx=0x7f2ce4005a20 comp rx=0 tx=0).stop 2024-04-26T23:58:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.137+0000 7f2cff000640 1 -- 172.21.15.40:0/242705294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8103490 msgr2=0x7f2cf8102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.137+0000 7f2cff000640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8103490 0x7f2cf8102e00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ce800ea60 tx=0x7f2ce800ef20 comp rx=0 tx=0).stop 2024-04-26T23:58:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.138+0000 7f2cff000640 1 -- 172.21.15.40:0/242705294 shutdown_connections 2024-04-26T23:58:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.138+0000 7f2cff000640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf81086f0 0x7f2cf8101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.138+0000 7f2cff000640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cf8105dc0 0x7f2cf8101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.138+0000 7f2cff000640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2cdc0a3c30 0x7f2cdc0a60f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.138+0000 7f2cff000640 1 --2- 172.21.15.40:0/242705294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8103490 0x7f2cf8102e00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.138+0000 7f2cff000640 1 -- 172.21.15.40:0/242705294 >> 172.21.15.40:0/242705294 conn(0x7f2cf80fce00 msgr2=0x7f2cf8103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.138+0000 7f2cff000640 1 -- 172.21.15.40:0/242705294 shutdown_connections 2024-04-26T23:58:34.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:34.138+0000 7f2cff000640 1 -- 172.21.15.40:0/242705294 wait complete. 2024-04-26T23:58:34.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:34 smithi040 ceph-mon[28095]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:34.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:34.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:34.741 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:34.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:34.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:34.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:34.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:34 smithi040 ceph-mon[28095]: Deploying daemon mds.g on smithi040 2024-04-26T23:58:34.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:34 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] up:boot 2024-04-26T23:58:34.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:34 smithi040 ceph-mon[28095]: fsmap 9 up:standby 2024-04-26T23:58:34.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T23:58:34.742 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:34 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/242705294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:34.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:34 smithi080 ceph-mon[32362]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:34.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:34.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:34.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:34.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:34.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:34.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:34.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:34 smithi080 ceph-mon[32362]: Deploying daemon mds.g on smithi040 2024-04-26T23:58:34.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:34 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] up:boot 2024-04-26T23:58:34.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:34 smithi080 ceph-mon[32362]: fsmap 9 up:standby 2024-04-26T23:58:34.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T23:58:34.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:34 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/242705294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:34.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:34 smithi103 ceph-mon[32952]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:34.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:34.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:34.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:34.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:34.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:34.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:34.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:34 smithi103 ceph-mon[32952]: Deploying daemon mds.g on smithi040 2024-04-26T23:58:34.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:34 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] up:boot 2024-04-26T23:58:34.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:34 smithi103 ceph-mon[32952]: fsmap 9 up:standby 2024-04-26T23:58:34.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T23:58:34.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:34 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/242705294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:36.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:36.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.404+0000 7fb4c5400640 1 -- 172.21.15.40:0/672988285 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4c0102860 msgr2=0x7fb4c0102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:36.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.404+0000 7fb4c5400640 1 --2- 172.21.15.40:0/672988285 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4c0102860 0x7fb4c0102ce0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009900 tx=0x7fb4ac02a410 comp rx=0 tx=0).stop 2024-04-26T23:58:36.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.404+0000 7fb4c5400640 1 -- 172.21.15.40:0/672988285 shutdown_connections 2024-04-26T23:58:36.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.404+0000 7fb4c5400640 1 --2- 172.21.15.40:0/672988285 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c0103220 0x7fb4c0109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:36.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.404+0000 7fb4c5400640 1 --2- 172.21.15.40:0/672988285 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4c0102860 0x7fb4c0102ce0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:36.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.404+0000 7fb4c5400640 1 --2- 172.21.15.40:0/672988285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4c0101670 0x7fb4c0101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:36.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.404+0000 7fb4c5400640 1 -- 172.21.15.40:0/672988285 >> 172.21.15.40:0/672988285 conn(0x7fb4c00fcbc0 msgr2=0x7fb4c00ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:36.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.404+0000 7fb4c5400640 1 -- 172.21.15.40:0/672988285 shutdown_connections 2024-04-26T23:58:36.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.404+0000 7fb4c5400640 1 -- 172.21.15.40:0/672988285 wait complete. 2024-04-26T23:58:36.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.406+0000 7fb4c5400640 1 Processor -- start 2024-04-26T23:58:36.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.406+0000 7fb4c5400640 1 -- start start 2024-04-26T23:58:36.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.407+0000 7fb4c5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c0101670 0x7fb4c01966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:36.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.407+0000 7fb4c5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4c0102860 0x7fb4c0196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:36.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.407+0000 7fb4c5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4c0103220 0x7fb4c019dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:36.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.407+0000 7fb4c5400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c010d1d0 con 0x7fb4c0103220 2024-04-26T23:58:36.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.407+0000 7fb4c5400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c010d050 con 0x7fb4c0102860 2024-04-26T23:58:36.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.407+0000 7fb4c5400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c010d350 con 0x7fb4c0101670 2024-04-26T23:58:36.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.407+0000 7fb4bfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c0101670 0x7fb4c01966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:36.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.407+0000 7fb4bfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c0101670 0x7fb4c01966a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43430/0 (socket says 172.21.15.40:43430) 2024-04-26T23:58:36.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.407+0000 7fb4bfe00640 1 -- 172.21.15.40:0/3027301697 learned_addr learned my addr 172.21.15.40:0/3027301697 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:36.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.407+0000 7fb4c4a00640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4c0103220 0x7fb4c019dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:36.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.408+0000 7fb4bf400640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4c0102860 0x7fb4c0196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:36.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.408+0000 7fb4bf400640 1 -- 172.21.15.40:0/3027301697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c0101670 msgr2=0x7fb4c01966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:36.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.408+0000 7fb4bf400640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c0101670 0x7fb4c01966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:36.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.408+0000 7fb4bf400640 1 -- 172.21.15.40:0/3027301697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4c0103220 msgr2=0x7fb4c019dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:36.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.408+0000 7fb4bf400640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4c0103220 0x7fb4c019dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:36.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.408+0000 7fb4bf400640 1 -- 172.21.15.40:0/3027301697 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c019e330 con 0x7fb4c0102860 2024-04-26T23:58:36.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.409+0000 7fb4c4a00640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4c0103220 0x7fb4c019dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:36.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.409+0000 7fb4bf400640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4c0102860 0x7fb4c0196be0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009850 tx=0x7fb4ac003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.409+0000 7fb4bcc00640 1 -- 172.21.15.40:0/3027301697 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac03f070 con 0x7fb4c0102860 2024-04-26T23:58:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.409+0000 7fb4bcc00640 1 -- 172.21.15.40:0/3027301697 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4ac031c50 con 0x7fb4c0102860 2024-04-26T23:58:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.409+0000 7fb4bcc00640 1 -- 172.21.15.40:0/3027301697 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac035030 con 0x7fb4c0102860 2024-04-26T23:58:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.409+0000 7fb4c5400640 1 -- 172.21.15.40:0/3027301697 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c019e5c0 con 0x7fb4c0102860 2024-04-26T23:58:36.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.410+0000 7fb4c5400640 1 -- 172.21.15.40:0/3027301697 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c019ea60 con 0x7fb4c0102860 2024-04-26T23:58:36.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.412+0000 7fb4b6000640 1 -- 172.21.15.40:0/3027301697 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4c0063530 con 0x7fb4c0102860 2024-04-26T23:58:36.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.415+0000 7fb4bcc00640 1 -- 172.21.15.40:0/3027301697 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb4ac042070 con 0x7fb4c0102860 2024-04-26T23:58:36.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.418+0000 7fb4bcc00640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb4a40a3db0 0x7fb4a40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:36.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.418+0000 7fb4bcc00640 1 -- 172.21.15.40:0/3027301697 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb4ac0f0850 con 0x7fb4c0102860 2024-04-26T23:58:36.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.419+0000 7fb4bfe00640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb4a40a3db0 0x7fb4a40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:36.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.420+0000 7fb4bfe00640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb4a40a3db0 0x7fb4a40a6270 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0005fd0 tx=0x7fb4b0005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:36.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.421+0000 7fb4bcc00640 1 -- 172.21.15.40:0/3027301697 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb4ac0bd630 con 0x7fb4c0102860 2024-04-26T23:58:36.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.678+0000 7fb4b6000640 1 -- 172.21.15.40:0/3027301697 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4c0063730 con 0x7fb4c0102860 2024-04-26T23:58:36.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.679+0000 7fb4bcc00640 1 -- 172.21.15.40:0/3027301697 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7fb4ac0bd630 con 0x7fb4c0102860 2024-04-26T23:58:36.681 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:36.681 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T23:58:36.681 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 10 2024-04-26T23:58:36.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.681+0000 7fb4b6000640 1 -- 172.21.15.40:0/3027301697 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb4a40a3db0 msgr2=0x7fb4a40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:36.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.681+0000 7fb4b6000640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb4a40a3db0 0x7fb4a40a6270 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0005fd0 tx=0x7fb4b0005e50 comp rx=0 tx=0).stop 2024-04-26T23:58:36.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.681+0000 7fb4b6000640 1 -- 172.21.15.40:0/3027301697 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4c0102860 msgr2=0x7fb4c0196be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:36.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.681+0000 7fb4b6000640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4c0102860 0x7fb4c0196be0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009850 tx=0x7fb4ac003b90 comp rx=0 tx=0).stop 2024-04-26T23:58:36.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.681+0000 7fb4b6000640 1 -- 172.21.15.40:0/3027301697 shutdown_connections 2024-04-26T23:58:36.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.681+0000 7fb4b6000640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4c0103220 0x7fb4c019dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:36.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.681+0000 7fb4b6000640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb4c0102860 0x7fb4c0196be0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:36.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.681+0000 7fb4b6000640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb4a40a3db0 0x7fb4a40a6270 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:36.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.681+0000 7fb4b6000640 1 --2- 172.21.15.40:0/3027301697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c0101670 0x7fb4c01966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:36.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.681+0000 7fb4b6000640 1 -- 172.21.15.40:0/3027301697 >> 172.21.15.40:0/3027301697 conn(0x7fb4c00fcbc0 msgr2=0x7fb4c00fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:36.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.682+0000 7fb4b6000640 1 -- 172.21.15.40:0/3027301697 shutdown_connections 2024-04-26T23:58:36.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:36.682+0000 7fb4b6000640 1 -- 172.21.15.40:0/3027301697 wait complete. 2024-04-26T23:58:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:36 smithi080 ceph-mon[32362]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:36 smithi103 ceph-mon[32952]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:36.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:36 smithi040 ceph-mon[28095]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:37.327 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 26 23:58:37 smithi040 podman[67109]: 2024-04-26 23:58:37.05410093 +0000 UTC m=+0.164035155 container create 337ff61668b7e1ef1dab8d55472fe9aa38786d645b937886c1fcd361467eb3b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-04-26T23:58:37.328 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 26 23:58:37 smithi040 podman[67109]: 2024-04-26 23:58:37.324660412 +0000 UTC m=+0.434594641 container init 337ff61668b7e1ef1dab8d55472fe9aa38786d645b937886c1fcd361467eb3b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-26T23:58:37.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:37 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3027301697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:37.717 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 26 23:58:37 smithi040 podman[67109]: 2024-04-26 23:58:37.336220704 +0000 UTC m=+0.446154916 container start 337ff61668b7e1ef1dab8d55472fe9aa38786d645b937886c1fcd361467eb3b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-26T23:58:37.717 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 26 23:58:37 smithi040 ceph-mds[67128]: -- 172.21.15.40:0/404470516 <== mon.2 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e7de721a40 con 0x55e7de6f4c00 2024-04-26T23:58:37.717 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 26 23:58:37 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g[67124]: starting mds.g at 2024-04-26T23:58:37.717 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 26 23:58:37 smithi040 bash[67109]: 337ff61668b7e1ef1dab8d55472fe9aa38786d645b937886c1fcd361467eb3b2 2024-04-26T23:58:37.717 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 26 23:58:37 smithi040 systemd[1]: Started Ceph mds.g for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:58:37.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:37 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3027301697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:37.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:37 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3027301697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:38.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:38.785 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:38 smithi040 ceph-mon[28095]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:38.785 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:38 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] up:boot 2024-04-26T23:58:38.785 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:38 smithi040 ceph-mon[28095]: fsmap 10 up:standby 2024-04-26T23:58:38.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T23:58:38.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:38.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:38.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:38.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:38.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:38.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:38.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:38 smithi080 ceph-mon[32362]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:38.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:38 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] up:boot 2024-04-26T23:58:38.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:38 smithi080 ceph-mon[32362]: fsmap 10 up:standby 2024-04-26T23:58:38.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T23:58:38.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:38.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:38.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:38.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:38.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:38.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:38.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:38 smithi103 ceph-mon[32952]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:38.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:38 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] up:boot 2024-04-26T23:58:38.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:38 smithi103 ceph-mon[32952]: fsmap 10 up:standby 2024-04-26T23:58:38.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T23:58:38.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:38.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:38.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:38.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:38.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:38.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:38.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.926+0000 7f2924c00640 1 -- 172.21.15.40:0/1425003357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2920105b80 msgr2=0x7f2920106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:38.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.926+0000 7f2924c00640 1 --2- 172.21.15.40:0/1425003357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2920105b80 0x7f2920106000 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f290c009900 tx=0x7f290c02a400 comp rx=0 tx=0).stop 2024-04-26T23:58:38.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.926+0000 7f2924c00640 1 -- 172.21.15.40:0/1425003357 shutdown_connections 2024-04-26T23:58:38.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.926+0000 7f2924c00640 1 --2- 172.21.15.40:0/1425003357 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2920106540 0x7f292010cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:38.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.926+0000 7f2924c00640 1 --2- 172.21.15.40:0/1425003357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2920105b80 0x7f2920106000 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:38.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.926+0000 7f2924c00640 1 --2- 172.21.15.40:0/1425003357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2920104990 0x7f2920104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:38.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.926+0000 7f2924c00640 1 -- 172.21.15.40:0/1425003357 >> 172.21.15.40:0/1425003357 conn(0x7f29200fff00 msgr2=0x7f2920102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:38.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.926+0000 7f2924c00640 1 -- 172.21.15.40:0/1425003357 shutdown_connections 2024-04-26T23:58:38.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.926+0000 7f2924c00640 1 -- 172.21.15.40:0/1425003357 wait complete. 2024-04-26T23:58:38.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.927+0000 7f2924c00640 1 Processor -- start 2024-04-26T23:58:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.927+0000 7f2924c00640 1 -- start start 2024-04-26T23:58:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.927+0000 7f2924c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2920104990 0x7f292019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f2924c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2920105b80 0x7f292019b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f2924c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2920106540 0x7f29201a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f2924c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f29201104f0 con 0x7f2920106540 2024-04-26T23:58:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f2924c00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f2920110370 con 0x7f2920104990 2024-04-26T23:58:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f2924c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2920110670 con 0x7f2920105b80 2024-04-26T23:58:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f291f400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2920104990 0x7f292019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f291fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2920106540 0x7f29201a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f291f400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2920104990 0x7f292019ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:59110/0 (socket says 172.21.15.40:59110) 2024-04-26T23:58:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f291f400640 1 -- 172.21.15.40:0/3066978146 learned_addr learned my addr 172.21.15.40:0/3066978146 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f291ea00640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2920105b80 0x7f292019b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:38.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f291f400640 1 -- 172.21.15.40:0/3066978146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2920105b80 msgr2=0x7f292019b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:38.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f291f400640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2920105b80 0x7f292019b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:38.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f291f400640 1 -- 172.21.15.40:0/3066978146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2920106540 msgr2=0x7f29201a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:38.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f291f400640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2920106540 0x7f29201a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:38.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f291f400640 1 -- 172.21.15.40:0/3066978146 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29201a27a0 con 0x7f2920104990 2024-04-26T23:58:38.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.928+0000 7f291ea00640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2920105b80 0x7f292019b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:58:38.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.929+0000 7f291f400640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2920104990 0x7f292019ab80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f291000d770 tx=0x7f291000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:38.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.929+0000 7f2917e00640 1 -- 172.21.15.40:0/3066978146 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910017070 con 0x7f2920104990 2024-04-26T23:58:38.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.929+0000 7f2917e00640 1 -- 172.21.15.40:0/3066978146 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2910009c30 con 0x7f2920104990 2024-04-26T23:58:38.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.929+0000 7f2924c00640 1 -- 172.21.15.40:0/3066978146 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29201a2a90 con 0x7f2920104990 2024-04-26T23:58:38.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.929+0000 7f2917e00640 1 -- 172.21.15.40:0/3066978146 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910003cc0 con 0x7f2920104990 2024-04-26T23:58:38.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.929+0000 7f2924c00640 1 -- 172.21.15.40:0/3066978146 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29201a3000 con 0x7f2920104990 2024-04-26T23:58:38.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.930+0000 7f2924c00640 1 -- 172.21.15.40:0/3066978146 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f292010bd80 con 0x7f2920104990 2024-04-26T23:58:38.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.932+0000 7f2917e00640 1 -- 172.21.15.40:0/3066978146 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2910003e60 con 0x7f2920104990 2024-04-26T23:58:38.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.934+0000 7f2917e00640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29040a3c30 0x7f29040a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:38.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.934+0000 7f2917e00640 1 -- 172.21.15.40:0/3066978146 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f29100cf910 con 0x7f2920104990 2024-04-26T23:58:38.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.934+0000 7f291ea00640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29040a3c30 0x7f29040a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:38.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.935+0000 7f291ea00640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29040a3c30 0x7f29040a60f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f290c000c00 tx=0x7f290c0314c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:38.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:38.938+0000 7f2917e00640 1 -- 172.21.15.40:0/3066978146 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2910009de0 con 0x7f2920104990 2024-04-26T23:58:39.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.196+0000 7f2924c00640 1 -- 172.21.15.40:0/3066978146 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f292002dad0 con 0x7f2920104990 2024-04-26T23:58:39.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.197+0000 7f2917e00640 1 -- 172.21.15.40:0/3066978146 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7f29100bc090 con 0x7f2920104990 2024-04-26T23:58:39.199 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:39.199 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-26T23:58:39.199 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 11 2024-04-26T23:58:39.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.200+0000 7f2924c00640 1 -- 172.21.15.40:0/3066978146 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29040a3c30 msgr2=0x7f29040a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:39.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.200+0000 7f2924c00640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29040a3c30 0x7f29040a60f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f290c000c00 tx=0x7f290c0314c0 comp rx=0 tx=0).stop 2024-04-26T23:58:39.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.200+0000 7f2924c00640 1 -- 172.21.15.40:0/3066978146 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2920104990 msgr2=0x7f292019ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:39.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.200+0000 7f2924c00640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2920104990 0x7f292019ab80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f291000d770 tx=0x7f291000dc30 comp rx=0 tx=0).stop 2024-04-26T23:58:39.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.200+0000 7f2924c00640 1 -- 172.21.15.40:0/3066978146 shutdown_connections 2024-04-26T23:58:39.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.200+0000 7f2924c00640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2920106540 0x7f29201a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:39.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.200+0000 7f2924c00640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29040a3c30 0x7f29040a60f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:39.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.200+0000 7f2924c00640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2920105b80 0x7f292019b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:39.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.200+0000 7f2924c00640 1 --2- 172.21.15.40:0/3066978146 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2920104990 0x7f292019ab80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:39.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.200+0000 7f2924c00640 1 -- 172.21.15.40:0/3066978146 >> 172.21.15.40:0/3066978146 conn(0x7f29200fff00 msgr2=0x7f2920108f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:39.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.200+0000 7f2924c00640 1 -- 172.21.15.40:0/3066978146 shutdown_connections 2024-04-26T23:58:39.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:39.200+0000 7f2924c00640 1 -- 172.21.15.40:0/3066978146 wait complete. 2024-04-26T23:58:39.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:39 smithi080 ceph-mon[32362]: Deploying daemon mds.a on smithi040 2024-04-26T23:58:39.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:39 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3066978146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:39.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:39 smithi103 ceph-mon[32952]: Deploying daemon mds.a on smithi040 2024-04-26T23:58:39.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:39 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3066978146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:39.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:39 smithi040 ceph-mon[28095]: Deploying daemon mds.a on smithi040 2024-04-26T23:58:39.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:39 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3066978146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:40.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:40 smithi080 ceph-mon[32362]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:40.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:40 smithi103 ceph-mon[32952]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:40.962 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:40 smithi040 ceph-mon[28095]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:41.217 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 26 23:58:40 smithi040 systemd[1]: Starting Ceph mds.a for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-26T23:58:41.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.415+0000 7f2c3e800640 1 -- 172.21.15.40:0/474406898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c381031f0 msgr2=0x7f2c381055e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:41.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.415+0000 7f2c3e800640 1 --2- 172.21.15.40:0/474406898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c381031f0 0x7f2c381055e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009900 tx=0x7f2c2802a410 comp rx=0 tx=0).stop 2024-04-26T23:58:41.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.415+0000 7f2c3e800640 1 -- 172.21.15.40:0/474406898 shutdown_connections 2024-04-26T23:58:41.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.415+0000 7f2c3e800640 1 --2- 172.21.15.40:0/474406898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c38108450 0x7f2c3810a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:41.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.415+0000 7f2c3e800640 1 --2- 172.21.15.40:0/474406898 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c38105b20 0x7f2c38107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:41.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.415+0000 7f2c3e800640 1 --2- 172.21.15.40:0/474406898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c381031f0 0x7f2c381055e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:41.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.415+0000 7f2c3e800640 1 -- 172.21.15.40:0/474406898 >> 172.21.15.40:0/474406898 conn(0x7f2c380fcbc0 msgr2=0x7f2c380ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:41.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.415+0000 7f2c3e800640 1 -- 172.21.15.40:0/474406898 shutdown_connections 2024-04-26T23:58:41.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.416+0000 7f2c3e800640 1 -- 172.21.15.40:0/474406898 wait complete. 2024-04-26T23:58:41.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.416+0000 7f2c3e800640 1 Processor -- start 2024-04-26T23:58:41.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.417+0000 7f2c3e800640 1 -- start start 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.417+0000 7f2c3e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c381031f0 0x7f2c3819ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.417+0000 7f2c3e800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c38105b20 0x7f2c3819b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.417+0000 7f2c3e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c38108450 0x7f2c381a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.417+0000 7f2c3e800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3810e2c0 con 0x7f2c381031f0 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.417+0000 7f2c3e800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3810e140 con 0x7f2c38105b20 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.417+0000 7f2c3e800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3810e440 con 0x7f2c38108450 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.417+0000 7f2c3d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c381031f0 0x7f2c3819ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.417+0000 7f2c3de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c38108450 0x7f2c381a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.417+0000 7f2c3d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c381031f0 0x7f2c3819ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43752/0 (socket says 172.21.15.40:43752) 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.417+0000 7f2c3d400640 1 -- 172.21.15.40:0/2559729813 learned_addr learned my addr 172.21.15.40:0/2559729813 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.417+0000 7f2c3ca00640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c38105b20 0x7f2c3819b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.418+0000 7f2c3de00640 1 -- 172.21.15.40:0/2559729813 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c38105b20 msgr2=0x7f2c3819b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.418+0000 7f2c3de00640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c38105b20 0x7f2c3819b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.418+0000 7f2c3de00640 1 -- 172.21.15.40:0/2559729813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c381031f0 msgr2=0x7f2c3819ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.418+0000 7f2c3de00640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c381031f0 0x7f2c3819ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.418+0000 7f2c3de00640 1 -- 172.21.15.40:0/2559729813 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c381a2790 con 0x7f2c38108450 2024-04-26T23:58:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.418+0000 7f2c3de00640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c38108450 0x7f2c381a2120 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009fd0 tx=0x7f2c3000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.418+0000 7f2c36000640 1 -- 172.21.15.40:0/2559729813 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3000ce20 con 0x7f2c38108450 2024-04-26T23:58:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.418+0000 7f2c36000640 1 -- 172.21.15.40:0/2559729813 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c30005020 con 0x7f2c38108450 2024-04-26T23:58:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.418+0000 7f2c3e800640 1 -- 172.21.15.40:0/2559729813 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c381a2a80 con 0x7f2c38108450 2024-04-26T23:58:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.418+0000 7f2c36000640 1 -- 172.21.15.40:0/2559729813 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c30010690 con 0x7f2c38108450 2024-04-26T23:58:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.418+0000 7f2c3e800640 1 -- 172.21.15.40:0/2559729813 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c381a2fd0 con 0x7f2c38108450 2024-04-26T23:58:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.419+0000 7f2c3e800640 1 -- 172.21.15.40:0/2559729813 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c38109fa0 con 0x7f2c38108450 2024-04-26T23:58:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.422+0000 7f2c36000640 1 -- 172.21.15.40:0/2559729813 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2c300054d0 con 0x7f2c38108450 2024-04-26T23:58:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.424+0000 7f2c36000640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2c1c0a3bb0 0x7f2c1c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.424+0000 7f2c36000640 1 -- 172.21.15.40:0/2559729813 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2c300d06d0 con 0x7f2c38108450 2024-04-26T23:58:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.424+0000 7f2c3d400640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2c1c0a3bb0 0x7f2c1c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.425+0000 7f2c3d400640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2c1c0a3bb0 0x7f2c1c0a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009230 tx=0x7f2c280058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:41.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.428+0000 7f2c36000640 1 -- 172.21.15.40:0/2559729813 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2c3009d4b0 con 0x7f2c38108450 2024-04-26T23:58:41.550 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 26 23:58:41 smithi040 podman[67454]: 2024-04-26 23:58:41.243522368 +0000 UTC m=+0.037408571 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:58:41.550 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 26 23:58:41 smithi040 podman[67454]: 2024-04-26 23:58:41.383569176 +0000 UTC m=+0.177455381 container create fab18cca5df1502457b49677eb9352f2ab2741f29425a19678cd16ef5ac2269d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-26T23:58:41.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.688+0000 7f2c3e800640 1 -- 172.21.15.40:0/2559729813 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c3802dad0 con 0x7f2c38108450 2024-04-26T23:58:41.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.689+0000 7f2c36000640 1 -- 172.21.15.40:0/2559729813 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7f2c3009cdb0 con 0x7f2c38108450 2024-04-26T23:58:41.690 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:41.691 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-26T23:58:41.691 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 11 2024-04-26T23:58:41.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.692+0000 7f2c3e800640 1 -- 172.21.15.40:0/2559729813 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2c1c0a3bb0 msgr2=0x7f2c1c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:41.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.692+0000 7f2c3e800640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2c1c0a3bb0 0x7f2c1c0a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009230 tx=0x7f2c280058d0 comp rx=0 tx=0).stop 2024-04-26T23:58:41.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.692+0000 7f2c3e800640 1 -- 172.21.15.40:0/2559729813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c38108450 msgr2=0x7f2c381a2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:41.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.692+0000 7f2c3e800640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c38108450 0x7f2c381a2120 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009fd0 tx=0x7f2c3000ef10 comp rx=0 tx=0).stop 2024-04-26T23:58:41.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.692+0000 7f2c3e800640 1 -- 172.21.15.40:0/2559729813 shutdown_connections 2024-04-26T23:58:41.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.692+0000 7f2c3e800640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2c1c0a3bb0 0x7f2c1c0a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:41.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.692+0000 7f2c3e800640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c38108450 0x7f2c381a2120 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:41.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.692+0000 7f2c3e800640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2c38105b20 0x7f2c3819b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:41.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.692+0000 7f2c3e800640 1 --2- 172.21.15.40:0/2559729813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c381031f0 0x7f2c3819ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:41.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.692+0000 7f2c3e800640 1 -- 172.21.15.40:0/2559729813 >> 172.21.15.40:0/2559729813 conn(0x7f2c380fcbc0 msgr2=0x7f2c38102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:41.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.692+0000 7f2c3e800640 1 -- 172.21.15.40:0/2559729813 shutdown_connections 2024-04-26T23:58:41.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:41.693+0000 7f2c3e800640 1 -- 172.21.15.40:0/2559729813 wait complete. 2024-04-26T23:58:41.968 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 26 23:58:41 smithi040 podman[67454]: 2024-04-26 23:58:41.653881598 +0000 UTC m=+0.447767854 container init fab18cca5df1502457b49677eb9352f2ab2741f29425a19678cd16ef5ac2269d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-26T23:58:41.968 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 26 23:58:41 smithi040 podman[67454]: 2024-04-26 23:58:41.663626336 +0000 UTC m=+0.457512544 container start fab18cca5df1502457b49677eb9352f2ab2741f29425a19678cd16ef5ac2269d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T23:58:41.968 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 26 23:58:41 smithi040 ceph-mds[67512]: -- 172.21.15.40:0/546528488 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c692f8fa40 con 0x55c692f63000 2024-04-26T23:58:41.968 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 26 23:58:41 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-a[67508]: starting mds.a at 2024-04-26T23:58:41.968 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 26 23:58:41 smithi040 bash[67454]: fab18cca5df1502457b49677eb9352f2ab2741f29425a19678cd16ef5ac2269d 2024-04-26T23:58:41.968 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 26 23:58:41 smithi040 systemd[1]: Started Ceph mds.a for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:58:42.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:42 smithi080 ceph-mon[32362]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:42.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:42 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2559729813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:42.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:42 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:42.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:42 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:42.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:42 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:42.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:42 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:42.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:42 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:42.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:42 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:42.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:42 smithi103 ceph-mon[32952]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:42.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:42 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2559729813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:42.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:42 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:42.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:42 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:42.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:42 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:42.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:42 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:42.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:42 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:42.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:42 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:42.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:42 smithi040 ceph-mon[28095]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:42.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:42 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2559729813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:42.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:42 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:42.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:42 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:42.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:42 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:42.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:42 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:58:42.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:42 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:58:42.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:42 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:43.713 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:43.781 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:43 smithi080 ceph-mon[32362]: Deploying daemon mds.b on smithi080 2024-04-26T23:58:43.782 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:43 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.40:6840/2265538751,v1:172.21.15.40:6841/2265538751] up:boot 2024-04-26T23:58:43.782 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:43 smithi080 ceph-mon[32362]: fsmap 11 up:standby 2024-04-26T23:58:43.782 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:43 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T23:58:43.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:43 smithi040 ceph-mon[28095]: Deploying daemon mds.b on smithi080 2024-04-26T23:58:43.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:43 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.40:6840/2265538751,v1:172.21.15.40:6841/2265538751] up:boot 2024-04-26T23:58:43.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:43 smithi040 ceph-mon[28095]: fsmap 11 up:standby 2024-04-26T23:58:43.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:43 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T23:58:43.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:43 smithi103 ceph-mon[32952]: Deploying daemon mds.b on smithi080 2024-04-26T23:58:43.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:43 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.40:6840/2265538751,v1:172.21.15.40:6841/2265538751] up:boot 2024-04-26T23:58:43.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:43 smithi103 ceph-mon[32952]: fsmap 11 up:standby 2024-04-26T23:58:43.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:43 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T23:58:43.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.937+0000 7f63eac00640 1 -- 172.21.15.40:0/1704679206 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f63e4105b70 msgr2=0x7f63e4105ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:43.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.937+0000 7f63eac00640 1 --2- 172.21.15.40:0/1704679206 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f63e4105b70 0x7f63e4105ff0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009900 tx=0x7f63d002a410 comp rx=0 tx=0).stop 2024-04-26T23:58:43.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.937+0000 7f63eac00640 1 -- 172.21.15.40:0/1704679206 shutdown_connections 2024-04-26T23:58:43.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.937+0000 7f63eac00640 1 --2- 172.21.15.40:0/1704679206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e4106530 0x7f63e410cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:43.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.937+0000 7f63eac00640 1 --2- 172.21.15.40:0/1704679206 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f63e4105b70 0x7f63e4105ff0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:43.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.937+0000 7f63eac00640 1 --2- 172.21.15.40:0/1704679206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e4104980 0x7f63e4104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:43.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.937+0000 7f63eac00640 1 -- 172.21.15.40:0/1704679206 >> 172.21.15.40:0/1704679206 conn(0x7f63e40fff10 msgr2=0x7f63e4102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:43.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.937+0000 7f63eac00640 1 -- 172.21.15.40:0/1704679206 shutdown_connections 2024-04-26T23:58:43.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.937+0000 7f63eac00640 1 -- 172.21.15.40:0/1704679206 wait complete. 2024-04-26T23:58:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.938+0000 7f63eac00640 1 Processor -- start 2024-04-26T23:58:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.938+0000 7f63eac00640 1 -- start start 2024-04-26T23:58:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.939+0000 7f63eac00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e4104980 0x7f63e419ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.939+0000 7f63eac00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f63e4105b70 0x7f63e419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.939+0000 7f63eac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e4106530 0x7f63e41a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.939+0000 7f63eac00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63e41104e0 con 0x7f63e4106530 2024-04-26T23:58:43.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.939+0000 7f63eac00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f63e4110360 con 0x7f63e4105b70 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.939+0000 7f63eac00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63e4110660 con 0x7f63e4104980 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.939+0000 7f63e8e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f63e4105b70 0x7f63e419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.939+0000 7f63e8e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f63e4105b70 0x7f63e419b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:54966/0 (socket says 172.21.15.40:54966) 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.939+0000 7f63e8e00640 1 -- 172.21.15.40:0/1709072294 learned_addr learned my addr 172.21.15.40:0/1709072294 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.939+0000 7f63e9800640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e4104980 0x7f63e419ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.939+0000 7f63ea200640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e4106530 0x7f63e41a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.940+0000 7f63e8e00640 1 -- 172.21.15.40:0/1709072294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e4104980 msgr2=0x7f63e419ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.940+0000 7f63e8e00640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e4104980 0x7f63e419ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.940+0000 7f63e8e00640 1 -- 172.21.15.40:0/1709072294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e4106530 msgr2=0x7f63e41a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.940+0000 7f63e8e00640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e4106530 0x7f63e41a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.940+0000 7f63e8e00640 1 -- 172.21.15.40:0/1709072294 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e41a27b0 con 0x7f63e4105b70 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.940+0000 7f63ea200640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e4106530 0x7f63e41a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:58:43.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.940+0000 7f63e8e00640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f63e4105b70 0x7f63e419b0d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f63d0005ab0 tx=0x7f63d0003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:43.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.940+0000 7f63e2000640 1 -- 172.21.15.40:0/1709072294 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d003f070 con 0x7f63e4105b70 2024-04-26T23:58:43.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.940+0000 7f63e2000640 1 -- 172.21.15.40:0/1709072294 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d0031c50 con 0x7f63e4105b70 2024-04-26T23:58:43.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.940+0000 7f63eac00640 1 -- 172.21.15.40:0/1709072294 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63e41a2a40 con 0x7f63e4105b70 2024-04-26T23:58:43.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.940+0000 7f63e2000640 1 -- 172.21.15.40:0/1709072294 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0035030 con 0x7f63e4105b70 2024-04-26T23:58:43.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.940+0000 7f63eac00640 1 -- 172.21.15.40:0/1709072294 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63e41a2f80 con 0x7f63e4105b70 2024-04-26T23:58:43.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.942+0000 7f63eac00640 1 -- 172.21.15.40:0/1709072294 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63e410bca0 con 0x7f63e4105b70 2024-04-26T23:58:43.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.943+0000 7f63e2000640 1 -- 172.21.15.40:0/1709072294 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f63d0042070 con 0x7f63e4105b70 2024-04-26T23:58:43.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.945+0000 7f63e2000640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f63cc0a3c30 0x7f63cc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:43.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.945+0000 7f63e2000640 1 -- 172.21.15.40:0/1709072294 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f63d00f0830 con 0x7f63e4105b70 2024-04-26T23:58:43.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.945+0000 7f63e9800640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f63cc0a3c30 0x7f63cc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:43.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.946+0000 7f63e9800640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f63cc0a3c30 0x7f63cc0a60f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f63e41059d0 tx=0x7f63d800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:43.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:43.949+0000 7f63e2000640 1 -- 172.21.15.40:0/1709072294 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f63d00bd610 con 0x7f63e4105b70 2024-04-26T23:58:44.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.195+0000 7f63eac00640 1 -- 172.21.15.40:0/1709072294 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63e402dad0 con 0x7f63e4105b70 2024-04-26T23:58:44.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.196+0000 7f63e2000640 1 -- 172.21.15.40:0/1709072294 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7f63d00bcf10 con 0x7f63e4105b70 2024-04-26T23:58:44.197 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:44.198 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/2265538751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":2265538751},{"type":"v1","addr":"172.21.15.40:6841","nonce":2265538751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-26T23:58:44.198 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 12 2024-04-26T23:58:44.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.199+0000 7f63eac00640 1 -- 172.21.15.40:0/1709072294 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f63cc0a3c30 msgr2=0x7f63cc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.199+0000 7f63eac00640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f63cc0a3c30 0x7f63cc0a60f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f63e41059d0 tx=0x7f63d800b390 comp rx=0 tx=0).stop 2024-04-26T23:58:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.199+0000 7f63eac00640 1 -- 172.21.15.40:0/1709072294 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f63e4105b70 msgr2=0x7f63e419b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.199+0000 7f63eac00640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f63e4105b70 0x7f63e419b0d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f63d0005ab0 tx=0x7f63d0003b90 comp rx=0 tx=0).stop 2024-04-26T23:58:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.199+0000 7f63eac00640 1 -- 172.21.15.40:0/1709072294 shutdown_connections 2024-04-26T23:58:44.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.199+0000 7f63eac00640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e4106530 0x7f63e41a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:44.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.199+0000 7f63eac00640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f63e4105b70 0x7f63e419b0d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:44.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.199+0000 7f63eac00640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f63cc0a3c30 0x7f63cc0a60f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:44.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.199+0000 7f63eac00640 1 --2- 172.21.15.40:0/1709072294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e4104980 0x7f63e419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:44.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.199+0000 7f63eac00640 1 -- 172.21.15.40:0/1709072294 >> 172.21.15.40:0/1709072294 conn(0x7f63e40fff10 msgr2=0x7f63e4108e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:44.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.199+0000 7f63eac00640 1 -- 172.21.15.40:0/1709072294 shutdown_connections 2024-04-26T23:58:44.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:44.199+0000 7f63eac00640 1 -- 172.21.15.40:0/1709072294 wait complete. 2024-04-26T23:58:44.434 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 26 23:58:44 smithi080 podman[63135]: 2024-04-26 23:58:44.295378873 +0000 UTC m=+0.017707285 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:58:44.435 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 26 23:58:44 smithi080 podman[63135]: 2024-04-26 23:58:44.433708724 +0000 UTC m=+0.156037128 container create cb060557aa61d7c2dcc70a3c8eda64c5f9cb595091f65b9358c1805a682d6737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:58:44.704 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:44 smithi080 ceph-mon[32362]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:44.705 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:44 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1709072294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:44.705 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 26 23:58:44 smithi080 podman[63135]: 2024-04-26 23:58:44.67824737 +0000 UTC m=+0.400575774 container init cb060557aa61d7c2dcc70a3c8eda64c5f9cb595091f65b9358c1805a682d6737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:58:44.705 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 26 23:58:44 smithi080 podman[63135]: 2024-04-26 23:58:44.680854347 +0000 UTC m=+0.403182754 container start cb060557aa61d7c2dcc70a3c8eda64c5f9cb595091f65b9358c1805a682d6737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T23:58:44.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:44 smithi103 ceph-mon[32952]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:44.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:44 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1709072294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:44.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:44 smithi040 ceph-mon[28095]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:44.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:44 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1709072294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:45.050 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 26 23:58:44 smithi080 ceph-mds[63154]: -- 172.21.15.80:0/3367721479 <== mon.2 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ffdfbda40 con 0x562ffdf91000 2024-04-26T23:58:45.050 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 26 23:58:44 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b[63150]: starting mds.b at 2024-04-26T23:58:45.050 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 26 23:58:44 smithi080 bash[63135]: cb060557aa61d7c2dcc70a3c8eda64c5f9cb595091f65b9358c1805a682d6737 2024-04-26T23:58:45.050 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 26 23:58:44 smithi080 systemd[1]: Started Ceph mds.b for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-26T23:58:46.205 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:45 smithi080 ceph-mon[32362]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:46.205 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.205 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.205 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.205 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.206 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.206 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:58:46.206 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:45 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] up:boot 2024-04-26T23:58:46.206 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:45 smithi080 ceph-mon[32362]: fsmap 12 up:standby 2024-04-26T23:58:46.206 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T23:58:46.215 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:45 smithi103 ceph-mon[32952]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:46.215 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.215 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.216 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.216 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.216 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.216 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:58:46.216 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:45 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] up:boot 2024-04-26T23:58:46.216 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:45 smithi103 ceph-mon[32952]: fsmap 12 up:standby 2024-04-26T23:58:46.216 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T23:58:46.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:45 smithi040 ceph-mon[28095]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:46.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:46.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:58:46.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:45 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] up:boot 2024-04-26T23:58:46.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:45 smithi040 ceph-mon[28095]: fsmap 12 up:standby 2024-04-26T23:58:46.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T23:58:46.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.414+0000 7f331de00640 1 -- 172.21.15.40:0/886157025 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3318109f80 msgr2=0x7f331810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.414+0000 7f331de00640 1 --2- 172.21.15.40:0/886157025 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3318109f80 0x7f331810a400 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3308009900 tx=0x7f330802a410 comp rx=0 tx=0).stop 2024-04-26T23:58:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.414+0000 7f331de00640 1 -- 172.21.15.40:0/886157025 shutdown_connections 2024-04-26T23:58:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.414+0000 7f331de00640 1 --2- 172.21.15.40:0/886157025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f331810a940 0x7f33181111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.414+0000 7f331de00640 1 --2- 172.21.15.40:0/886157025 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3318109f80 0x7f331810a400 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.414+0000 7f331de00640 1 --2- 172.21.15.40:0/886157025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3318108d90 0x7f33181091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.414+0000 7f331de00640 1 -- 172.21.15.40:0/886157025 >> 172.21.15.40:0/886157025 conn(0x7f3318077d30 msgr2=0x7f331807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.414+0000 7f331de00640 1 -- 172.21.15.40:0/886157025 shutdown_connections 2024-04-26T23:58:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.414+0000 7f331de00640 1 -- 172.21.15.40:0/886157025 wait complete. 2024-04-26T23:58:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.415+0000 7f331de00640 1 Processor -- start 2024-04-26T23:58:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.415+0000 7f331de00640 1 -- start start 2024-04-26T23:58:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f331de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3318108d90 0x7f331819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f331de00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3318109f80 0x7f331819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:46.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f331de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331810a940 0x7f33181a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f331de00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f33181148f0 con 0x7f3318108d90 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f331de00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f3318114770 con 0x7f3318109f80 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f331de00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3318114a70 con 0x7f331810a940 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f331ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3318108d90 0x7f331819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f3317e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3318109f80 0x7f331819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f331ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3318108d90 0x7f331819ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43798/0 (socket says 172.21.15.40:43798) 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f331ca00640 1 -- 172.21.15.40:0/3939481214 learned_addr learned my addr 172.21.15.40:0/3939481214 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f331d400640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331810a940 0x7f33181a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f3317e00640 1 -- 172.21.15.40:0/3939481214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331810a940 msgr2=0x7f33181a6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f3317e00640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331810a940 0x7f33181a6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f3317e00640 1 -- 172.21.15.40:0/3939481214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3318108d90 msgr2=0x7f331819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f3317e00640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3318108d90 0x7f331819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.416+0000 7f3317e00640 1 -- 172.21.15.40:0/3939481214 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33181a6ba0 con 0x7f3318109f80 2024-04-26T23:58:46.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.417+0000 7f3317e00640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3318109f80 0x7f331819f4c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f330802aa00 tx=0x7f3308003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:46.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.417+0000 7f3315600640 1 -- 172.21.15.40:0/3939481214 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f330803f070 con 0x7f3318109f80 2024-04-26T23:58:46.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.417+0000 7f3315600640 1 -- 172.21.15.40:0/3939481214 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3308031c50 con 0x7f3318109f80 2024-04-26T23:58:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.417+0000 7f3315600640 1 -- 172.21.15.40:0/3939481214 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3308035030 con 0x7f3318109f80 2024-04-26T23:58:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.417+0000 7f331de00640 1 -- 172.21.15.40:0/3939481214 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33181a6e30 con 0x7f3318109f80 2024-04-26T23:58:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.417+0000 7f331de00640 1 -- 172.21.15.40:0/3939481214 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33181a7240 con 0x7f3318109f80 2024-04-26T23:58:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.418+0000 7f331de00640 1 -- 172.21.15.40:0/3939481214 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3318110180 con 0x7f3318109f80 2024-04-26T23:58:46.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.420+0000 7f3315600640 1 -- 172.21.15.40:0/3939481214 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3308042070 con 0x7f3318109f80 2024-04-26T23:58:46.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.421+0000 7f3315600640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f32fc0a3bb0 0x7f32fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:46.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.422+0000 7f3315600640 1 -- 172.21.15.40:0/3939481214 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f33080f0640 con 0x7f3318109f80 2024-04-26T23:58:46.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.422+0000 7f331ca00640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f32fc0a3bb0 0x7f32fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:46.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.423+0000 7f331ca00640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f32fc0a3bb0 0x7f32fc0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f33040058c0 tx=0x7f3304005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:46.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.426+0000 7f3315600640 1 -- 172.21.15.40:0/3939481214 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f33080bcdd0 con 0x7f3318109f80 2024-04-26T23:58:46.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.676+0000 7f331de00640 1 -- 172.21.15.40:0/3939481214 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f331802dad0 con 0x7f3318109f80 2024-04-26T23:58:46.678 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:46.678 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/2265538751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":2265538751},{"type":"v1","addr":"172.21.15.40:6841","nonce":2265538751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24590,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6839/3536638444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6838","nonce":3536638444},{"type":"v1","addr":"172.21.15.80:6839","nonce":3536638444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-26T23:58:46.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.677+0000 7f3315600640 1 -- 172.21.15.40:0/3939481214 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10482 (secure 0 0 0) 0x7f33080c10d0 con 0x7f3318109f80 2024-04-26T23:58:46.679 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 13 2024-04-26T23:58:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.680+0000 7f331de00640 1 -- 172.21.15.40:0/3939481214 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f32fc0a3bb0 msgr2=0x7f32fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.680+0000 7f331de00640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f32fc0a3bb0 0x7f32fc0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f33040058c0 tx=0x7f3304005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.680+0000 7f331de00640 1 -- 172.21.15.40:0/3939481214 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3318109f80 msgr2=0x7f331819f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.680+0000 7f331de00640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3318109f80 0x7f331819f4c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f330802aa00 tx=0x7f3308003c10 comp rx=0 tx=0).stop 2024-04-26T23:58:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.680+0000 7f331de00640 1 -- 172.21.15.40:0/3939481214 shutdown_connections 2024-04-26T23:58:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.680+0000 7f331de00640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f32fc0a3bb0 0x7f32fc0a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.680+0000 7f331de00640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f331810a940 0x7f33181a6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.680+0000 7f331de00640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3318109f80 0x7f331819f4c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.680+0000 7f331de00640 1 --2- 172.21.15.40:0/3939481214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3318108d90 0x7f331819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.680+0000 7f331de00640 1 -- 172.21.15.40:0/3939481214 >> 172.21.15.40:0/3939481214 conn(0x7f3318077d30 msgr2=0x7f331810d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.680+0000 7f331de00640 1 -- 172.21.15.40:0/3939481214 shutdown_connections 2024-04-26T23:58:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.680+0000 7f331de00640 1 -- 172.21.15.40:0/3939481214 wait complete. 2024-04-26T23:58:46.700 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:58:46.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.896+0000 7f95bca00640 1 -- 172.21.15.40:0/3680310672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b8101060 msgr2=0x7f95b8101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:46.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.896+0000 7f95bca00640 1 --2- 172.21.15.40:0/3680310672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b8101060 0x7f95b8101480 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009900 tx=0x7f95a802a480 comp rx=0 tx=0).stop 2024-04-26T23:58:46.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.896+0000 7f95bca00640 1 -- 172.21.15.40:0/3680310672 shutdown_connections 2024-04-26T23:58:46.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.896+0000 7f95bca00640 1 --2- 172.21.15.40:0/3680310672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95b8102360 0x7f95b81027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.897+0000 7f95bca00640 1 --2- 172.21.15.40:0/3680310672 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f95b81019c0 0x7f95b8101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.897+0000 7f95bca00640 1 --2- 172.21.15.40:0/3680310672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b8101060 0x7f95b8101480 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.897+0000 7f95bca00640 1 -- 172.21.15.40:0/3680310672 >> 172.21.15.40:0/3680310672 conn(0x7f95b80fcbc0 msgr2=0x7f95b80ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:46.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.897+0000 7f95bca00640 1 -- 172.21.15.40:0/3680310672 shutdown_connections 2024-04-26T23:58:46.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.897+0000 7f95bca00640 1 -- 172.21.15.40:0/3680310672 wait complete. 2024-04-26T23:58:46.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.898+0000 7f95bca00640 1 Processor -- start 2024-04-26T23:58:46.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.898+0000 7f95bca00640 1 -- start start 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.898+0000 7f95bca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b8101060 0x7f95b819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.898+0000 7f95bca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95b81019c0 0x7f95b819b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.898+0000 7f95bca00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f95b8102360 0x7f95b81a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.898+0000 7f95bca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f95b81104c0 con 0x7f95b8101060 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.898+0000 7f95bca00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f95b8110340 con 0x7f95b8102360 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.898+0000 7f95bca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f95b8110640 con 0x7f95b81019c0 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.898+0000 7f95b6a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95b81019c0 0x7f95b819b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.898+0000 7f95b6a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95b81019c0 0x7f95b819b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:55964/0 (socket says 172.21.15.40:55964) 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.898+0000 7f95b6a00640 1 -- 172.21.15.40:0/58229859 learned_addr learned my addr 172.21.15.40:0/58229859 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.899+0000 7f95b7e00640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f95b8102360 0x7f95b81a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.899+0000 7f95b7400640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b8101060 0x7f95b819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.899+0000 7f95b6a00640 1 -- 172.21.15.40:0/58229859 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f95b8102360 msgr2=0x7f95b81a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.899+0000 7f95b6a00640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f95b8102360 0x7f95b81a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.899+0000 7f95b6a00640 1 -- 172.21.15.40:0/58229859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b8101060 msgr2=0x7f95b819ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:46.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.899+0000 7f95b6a00640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b8101060 0x7f95b819ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:46.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.899+0000 7f95b6a00640 1 -- 172.21.15.40:0/58229859 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b81a2780 con 0x7f95b81019c0 2024-04-26T23:58:46.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.899+0000 7f95b6a00640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95b81019c0 0x7f95b819b0a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f95a400e9f0 tx=0x7f95a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:46.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.899+0000 7f95afe00640 1 -- 172.21.15.40:0/58229859 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a400cd80 con 0x7f95b81019c0 2024-04-26T23:58:46.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.900+0000 7f95afe00640 1 -- 172.21.15.40:0/58229859 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95a40108e0 con 0x7f95b81019c0 2024-04-26T23:58:46.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.900+0000 7f95bca00640 1 -- 172.21.15.40:0/58229859 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b81a2a70 con 0x7f95b81019c0 2024-04-26T23:58:46.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.900+0000 7f95afe00640 1 -- 172.21.15.40:0/58229859 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a4018980 con 0x7f95b81019c0 2024-04-26T23:58:46.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.900+0000 7f95bca00640 1 -- 172.21.15.40:0/58229859 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b81a3010 con 0x7f95b81019c0 2024-04-26T23:58:46.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.901+0000 7f95bca00640 1 -- 172.21.15.40:0/58229859 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95b810bdb0 con 0x7f95b81019c0 2024-04-26T23:58:46.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.903+0000 7f95afe00640 1 -- 172.21.15.40:0/58229859 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f95a4010430 con 0x7f95b81019c0 2024-04-26T23:58:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.905+0000 7f95afe00640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f959c0a3c30 0x7f959c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:46.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.905+0000 7f95afe00640 1 -- 172.21.15.40:0/58229859 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f95a40d0310 con 0x7f95b81019c0 2024-04-26T23:58:46.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.905+0000 7f95b7400640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f959c0a3c30 0x7f959c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:46.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.906+0000 7f95b7400640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f959c0a3c30 0x7f959c0a60f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f95a8000c00 tx=0x7f95a80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:46.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:46.909+0000 7f95afe00640 1 -- 172.21.15.40:0/58229859 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f95a409d0f0 con 0x7f95b81019c0 2024-04-26T23:58:47.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.126+0000 7f95bca00640 1 -- 172.21.15.40:0/58229859 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f95b80635d0 con 0x7f95b81019c0 2024-04-26T23:58:47.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.127+0000 7f95afe00640 1 -- 172.21.15.40:0/58229859 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18418 (secure 0 0 0) 0x7f95a409c9f0 con 0x7f95b81019c0 2024-04-26T23:58:47.129 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:47.129 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:57:52.532046+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:58:47.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.130+0000 7f95bca00640 1 -- 172.21.15.40:0/58229859 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f959c0a3c30 msgr2=0x7f959c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:47.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.130+0000 7f95bca00640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f959c0a3c30 0x7f959c0a60f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f95a8000c00 tx=0x7f95a80314a0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.130+0000 7f95bca00640 1 -- 172.21.15.40:0/58229859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95b81019c0 msgr2=0x7f95b819b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:47.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.130+0000 7f95bca00640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95b81019c0 0x7f95b819b0a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f95a400e9f0 tx=0x7f95a400eeb0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.130+0000 7f95bca00640 1 -- 172.21.15.40:0/58229859 shutdown_connections 2024-04-26T23:58:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.130+0000 7f95bca00640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f95b8102360 0x7f95b81a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.130+0000 7f95bca00640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f959c0a3c30 0x7f959c0a60f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.130+0000 7f95bca00640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95b81019c0 0x7f95b819b0a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.130+0000 7f95bca00640 1 --2- 172.21.15.40:0/58229859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95b8101060 0x7f95b819ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.130+0000 7f95bca00640 1 -- 172.21.15.40:0/58229859 >> 172.21.15.40:0/58229859 conn(0x7f95b80fcbc0 msgr2=0x7f95b80fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.130+0000 7f95bca00640 1 -- 172.21.15.40:0/58229859 shutdown_connections 2024-04-26T23:58:47.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.130+0000 7f95bca00640 1 -- 172.21.15.40:0/58229859 wait complete. 2024-04-26T23:58:47.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:46 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3939481214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:47.149 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:53:32.708818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:58:47.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:58:47.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:46 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3939481214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:47.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:46 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3939481214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.359+0000 7f5d10000640 1 -- 172.21.15.40:0/3788582985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08102980 msgr2=0x7f5d08102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.359+0000 7f5d10000640 1 --2- 172.21.15.40:0/3788582985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08102980 0x7f5d08102da0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc009860 tx=0x7f5cfc02a340 comp rx=0 tx=0).stop 2024-04-26T23:58:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.359+0000 7f5d10000640 1 -- 172.21.15.40:0/3788582985 shutdown_connections 2024-04-26T23:58:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.359+0000 7f5d10000640 1 --2- 172.21.15.40:0/3788582985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08104530 0x7f5d0810adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.359+0000 7f5d10000640 1 --2- 172.21.15.40:0/3788582985 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5d08103b70 0x7f5d08103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.359+0000 7f5d10000640 1 --2- 172.21.15.40:0/3788582985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08102980 0x7f5d08102da0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.359+0000 7f5d10000640 1 -- 172.21.15.40:0/3788582985 >> 172.21.15.40:0/3788582985 conn(0x7f5d080fdf10 msgr2=0x7f5d08100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:47.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.359+0000 7f5d10000640 1 -- 172.21.15.40:0/3788582985 shutdown_connections 2024-04-26T23:58:47.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.359+0000 7f5d10000640 1 -- 172.21.15.40:0/3788582985 wait complete. 2024-04-26T23:58:47.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.360+0000 7f5d10000640 1 Processor -- start 2024-04-26T23:58:47.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.360+0000 7f5d10000640 1 -- start start 2024-04-26T23:58:47.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.360+0000 7f5d10000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08102980 0x7f5d0819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:47.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d10000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5d08103b70 0x7f5d0819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:47.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d10000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08104530 0x7f5d081a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:47.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d10000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0810e480 con 0x7f5d08102980 2024-04-26T23:58:47.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d10000640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0810e300 con 0x7f5d08103b70 2024-04-26T23:58:47.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d10000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0810e600 con 0x7f5d08104530 2024-04-26T23:58:47.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d0d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08102980 0x7f5d0819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:47.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d0d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08102980 0x7f5d0819ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43834/0 (socket says 172.21.15.40:43834) 2024-04-26T23:58:47.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d0d600640 1 -- 172.21.15.40:0/3767348777 learned_addr learned my addr 172.21.15.40:0/3767348777 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:47.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d0cc00640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5d08103b70 0x7f5d0819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d0e000640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08104530 0x7f5d081a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d0d600640 1 -- 172.21.15.40:0/3767348777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08104530 msgr2=0x7f5d081a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d0d600640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08104530 0x7f5d081a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d0d600640 1 -- 172.21.15.40:0/3767348777 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5d08103b70 msgr2=0x7f5d0819b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d0d600640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5d08103b70 0x7f5d0819b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d0d600640 1 -- 172.21.15.40:0/3767348777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d081a2a00 con 0x7f5d08102980 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d0cc00640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5d08103b70 0x7f5d0819b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.361+0000 7f5d0d600640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08102980 0x7f5d0819ade0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc005940 tx=0x7f5cfc032dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.362+0000 7f5d06000640 1 -- 172.21.15.40:0/3767348777 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc040020 con 0x7f5d08102980 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.362+0000 7f5d06000640 1 -- 172.21.15.40:0/3767348777 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cfc030dd0 con 0x7f5d08102980 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.362+0000 7f5d10000640 1 -- 172.21.15.40:0/3767348777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d081a2c90 con 0x7f5d08102980 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.362+0000 7f5d06000640 1 -- 172.21.15.40:0/3767348777 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc037040 con 0x7f5d08102980 2024-04-26T23:58:47.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.362+0000 7f5d10000640 1 -- 172.21.15.40:0/3767348777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d081a3180 con 0x7f5d08102980 2024-04-26T23:58:47.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.363+0000 7f5d10000640 1 -- 172.21.15.40:0/3767348777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d081a3350 con 0x7f5d08102980 2024-04-26T23:58:47.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.366+0000 7f5d06000640 1 -- 172.21.15.40:0/3767348777 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5cfc043020 con 0x7f5d08102980 2024-04-26T23:58:47.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.367+0000 7f5d06000640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5cf00a3bb0 0x7f5cf00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:47.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.367+0000 7f5d06000640 1 -- 172.21.15.40:0/3767348777 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5cfc0efdd0 con 0x7f5d08102980 2024-04-26T23:58:47.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.367+0000 7f5d0cc00640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5cf00a3bb0 0x7f5cf00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:47.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.368+0000 7f5d0cc00640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5cf00a3bb0 0x7f5cf00a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5d0819c2f0 tx=0x7f5cf4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:47.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.372+0000 7f5d06000640 1 -- 172.21.15.40:0/3767348777 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5cfc04a020 con 0x7f5d08102980 2024-04-26T23:58:47.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.577+0000 7f5d10000640 1 -- 172.21.15.40:0/3767348777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5d081a3520 con 0x7f5d08102980 2024-04-26T23:58:47.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.577+0000 7f5d06000640 1 -- 172.21.15.40:0/3767348777 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f5cfc0328c0 con 0x7f5d08102980 2024-04-26T23:58:47.578 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:58:47.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.579+0000 7f5d10000640 1 -- 172.21.15.40:0/3767348777 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5cf00a3bb0 msgr2=0x7f5cf00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:47.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.579+0000 7f5d10000640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5cf00a3bb0 0x7f5cf00a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5d0819c2f0 tx=0x7f5cf4009260 comp rx=0 tx=0).stop 2024-04-26T23:58:47.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.579+0000 7f5d10000640 1 -- 172.21.15.40:0/3767348777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08102980 msgr2=0x7f5d0819ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.579+0000 7f5d10000640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08102980 0x7f5d0819ade0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc005940 tx=0x7f5cfc032dc0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.579+0000 7f5d10000640 1 -- 172.21.15.40:0/3767348777 shutdown_connections 2024-04-26T23:58:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.579+0000 7f5d10000640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5cf00a3bb0 0x7f5cf00a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.579+0000 7f5d10000640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08104530 0x7f5d081a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.579+0000 7f5d10000640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5d08103b70 0x7f5d0819b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.579+0000 7f5d10000640 1 --2- 172.21.15.40:0/3767348777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08102980 0x7f5d0819ade0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.579+0000 7f5d10000640 1 -- 172.21.15.40:0/3767348777 >> 172.21.15.40:0/3767348777 conn(0x7f5d080fdf10 msgr2=0x7f5d08109310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.580+0000 7f5d10000640 1 -- 172.21.15.40:0/3767348777 shutdown_connections 2024-04-26T23:58:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.580+0000 7f5d10000640 1 -- 172.21.15.40:0/3767348777 wait complete. 2024-04-26T23:58:47.597 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T23:58:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.784+0000 7fbaaec00640 1 -- 172.21.15.40:0/2364292115 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbaa80768f0 msgr2=0x7fbaa8074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.784+0000 7fbaaec00640 1 --2- 172.21.15.40:0/2364292115 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbaa80768f0 0x7fbaa8074d40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fba98009900 tx=0x7fba9802a480 comp rx=0 tx=0).stop 2024-04-26T23:58:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.784+0000 7fbaaec00640 1 -- 172.21.15.40:0/2364292115 shutdown_connections 2024-04-26T23:58:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.784+0000 7fbaaec00640 1 --2- 172.21.15.40:0/2364292115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaa810a830 0x7fbaa810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.784+0000 7fbaaec00640 1 --2- 172.21.15.40:0/2364292115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaa8075310 0x7fbaa8075770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.784+0000 7fbaaec00640 1 --2- 172.21.15.40:0/2364292115 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbaa80768f0 0x7fbaa8074d40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.784+0000 7fbaaec00640 1 -- 172.21.15.40:0/2364292115 >> 172.21.15.40:0/2364292115 conn(0x7fbaa80fdf10 msgr2=0x7fbaa8100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.784+0000 7fbaaec00640 1 -- 172.21.15.40:0/2364292115 shutdown_connections 2024-04-26T23:58:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.784+0000 7fbaaec00640 1 -- 172.21.15.40:0/2364292115 wait complete. 2024-04-26T23:58:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.785+0000 7fbaaec00640 1 Processor -- start 2024-04-26T23:58:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.785+0000 7fbaaec00640 1 -- start start 2024-04-26T23:58:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.786+0000 7fbaaec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaa8075310 0x7fbaa819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.786+0000 7fbaaec00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbaa80768f0 0x7fbaa819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.786+0000 7fbaaec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaa810a830 0x7fbaa81a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.786+0000 7fbaaec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa8110700 con 0x7fbaa810a830 2024-04-26T23:58:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.786+0000 7fbaaec00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa8110580 con 0x7fbaa80768f0 2024-04-26T23:58:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.786+0000 7fbaaec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa8110880 con 0x7fbaa8075310 2024-04-26T23:58:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.786+0000 7fbaa7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaa8075310 0x7fbaa819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.786+0000 7fbaa7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaa8075310 0x7fbaa819adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56006/0 (socket says 172.21.15.40:56006) 2024-04-26T23:58:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.786+0000 7fbaacc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaa810a830 0x7fbaa81a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:47.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.786+0000 7fbaa7e00640 1 -- 172.21.15.40:0/109383336 learned_addr learned my addr 172.21.15.40:0/109383336 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:47.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.786+0000 7fbaa7400640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbaa80768f0 0x7fbaa819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.787+0000 7fbaa7e00640 1 -- 172.21.15.40:0/109383336 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbaa80768f0 msgr2=0x7fbaa819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.787+0000 7fbaa7e00640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbaa80768f0 0x7fbaa819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.787+0000 7fbaa7e00640 1 -- 172.21.15.40:0/109383336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaa810a830 msgr2=0x7fbaa81a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.787+0000 7fbaa7e00640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaa810a830 0x7fbaa81a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.787+0000 7fbaa7e00640 1 -- 172.21.15.40:0/109383336 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa81a29c0 con 0x7fbaa8075310 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.787+0000 7fbaacc00640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaa810a830 0x7fbaa81a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.787+0000 7fbaa7400640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbaa80768f0 0x7fbaa819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.787+0000 7fbaa7e00640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaa8075310 0x7fbaa819adb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fba98005ab0 tx=0x7fba98003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.787+0000 7fbaa4c00640 1 -- 172.21.15.40:0/109383336 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9803f070 con 0x7fbaa8075310 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.788+0000 7fbaaec00640 1 -- 172.21.15.40:0/109383336 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa81a2c50 con 0x7fbaa8075310 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.788+0000 7fbaa4c00640 1 -- 172.21.15.40:0/109383336 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba98031c50 con 0x7fbaa8075310 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.788+0000 7fbaa4c00640 1 -- 172.21.15.40:0/109383336 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba98035030 con 0x7fbaa8075310 2024-04-26T23:58:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.788+0000 7fbaaec00640 1 -- 172.21.15.40:0/109383336 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa81a3190 con 0x7fbaa8075310 2024-04-26T23:58:47.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.789+0000 7fbaaec00640 1 -- 172.21.15.40:0/109383336 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa81095f0 con 0x7fbaa8075310 2024-04-26T23:58:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.791+0000 7fbaa4c00640 1 -- 172.21.15.40:0/109383336 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fba98042070 con 0x7fbaa8075310 2024-04-26T23:58:47.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.792+0000 7fbaa4c00640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fba8c0a3bb0 0x7fba8c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:47.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.792+0000 7fbaa4c00640 1 -- 172.21.15.40:0/109383336 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fba980f07b0 con 0x7fbaa8075310 2024-04-26T23:58:47.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.793+0000 7fbaa7400640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fba8c0a3bb0 0x7fba8c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:47.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.793+0000 7fbaa7400640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fba8c0a3bb0 0x7fba8c0a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fba94009fc0 tx=0x7fba94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:47.796+0000 7fbaa4c00640 1 -- 172.21.15.40:0/109383336 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fba980bcf40 con 0x7fbaa8075310 2024-04-26T23:58:48.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.031+0000 7fbaaec00640 1 -- 172.21.15.40:0/109383336 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fbaa8063530 con 0x7fbaa8075310 2024-04-26T23:58:48.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.031+0000 7fbaa4c00640 1 -- 172.21.15.40:0/109383336 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fba980c1240 con 0x7fbaa8075310 2024-04-26T23:58:48.032 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:48.032 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-04-26T23:58:48.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-26T23:58:48.032 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-04-26T23:58:48.032 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-04-26T23:58:48.033 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:58:48.033 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-04-26T23:58:48.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.034+0000 7fbaaec00640 1 -- 172.21.15.40:0/109383336 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fba8c0a3bb0 msgr2=0x7fba8c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:48.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.034+0000 7fbaaec00640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fba8c0a3bb0 0x7fba8c0a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fba94009fc0 tx=0x7fba94005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:48.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.034+0000 7fbaaec00640 1 -- 172.21.15.40:0/109383336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaa8075310 msgr2=0x7fbaa819adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:48.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.034+0000 7fbaaec00640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaa8075310 0x7fbaa819adb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fba98005ab0 tx=0x7fba98003b90 comp rx=0 tx=0).stop 2024-04-26T23:58:48.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.034+0000 7fbaaec00640 1 -- 172.21.15.40:0/109383336 shutdown_connections 2024-04-26T23:58:48.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.034+0000 7fbaaec00640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbaa810a830 0x7fbaa81a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:48.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.034+0000 7fbaaec00640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbaa80768f0 0x7fbaa819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:48.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.034+0000 7fbaaec00640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fba8c0a3bb0 0x7fba8c0a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:48.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.034+0000 7fbaaec00640 1 --2- 172.21.15.40:0/109383336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaa8075310 0x7fbaa819adb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:48.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.034+0000 7fbaaec00640 1 -- 172.21.15.40:0/109383336 >> 172.21.15.40:0/109383336 conn(0x7fbaa80fdf10 msgr2=0x7fbaa81067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:48.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.034+0000 7fbaaec00640 1 -- 172.21.15.40:0/109383336 shutdown_connections 2024-04-26T23:58:48.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.034+0000 7fbaaec00640 1 -- 172.21.15.40:0/109383336 wait complete. 2024-04-26T23:58:48.052 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T23:58:48.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T23:58:48.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:47 smithi040 ceph-mon[28095]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:48.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:47 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:48.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:47 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:48.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:47 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:48.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:47 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:48.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:47 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/58229859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:48.218 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:47 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3767348777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:58:48.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.244+0000 7f1cb7000640 1 -- 172.21.15.40:0/3092285018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cb0102410 msgr2=0x7f1cb0102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:48.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.244+0000 7f1cb7000640 1 --2- 172.21.15.40:0/3092285018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cb0102410 0x7f1cb0102890 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8009ca0 tx=0x7f1ca802a8b0 comp rx=0 tx=0).stop 2024-04-26T23:58:48.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.244+0000 7f1cb7000640 1 -- 172.21.15.40:0/3092285018 shutdown_connections 2024-04-26T23:58:48.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.244+0000 7f1cb7000640 1 --2- 172.21.15.40:0/3092285018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cb0102410 0x7f1cb0102890 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:48.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.244+0000 7f1cb7000640 1 --2- 172.21.15.40:0/3092285018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cb0101a70 0x7f1cb0101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:48.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.244+0000 7f1cb7000640 1 --2- 172.21.15.40:0/3092285018 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1cb0101110 0x7f1cb0101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:48.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.244+0000 7f1cb7000640 1 -- 172.21.15.40:0/3092285018 >> 172.21.15.40:0/3092285018 conn(0x7f1cb00fccd0 msgr2=0x7f1cb00ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:48.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.244+0000 7f1cb7000640 1 -- 172.21.15.40:0/3092285018 shutdown_connections 2024-04-26T23:58:48.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.245+0000 7f1cb7000640 1 -- 172.21.15.40:0/3092285018 wait complete. 2024-04-26T23:58:48.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.245+0000 7f1cb7000640 1 Processor -- start 2024-04-26T23:58:48.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb7000640 1 -- start start 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb7000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cb0101110 0x7f1cb019ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb7000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cb0101a70 0x7f1cb019b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb7000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1cb0102410 0x7f1cb01a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb7000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb0110590 con 0x7f1cb0101a70 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb7000640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb0110410 con 0x7f1cb0102410 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb7000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb0110710 con 0x7f1cb0101110 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cb0101a70 0x7f1cb019b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb5c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cb0101110 0x7f1cb019ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb5c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cb0101110 0x7f1cb019ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56026/0 (socket says 172.21.15.40:56026) 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cb0101a70 0x7f1cb019b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43878/0 (socket says 172.21.15.40:43878) 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb5c00640 1 -- 172.21.15.40:0/1790042469 learned_addr learned my addr 172.21.15.40:0/1790042469 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.246+0000 7f1cb6600640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1cb0102410 0x7f1cb01a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.247+0000 7f1cb5c00640 1 -- 172.21.15.40:0/1790042469 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1cb0102410 msgr2=0x7f1cb01a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:48.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.247+0000 7f1cb5c00640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1cb0102410 0x7f1cb01a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:48.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.247+0000 7f1cb5c00640 1 -- 172.21.15.40:0/1790042469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cb0101a70 msgr2=0x7f1cb019b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:48.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.247+0000 7f1cb5c00640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cb0101a70 0x7f1cb019b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:48.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.247+0000 7f1cb5c00640 1 -- 172.21.15.40:0/1790042469 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb01a2830 con 0x7f1cb0101110 2024-04-26T23:58:48.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.247+0000 7f1cb5200640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cb0101a70 0x7f1cb019b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:48.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.247+0000 7f1cb5c00640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cb0101110 0x7f1cb019ac20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400e9f0 tx=0x7f1ca400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:48.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.247+0000 7f1cae000640 1 -- 172.21.15.40:0/1790042469 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca400cd80 con 0x7f1cb0101110 2024-04-26T23:58:48.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.247+0000 7f1cb7000640 1 -- 172.21.15.40:0/1790042469 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb01a2b20 con 0x7f1cb0101110 2024-04-26T23:58:48.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.247+0000 7f1cae000640 1 -- 172.21.15.40:0/1790042469 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ca40108e0 con 0x7f1cb0101110 2024-04-26T23:58:48.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.247+0000 7f1cae000640 1 -- 172.21.15.40:0/1790042469 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4018980 con 0x7f1cb0101110 2024-04-26T23:58:48.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.247+0000 7f1cb7000640 1 -- 172.21.15.40:0/1790042469 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb01a3070 con 0x7f1cb0101110 2024-04-26T23:58:48.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.248+0000 7f1cb7000640 1 -- 172.21.15.40:0/1790042469 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cb010bff0 con 0x7f1cb0101110 2024-04-26T23:58:48.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.251+0000 7f1cae000640 1 -- 172.21.15.40:0/1790042469 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1ca4010430 con 0x7f1cb0101110 2024-04-26T23:58:48.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.252+0000 7f1cae000640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1c980a3bb0 0x7f1c980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:48.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.252+0000 7f1cae000640 1 -- 172.21.15.40:0/1790042469 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1ca40d0860 con 0x7f1cb0101110 2024-04-26T23:58:48.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.252+0000 7f1cb5200640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1c980a3bb0 0x7f1c980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:48.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.253+0000 7f1cb5200640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1c980a3bb0 0x7f1c980a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c009fc0 tx=0x7f1c9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:48.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.254+0000 7f1cae000640 1 -- 172.21.15.40:0/1790042469 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1ca409cff0 con 0x7f1cb0101110 2024-04-26T23:58:48.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:47 smithi080 ceph-mon[32362]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:48.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:47 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:48.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:47 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:48.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:47 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:48.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:47 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:48.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:47 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/58229859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:48.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:47 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3767348777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:58:48.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:47 smithi103 ceph-mon[32952]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:48.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:47 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:48.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:47 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:48.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:47 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:48.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:47 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:48.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:47 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/58229859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:48.349 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:47 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3767348777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:58:48.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:48.494+0000 7f1cb7000640 1 -- 172.21.15.40:0/1790042469 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f1cb002dad0 con 0x7f1cb0101110 2024-04-26T23:58:49.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.087+0000 7f1cae000640 1 -- 172.21.15.40:0/1790042469 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f1ca40a12f0 con 0x7f1cb0101110 2024-04-26T23:58:49.089 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-04-26T23:58:49.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.095+0000 7f1cb7000640 1 -- 172.21.15.40:0/1790042469 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1c980a3bb0 msgr2=0x7f1c980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:49.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.095+0000 7f1cb7000640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1c980a3bb0 0x7f1c980a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c009fc0 tx=0x7f1c9c005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:49.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.095+0000 7f1cb7000640 1 -- 172.21.15.40:0/1790042469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cb0101110 msgr2=0x7f1cb019ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:49.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.095+0000 7f1cb7000640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cb0101110 0x7f1cb019ac20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400e9f0 tx=0x7f1ca400eeb0 comp rx=0 tx=0).stop 2024-04-26T23:58:49.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.096+0000 7f1cb7000640 1 -- 172.21.15.40:0/1790042469 shutdown_connections 2024-04-26T23:58:49.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.096+0000 7f1cb7000640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1cb0102410 0x7f1cb01a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:49.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.096+0000 7f1cb7000640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cb0101a70 0x7f1cb019b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:49.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.096+0000 7f1cb7000640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1c980a3bb0 0x7f1c980a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:49.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.096+0000 7f1cb7000640 1 --2- 172.21.15.40:0/1790042469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1cb0101110 0x7f1cb019ac20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:49.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.096+0000 7f1cb7000640 1 -- 172.21.15.40:0/1790042469 >> 172.21.15.40:0/1790042469 conn(0x7f1cb00fccd0 msgr2=0x7f1cb00fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:49.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.096+0000 7f1cb7000640 1 -- 172.21.15.40:0/1790042469 shutdown_connections 2024-04-26T23:58:49.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.096+0000 7f1cb7000640 1 -- 172.21.15.40:0/1790042469 wait complete. 2024-04-26T23:58:49.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-26T23:58:49.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:48 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/109383336' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T23:58:49.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:49.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:49.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:49.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:58:49.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:49.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:58:49.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:48 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1790042469' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T23:58:49.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:48 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T23:58:49.259 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:48 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/109383336' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T23:58:49.260 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:49.260 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:49.260 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:49.260 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:58:49.260 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:49.260 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:58:49.260 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:48 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1790042469' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T23:58:49.260 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:48 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T23:58:49.296 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:48 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/109383336' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T23:58:49.296 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:49.296 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:49.296 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:49.296 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:58:49.296 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:49.296 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:58:49.296 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:48 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1790042469' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T23:58:49.296 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:48 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T23:58:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.299+0000 7fade1800640 1 -- 172.21.15.40:0/2287167770 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faddc078530 msgr2=0x7faddc078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.299+0000 7fade1800640 1 --2- 172.21.15.40:0/2287167770 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faddc078530 0x7faddc078990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009900 tx=0x7fadc802a410 comp rx=0 tx=0).stop 2024-04-26T23:58:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.299+0000 7fade1800640 1 -- 172.21.15.40:0/2287167770 shutdown_connections 2024-04-26T23:58:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.299+0000 7fade1800640 1 --2- 172.21.15.40:0/2287167770 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc078ed0 0x7faddc079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.299+0000 7fade1800640 1 --2- 172.21.15.40:0/2287167770 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faddc078530 0x7faddc078990 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:49.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.299+0000 7fade1800640 1 --2- 172.21.15.40:0/2287167770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc0772f0 0x7faddc077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:49.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.299+0000 7fade1800640 1 -- 172.21.15.40:0/2287167770 >> 172.21.15.40:0/2287167770 conn(0x7faddc100120 msgr2=0x7faddc102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:49.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.299+0000 7fade1800640 1 -- 172.21.15.40:0/2287167770 shutdown_connections 2024-04-26T23:58:49.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.299+0000 7fade1800640 1 -- 172.21.15.40:0/2287167770 wait complete. 2024-04-26T23:58:49.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.301+0000 7fade1800640 1 Processor -- start 2024-04-26T23:58:49.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.301+0000 7fade1800640 1 -- start start 2024-04-26T23:58:49.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.301+0000 7fade1800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc0772f0 0x7faddc19f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.301+0000 7fade1800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc078530 0x7faddc19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.301+0000 7fade1800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faddc078ed0 0x7faddc1a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.301+0000 7fade1800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faddc114af0 con 0x7faddc0772f0 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.301+0000 7fade1800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7faddc114970 con 0x7faddc078ed0 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.301+0000 7fade1800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faddc114c70 con 0x7faddc078530 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.301+0000 7faddaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc0772f0 0x7faddc19f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.301+0000 7fadda000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc078530 0x7faddc19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.301+0000 7fadda000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc078530 0x7faddc19f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56038/0 (socket says 172.21.15.40:56038) 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.301+0000 7fadda000640 1 -- 172.21.15.40:0/3072760957 learned_addr learned my addr 172.21.15.40:0/3072760957 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.302+0000 7faddb400640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faddc078ed0 0x7faddc1a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.302+0000 7fadda000640 1 -- 172.21.15.40:0/3072760957 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faddc078ed0 msgr2=0x7faddc1a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.302+0000 7fadda000640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faddc078ed0 0x7faddc1a67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:49.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.302+0000 7fadda000640 1 -- 172.21.15.40:0/3072760957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc0772f0 msgr2=0x7faddc19f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:49.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.302+0000 7fadda000640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc0772f0 0x7faddc19f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:49.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.302+0000 7fadda000640 1 -- 172.21.15.40:0/3072760957 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faddc1a6e10 con 0x7faddc078530 2024-04-26T23:58:49.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.302+0000 7fadda000640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc078530 0x7faddc19f730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fadc8005ab0 tx=0x7fadc8003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:49.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.303+0000 7fadd3400640 1 -- 172.21.15.40:0/3072760957 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc803f070 con 0x7faddc078530 2024-04-26T23:58:49.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.303+0000 7fadd3400640 1 -- 172.21.15.40:0/3072760957 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadc8031c50 con 0x7faddc078530 2024-04-26T23:58:49.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.303+0000 7fadd3400640 1 -- 172.21.15.40:0/3072760957 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc8035030 con 0x7faddc078530 2024-04-26T23:58:49.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.303+0000 7fade1800640 1 -- 172.21.15.40:0/3072760957 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faddc1a70a0 con 0x7faddc078530 2024-04-26T23:58:49.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.303+0000 7fade1800640 1 -- 172.21.15.40:0/3072760957 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faddc1a7540 con 0x7faddc078530 2024-04-26T23:58:49.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.304+0000 7fade1800640 1 -- 172.21.15.40:0/3072760957 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faddc111520 con 0x7faddc078530 2024-04-26T23:58:49.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.307+0000 7fadd3400640 1 -- 172.21.15.40:0/3072760957 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fadc8042070 con 0x7faddc078530 2024-04-26T23:58:49.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.309+0000 7fadd3400640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fadc00a3c30 0x7fadc00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:49.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.309+0000 7fadd3400640 1 -- 172.21.15.40:0/3072760957 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7fadc80f07d0 con 0x7faddc078530 2024-04-26T23:58:49.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.309+0000 7faddaa00640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fadc00a3c30 0x7fadc00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:49.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.310+0000 7faddaa00640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fadc00a3c30 0x7fadc00a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009bd0 tx=0x7fadcc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:49.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.313+0000 7fadd3400640 1 -- 172.21.15.40:0/3072760957 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fadc80bcdf0 con 0x7faddc078530 2024-04-26T23:58:49.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:49.524+0000 7fade1800640 1 -- 172.21.15.40:0/3072760957 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7faddc063560 con 0x7faddc078530 2024-04-26T23:58:50.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.092+0000 7fadd3400640 1 -- 172.21.15.40:0/3072760957 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) ==== 171+0+0 (secure 0 0 0) 0x7fadc80c10f0 con 0x7faddc078530 2024-04-26T23:58:50.094 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-04-26T23:58:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.100+0000 7fade1800640 1 -- 172.21.15.40:0/3072760957 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fadc00a3c30 msgr2=0x7fadc00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.100+0000 7fade1800640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fadc00a3c30 0x7fadc00a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009bd0 tx=0x7fadcc005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.100+0000 7fade1800640 1 -- 172.21.15.40:0/3072760957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc078530 msgr2=0x7faddc19f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.100+0000 7fade1800640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc078530 0x7faddc19f730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fadc8005ab0 tx=0x7fadc8003b90 comp rx=0 tx=0).stop 2024-04-26T23:58:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.101+0000 7fade1800640 1 -- 172.21.15.40:0/3072760957 shutdown_connections 2024-04-26T23:58:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.101+0000 7fade1800640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faddc078ed0 0x7faddc1a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.101+0000 7fade1800640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fadc00a3c30 0x7fadc00a60f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.101+0000 7fade1800640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc078530 0x7faddc19f730 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.101+0000 7fade1800640 1 --2- 172.21.15.40:0/3072760957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc0772f0 0x7faddc19f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.101+0000 7fade1800640 1 -- 172.21.15.40:0/3072760957 >> 172.21.15.40:0/3072760957 conn(0x7faddc100120 msgr2=0x7faddc101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.101+0000 7fade1800640 1 -- 172.21.15.40:0/3072760957 shutdown_connections 2024-04-26T23:58:50.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.101+0000 7fade1800640 1 -- 172.21.15.40:0/3072760957 wait complete. 2024-04-26T23:58:50.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T23:58:50.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.326+0000 7fe427e00640 1 -- 172.21.15.40:0/2857100806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe428101a90 msgr2=0x7fe428101ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:50.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.326+0000 7fe427e00640 1 --2- 172.21.15.40:0/2857100806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe428101a90 0x7fe428101ef0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe418009900 tx=0x7fe41802a480 comp rx=0 tx=0).stop 2024-04-26T23:58:50.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.326+0000 7fe427e00640 1 -- 172.21.15.40:0/2857100806 shutdown_connections 2024-04-26T23:58:50.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.326+0000 7fe427e00640 1 --2- 172.21.15.40:0/2857100806 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe428102430 0x7fe4281028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:50.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.326+0000 7fe427e00640 1 --2- 172.21.15.40:0/2857100806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe428101a90 0x7fe428101ef0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:50.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.326+0000 7fe427e00640 1 --2- 172.21.15.40:0/2857100806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe428101130 0x7fe428101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:50.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.326+0000 7fe427e00640 1 -- 172.21.15.40:0/2857100806 >> 172.21.15.40:0/2857100806 conn(0x7fe4280fccd0 msgr2=0x7fe4280ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:50.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.326+0000 7fe427e00640 1 -- 172.21.15.40:0/2857100806 shutdown_connections 2024-04-26T23:58:50.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.326+0000 7fe427e00640 1 -- 172.21.15.40:0/2857100806 wait complete. 2024-04-26T23:58:50.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.327+0000 7fe427e00640 1 Processor -- start 2024-04-26T23:58:50.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.327+0000 7fe427e00640 1 -- start start 2024-04-26T23:58:50.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe427e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe428101130 0x7fe42819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:50.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe427e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe428101a90 0x7fe42819b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe427e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe428102430 0x7fe4281a21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe427e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe428110550 con 0x7fe428101130 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe427e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fe4281103d0 con 0x7fe428102430 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe427e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4281106d0 con 0x7fe428101a90 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe426a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe428101130 0x7fe42819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe427400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe428102430 0x7fe4281a21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe426a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe428101130 0x7fe42819ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52824/0 (socket says 172.21.15.40:52824) 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe426a00640 1 -- 172.21.15.40:0/3853666189 learned_addr learned my addr 172.21.15.40:0/3853666189 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe426000640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe428101a90 0x7fe42819b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe427400640 1 -- 172.21.15.40:0/3853666189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe428101a90 msgr2=0x7fe42819b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe427400640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe428101a90 0x7fe42819b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe427400640 1 -- 172.21.15.40:0/3853666189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe428101130 msgr2=0x7fe42819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe427400640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe428101130 0x7fe42819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.328+0000 7fe427400640 1 -- 172.21.15.40:0/3853666189 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4281a2890 con 0x7fe428102430 2024-04-26T23:58:50.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.329+0000 7fe427400640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe428102430 0x7fe4281a21f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe420009fd0 tx=0x7fe42000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:50.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.329+0000 7fe41f400640 1 -- 172.21.15.40:0/3853666189 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42000ce20 con 0x7fe428102430 2024-04-26T23:58:50.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.329+0000 7fe41f400640 1 -- 172.21.15.40:0/3853666189 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe420005020 con 0x7fe428102430 2024-04-26T23:58:50.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.329+0000 7fe427e00640 1 -- 172.21.15.40:0/3853666189 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4281a2b80 con 0x7fe428102430 2024-04-26T23:58:50.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.329+0000 7fe41f400640 1 -- 172.21.15.40:0/3853666189 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420010690 con 0x7fe428102430 2024-04-26T23:58:50.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.329+0000 7fe427e00640 1 -- 172.21.15.40:0/3853666189 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4281a30d0 con 0x7fe428102430 2024-04-26T23:58:50.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.330+0000 7fe427e00640 1 -- 172.21.15.40:0/3853666189 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe42810be80 con 0x7fe428102430 2024-04-26T23:58:50.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.332+0000 7fe41f400640 1 -- 172.21.15.40:0/3853666189 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe4200054d0 con 0x7fe428102430 2024-04-26T23:58:50.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.333+0000 7fe41f400640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe40c0a3c30 0x7fe40c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:50.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.333+0000 7fe41f400640 1 -- 172.21.15.40:0/3853666189 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7fe4200d09c0 con 0x7fe428102430 2024-04-26T23:58:50.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.334+0000 7fe426a00640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe40c0a3c30 0x7fe40c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:50.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.334+0000 7fe426a00640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe40c0a3c30 0x7fe40c0a60f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe42819c150 tx=0x7fe414005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:50.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.337+0000 7fe41f400640 1 -- 172.21.15.40:0/3853666189 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe42009d4b0 con 0x7fe428102430 2024-04-26T23:58:50.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:50 smithi040 ceph-mon[28095]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:50.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:50 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T23:58:50.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:50 smithi040 ceph-mon[28095]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T23:58:50.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:50 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3072760957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T23:58:50.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:50 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T23:58:50.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:50 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:50 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:50 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:50 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:50 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:50 smithi080 ceph-mon[32362]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:50.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:50 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T23:58:50.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:50 smithi080 ceph-mon[32362]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T23:58:50.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:50 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3072760957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T23:58:50.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:50 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T23:58:50.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:50 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:50 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:50 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:50 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:50 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:50 smithi103 ceph-mon[32952]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:50.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:50 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T23:58:50.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:50 smithi103 ceph-mon[32952]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T23:58:50.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:50 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3072760957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T23:58:50.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:50 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T23:58:50.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:50 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:50 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:50 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:50 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:50 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:50.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:50.615+0000 7fe427e00640 1 -- 172.21.15.40:0/3853666189 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe42802dad0 con 0x7fe428102430 2024-04-26T23:58:51.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.121+0000 7fe41f400640 1 -- 172.21.15.40:0/3853666189 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fe42009cdb0 con 0x7fe428102430 2024-04-26T23:58:51.123 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T23:58:51.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.128+0000 7fe427e00640 1 -- 172.21.15.40:0/3853666189 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe40c0a3c30 msgr2=0x7fe40c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:51.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.128+0000 7fe427e00640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe40c0a3c30 0x7fe40c0a60f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe42819c150 tx=0x7fe414005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:51.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.128+0000 7fe427e00640 1 -- 172.21.15.40:0/3853666189 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe428102430 msgr2=0x7fe4281a21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:51.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.128+0000 7fe427e00640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe428102430 0x7fe4281a21f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe420009fd0 tx=0x7fe42000ef10 comp rx=0 tx=0).stop 2024-04-26T23:58:51.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.134+0000 7fe427e00640 1 -- 172.21.15.40:0/3853666189 shutdown_connections 2024-04-26T23:58:51.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.134+0000 7fe427e00640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe428102430 0x7fe4281a21f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:51.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.134+0000 7fe427e00640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe40c0a3c30 0x7fe40c0a60f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:51.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.134+0000 7fe427e00640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe428101a90 0x7fe42819b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:51.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.134+0000 7fe427e00640 1 --2- 172.21.15.40:0/3853666189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe428101130 0x7fe42819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:51.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.134+0000 7fe427e00640 1 -- 172.21.15.40:0/3853666189 >> 172.21.15.40:0/3853666189 conn(0x7fe4280fccd0 msgr2=0x7fe4280fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:51.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.138+0000 7fe427e00640 1 -- 172.21.15.40:0/3853666189 shutdown_connections 2024-04-26T23:58:51.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.138+0000 7fe427e00640 1 -- 172.21.15.40:0/3853666189 wait complete. 2024-04-26T23:58:51.174 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-26T23:58:51.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-26T23:58:51.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.363+0000 7f134e000640 1 -- 172.21.15.40:0/704579133 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1348104b90 msgr2=0x7f1348104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:51.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.363+0000 7f134e000640 1 --2- 172.21.15.40:0/704579133 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1348104b90 0x7f1348104fb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1338009900 tx=0x7f133802a3f0 comp rx=0 tx=0).stop 2024-04-26T23:58:51.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.363+0000 7f134e000640 1 -- 172.21.15.40:0/704579133 shutdown_connections 2024-04-26T23:58:51.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.363+0000 7f134e000640 1 --2- 172.21.15.40:0/704579133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1348106740 0x7f134810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:51.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.363+0000 7f134e000640 1 --2- 172.21.15.40:0/704579133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348105d80 0x7f1348106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:51.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.363+0000 7f134e000640 1 --2- 172.21.15.40:0/704579133 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1348104b90 0x7f1348104fb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:51.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.363+0000 7f134e000640 1 -- 172.21.15.40:0/704579133 >> 172.21.15.40:0/704579133 conn(0x7f1348100120 msgr2=0x7f1348102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:51.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.364+0000 7f134e000640 1 -- 172.21.15.40:0/704579133 shutdown_connections 2024-04-26T23:58:51.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.364+0000 7f134e000640 1 -- 172.21.15.40:0/704579133 wait complete. 2024-04-26T23:58:51.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.365+0000 7f134e000640 1 Processor -- start 2024-04-26T23:58:51.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.365+0000 7f134e000640 1 -- start start 2024-04-26T23:58:51.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.365+0000 7f134e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348104b90 0x7f134819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:51.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.365+0000 7f134e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1348105d80 0x7f134819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.365+0000 7f134e000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1348106740 0x7f13481a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.365+0000 7f134e000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1348110710 con 0x7f1348104b90 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.365+0000 7f134e000640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1348110590 con 0x7f1348106740 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.365+0000 7f134e000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1348110890 con 0x7f1348105d80 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.366+0000 7f1347400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348104b90 0x7f134819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.366+0000 7f133ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1348105d80 0x7f134819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.366+0000 7f1347e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1348106740 0x7f13481a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.366+0000 7f1347400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348104b90 0x7f134819adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52848/0 (socket says 172.21.15.40:52848) 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.366+0000 7f1347400640 1 -- 172.21.15.40:0/52260221 learned_addr learned my addr 172.21.15.40:0/52260221 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.366+0000 7f1347e00640 1 -- 172.21.15.40:0/52260221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1348105d80 msgr2=0x7f134819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.366+0000 7f1347e00640 1 --2- 172.21.15.40:0/52260221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1348105d80 0x7f134819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.366+0000 7f1347e00640 1 -- 172.21.15.40:0/52260221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348104b90 msgr2=0x7f134819adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:51.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.366+0000 7f1347e00640 1 --2- 172.21.15.40:0/52260221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348104b90 0x7f134819adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:51.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.366+0000 7f1347e00640 1 -- 172.21.15.40:0/52260221 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13481a29c0 con 0x7f1348106740 2024-04-26T23:58:51.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.366+0000 7f1347e00640 1 --2- 172.21.15.40:0/52260221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1348106740 0x7f13481a2320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1340009fd0 tx=0x7f134000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:51.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.367+0000 7f1344c00640 1 -- 172.21.15.40:0/52260221 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134000ce20 con 0x7f1348106740 2024-04-26T23:58:51.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.367+0000 7f134e000640 1 -- 172.21.15.40:0/52260221 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13481a2cb0 con 0x7f1348106740 2024-04-26T23:58:51.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.367+0000 7f1344c00640 1 -- 172.21.15.40:0/52260221 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1340005020 con 0x7f1348106740 2024-04-26T23:58:51.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.367+0000 7f1344c00640 1 -- 172.21.15.40:0/52260221 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1340010690 con 0x7f1348106740 2024-04-26T23:58:51.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.367+0000 7f134e000640 1 -- 172.21.15.40:0/52260221 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13481a31a0 con 0x7f1348106740 2024-04-26T23:58:51.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.368+0000 7f134e000640 1 -- 172.21.15.40:0/52260221 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f134810bf30 con 0x7f1348106740 2024-04-26T23:58:51.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.370+0000 7f1344c00640 1 -- 172.21.15.40:0/52260221 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f13400054d0 con 0x7f1348106740 2024-04-26T23:58:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.371+0000 7f1344c00640 1 --2- 172.21.15.40:0/52260221 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f132c0a3bb0 0x7f132c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.371+0000 7f1344c00640 1 -- 172.21.15.40:0/52260221 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f13400d0ab0 con 0x7f1348106740 2024-04-26T23:58:51.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.372+0000 7f1347400640 1 --2- 172.21.15.40:0/52260221 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f132c0a3bb0 0x7f132c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:51.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.372+0000 7f1347400640 1 --2- 172.21.15.40:0/52260221 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f132c0a3bb0 0x7f132c0a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1338006010 tx=0x7f1338005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:51.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.375+0000 7f1344c00640 1 -- 172.21.15.40:0/52260221 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f134009d540 con 0x7f1348106740 2024-04-26T23:58:51.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:51 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T23:58:51.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:51 smithi040 ceph-mon[28095]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T23:58:51.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:51 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3853666189' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T23:58:51.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:51 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T23:58:51.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:51 smithi040 ceph-mon[28095]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T23:58:51.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:51 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T23:58:51.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:51 smithi080 ceph-mon[32362]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T23:58:51.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:51 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3853666189' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T23:58:51.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:51 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T23:58:51.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:51 smithi080 ceph-mon[32362]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T23:58:51.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:51 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T23:58:51.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:51 smithi103 ceph-mon[32952]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T23:58:51.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:51 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3853666189' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T23:58:51.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:51 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T23:58:51.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:51 smithi103 ceph-mon[32952]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T23:58:51.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:51.600+0000 7f134e000640 1 -- 172.21.15.40:0/52260221 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f134802dad0 con 0x7f1348106740 2024-04-26T23:58:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.156+0000 7f1344c00640 1 -- 172.21.15.40:0/52260221 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) ==== 157+0+0 (secure 0 0 0) 0x7f134009ce40 con 0x7f1348106740 2024-04-26T23:58:52.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.160+0000 7f134e000640 1 -- 172.21.15.40:0/52260221 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f132c0a3bb0 msgr2=0x7f132c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.160+0000 7f134e000640 1 --2- 172.21.15.40:0/52260221 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f132c0a3bb0 0x7f132c0a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1338006010 tx=0x7f1338005a20 comp rx=0 tx=0).stop 2024-04-26T23:58:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.160+0000 7f134e000640 1 -- 172.21.15.40:0/52260221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1348106740 msgr2=0x7f13481a2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.160+0000 7f134e000640 1 --2- 172.21.15.40:0/52260221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1348106740 0x7f13481a2320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1340009fd0 tx=0x7f134000ef10 comp rx=0 tx=0).stop 2024-04-26T23:58:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.160+0000 7f134e000640 1 -- 172.21.15.40:0/52260221 shutdown_connections 2024-04-26T23:58:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.160+0000 7f134e000640 1 --2- 172.21.15.40:0/52260221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1348106740 0x7f13481a2320 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.160+0000 7f134e000640 1 --2- 172.21.15.40:0/52260221 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f132c0a3bb0 0x7f132c0a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.160+0000 7f134e000640 1 --2- 172.21.15.40:0/52260221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1348105d80 0x7f134819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.160+0000 7f134e000640 1 --2- 172.21.15.40:0/52260221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348104b90 0x7f134819adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:52.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.160+0000 7f134e000640 1 -- 172.21.15.40:0/52260221 >> 172.21.15.40:0/52260221 conn(0x7f1348100120 msgr2=0x7f1348109100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:52.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.160+0000 7f134e000640 1 -- 172.21.15.40:0/52260221 shutdown_connections 2024-04-26T23:58:52.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.160+0000 7f134e000640 1 -- 172.21.15.40:0/52260221 wait complete. 2024-04-26T23:58:52.185 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T23:58:52.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.415+0000 7f84c6800640 1 -- 172.21.15.40:0/3927859206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84c0101060 msgr2=0x7f84c0101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.415+0000 7f84c6800640 1 --2- 172.21.15.40:0/3927859206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84c0101060 0x7f84c0101480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009900 tx=0x7f84b402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:58:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.415+0000 7f84c6800640 1 -- 172.21.15.40:0/3927859206 shutdown_connections 2024-04-26T23:58:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.415+0000 7f84c6800640 1 --2- 172.21.15.40:0/3927859206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c0102360 0x7f84c01027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.415+0000 7f84c6800640 1 --2- 172.21.15.40:0/3927859206 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f84c01019c0 0x7f84c0101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.415+0000 7f84c6800640 1 --2- 172.21.15.40:0/3927859206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84c0101060 0x7f84c0101480 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.415+0000 7f84c6800640 1 -- 172.21.15.40:0/3927859206 >> 172.21.15.40:0/3927859206 conn(0x7f84c00fcc00 msgr2=0x7f84c00ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.416+0000 7f84c6800640 1 -- 172.21.15.40:0/3927859206 shutdown_connections 2024-04-26T23:58:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.416+0000 7f84c6800640 1 -- 172.21.15.40:0/3927859206 wait complete. 2024-04-26T23:58:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.417+0000 7f84c6800640 1 Processor -- start 2024-04-26T23:58:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.417+0000 7f84c6800640 1 -- start start 2024-04-26T23:58:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.417+0000 7f84c6800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f84c0101060 0x7f84c019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.417+0000 7f84c6800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84c01019c0 0x7f84c019b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.417+0000 7f84c6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c0102360 0x7f84c01a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.417+0000 7f84c6800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f84c01104a0 con 0x7f84c0102360 2024-04-26T23:58:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.417+0000 7f84c6800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f84c0110320 con 0x7f84c0101060 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.417+0000 7f84c6800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f84c0110620 con 0x7f84c01019c0 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.417+0000 7f84c5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f84c0101060 0x7f84c019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.417+0000 7f84c5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f84c0101060 0x7f84c019abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:40622/0 (socket says 172.21.15.40:40622) 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.417+0000 7f84c5400640 1 -- 172.21.15.40:0/1855288653 learned_addr learned my addr 172.21.15.40:0/1855288653 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.418+0000 7f84c4a00640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84c01019c0 0x7f84c019b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.417+0000 7f84c5e00640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c0102360 0x7f84c01a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.418+0000 7f84c5e00640 1 -- 172.21.15.40:0/1855288653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84c01019c0 msgr2=0x7f84c019b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.418+0000 7f84c5e00640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84c01019c0 0x7f84c019b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.418+0000 7f84c5e00640 1 -- 172.21.15.40:0/1855288653 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f84c0101060 msgr2=0x7f84c019abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.418+0000 7f84c5e00640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f84c0101060 0x7f84c019abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.418+0000 7f84c5e00640 1 -- 172.21.15.40:0/1855288653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c01a2800 con 0x7f84c0102360 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.418+0000 7f84c4a00640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84c01019c0 0x7f84c019b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.418+0000 7f84c5400640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f84c0101060 0x7f84c019abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.418+0000 7f84c5e00640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c0102360 0x7f84c01a2190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009fd0 tx=0x7f84b800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.419+0000 7f84be000640 1 -- 172.21.15.40:0/1855288653 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b800ce20 con 0x7f84c0102360 2024-04-26T23:58:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.419+0000 7f84be000640 1 -- 172.21.15.40:0/1855288653 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84b8005020 con 0x7f84c0102360 2024-04-26T23:58:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.419+0000 7f84c6800640 1 -- 172.21.15.40:0/1855288653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c01a2af0 con 0x7f84c0102360 2024-04-26T23:58:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.419+0000 7f84be000640 1 -- 172.21.15.40:0/1855288653 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b8010690 con 0x7f84c0102360 2024-04-26T23:58:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.419+0000 7f84c6800640 1 -- 172.21.15.40:0/1855288653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c01a2ff0 con 0x7f84c0102360 2024-04-26T23:58:52.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.420+0000 7f84c6800640 1 -- 172.21.15.40:0/1855288653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84c010bdb0 con 0x7f84c0102360 2024-04-26T23:58:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.423+0000 7f84be000640 1 -- 172.21.15.40:0/1855288653 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f84b80054d0 con 0x7f84c0102360 2024-04-26T23:58:52.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.424+0000 7f84be000640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f84a80a3bb0 0x7f84a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:52.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.424+0000 7f84be000640 1 -- 172.21.15.40:0/1855288653 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8277+0+0 (secure 0 0 0) 0x7f84b80d0b30 con 0x7f84c0102360 2024-04-26T23:58:52.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.424+0000 7f84c5400640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f84a80a3bb0 0x7f84a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:52.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.425+0000 7f84c5400640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f84a80a3bb0 0x7f84a80a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f84b40096c0 tx=0x7f84b40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.427+0000 7f84be000640 1 -- 172.21.15.40:0/1855288653 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f84b809d480 con 0x7f84c0102360 2024-04-26T23:58:52.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: pgmap v317: 97 pgs: 10 creating+peering, 6 active+clean, 81 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T23:58:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T23:58:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T23:58:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T23:58:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T23:58:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:58:52.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:52.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/52260221' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T23:58:52.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T23:58:52.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] up:active 2024-04-26T23:58:52.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:52 smithi040 ceph-mon[28095]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:58:52.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: pgmap v317: 97 pgs: 10 creating+peering, 6 active+clean, 81 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:52.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T23:58:52.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T23:58:52.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T23:58:52.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T23:58:52.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T23:58:52.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:52.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:52.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:52.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:58:52.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:52.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/52260221' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T23:58:52.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T23:58:52.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] up:active 2024-04-26T23:58:52.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:52 smithi080 ceph-mon[32362]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:58:52.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: pgmap v317: 97 pgs: 10 creating+peering, 6 active+clean, 81 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:58:52.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T23:58:52.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T23:58:52.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T23:58:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T23:58:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T23:58:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:58:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:58:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/52260221' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T23:58:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T23:58:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] up:active 2024-04-26T23:58:52.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:52 smithi103 ceph-mon[32952]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:58:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:52.634+0000 7f84c6800640 1 -- 172.21.15.40:0/1855288653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f84c002dad0 con 0x7f84c0102360 2024-04-26T23:58:53.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:53 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T23:58:53.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:53 smithi040 ceph-mon[28095]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T23:58:53.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:53 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1855288653' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T23:58:53.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:53 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T23:58:53.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:53 smithi080 ceph-mon[32362]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T23:58:53.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:53 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1855288653' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T23:58:53.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:53 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T23:58:53.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:53 smithi103 ceph-mon[32952]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T23:58:53.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:53 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1855288653' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T23:58:54.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.160+0000 7f84be000640 1 -- 172.21.15.40:0/1855288653 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) ==== 228+0+0 (secure 0 0 0) 0x7f84b809cd80 con 0x7f84c0102360 2024-04-26T23:58:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data_ec' created 2024-04-26T23:58:54.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.167+0000 7f84c6800640 1 -- 172.21.15.40:0/1855288653 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f84a80a3bb0 msgr2=0x7f84a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:54.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.167+0000 7f84c6800640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f84a80a3bb0 0x7f84a80a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f84b40096c0 tx=0x7f84b40314a0 comp rx=0 tx=0).stop 2024-04-26T23:58:54.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.167+0000 7f84c6800640 1 -- 172.21.15.40:0/1855288653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c0102360 msgr2=0x7f84c01a2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:54.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.167+0000 7f84c6800640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c0102360 0x7f84c01a2190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009fd0 tx=0x7f84b800ef10 comp rx=0 tx=0).stop 2024-04-26T23:58:54.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.168+0000 7f84c6800640 1 -- 172.21.15.40:0/1855288653 shutdown_connections 2024-04-26T23:58:54.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.168+0000 7f84c6800640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c0102360 0x7f84c01a2190 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:54.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.168+0000 7f84c6800640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f84a80a3bb0 0x7f84a80a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:54.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.168+0000 7f84c6800640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84c01019c0 0x7f84c019b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:54.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.168+0000 7f84c6800640 1 --2- 172.21.15.40:0/1855288653 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f84c0101060 0x7f84c019abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:54.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.168+0000 7f84c6800640 1 -- 172.21.15.40:0/1855288653 >> 172.21.15.40:0/1855288653 conn(0x7f84c00fcc00 msgr2=0x7f84c00fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:54.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.170+0000 7f84c6800640 1 -- 172.21.15.40:0/1855288653 shutdown_connections 2024-04-26T23:58:54.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.171+0000 7f84c6800640 1 -- 172.21.15.40:0/1855288653 wait complete. 2024-04-26T23:58:54.204 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-26T23:58:54.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.391+0000 7f1208e00640 1 -- 172.21.15.40:0/2282924188 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1204103a50 msgr2=0x7f1204103ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:54.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.391+0000 7f1208e00640 1 --2- 172.21.15.40:0/2282924188 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1204103a50 0x7f1204103ed0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009900 tx=0x7f11ec02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:58:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.391+0000 7f1208e00640 1 -- 172.21.15.40:0/2282924188 shutdown_connections 2024-04-26T23:58:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.391+0000 7f1208e00640 1 --2- 172.21.15.40:0/2282924188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1204104410 0x7f120410aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.391+0000 7f1208e00640 1 --2- 172.21.15.40:0/2282924188 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1204103a50 0x7f1204103ed0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.391+0000 7f1208e00640 1 --2- 172.21.15.40:0/2282924188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1204102860 0x7f1204102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.391+0000 7f1208e00640 1 -- 172.21.15.40:0/2282924188 >> 172.21.15.40:0/2282924188 conn(0x7f12040fddd0 msgr2=0x7f1204100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.391+0000 7f1208e00640 1 -- 172.21.15.40:0/2282924188 shutdown_connections 2024-04-26T23:58:54.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.391+0000 7f1208e00640 1 -- 172.21.15.40:0/2282924188 wait complete. 2024-04-26T23:58:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.392+0000 7f1208e00640 1 Processor -- start 2024-04-26T23:58:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.392+0000 7f1208e00640 1 -- start start 2024-04-26T23:58:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.393+0000 7f1208e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1204102860 0x7f1204076690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.393+0000 7f1208e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1204103a50 0x7f1204074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.393+0000 7f1208e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1204104410 0x7f1204075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.393+0000 7f1208e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f120410e390 con 0x7f1204102860 2024-04-26T23:58:54.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.393+0000 7f1208e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f120410e210 con 0x7f1204104410 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.393+0000 7f1208e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f120410e510 con 0x7f1204103a50 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.393+0000 7f1203400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1204102860 0x7f1204076690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.393+0000 7f1203400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1204102860 0x7f1204076690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52860/0 (socket says 172.21.15.40:52860) 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.393+0000 7f1203400640 1 -- 172.21.15.40:0/186707982 learned_addr learned my addr 172.21.15.40:0/186707982 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.393+0000 7f1202a00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1204103a50 0x7f1204074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.393+0000 7f1203e00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1204104410 0x7f1204075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f1203400640 1 -- 172.21.15.40:0/186707982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1204103a50 msgr2=0x7f1204074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f1203400640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1204103a50 0x7f1204074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f1203400640 1 -- 172.21.15.40:0/186707982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1204104410 msgr2=0x7f1204075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f1203400640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1204104410 0x7f1204075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f1203400640 1 -- 172.21.15.40:0/186707982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1204075b20 con 0x7f1204102860 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f1202a00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1204103a50 0x7f1204074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:54.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f1203e00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1204104410 0x7f1204075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:58:54.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f1203400640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1204102860 0x7f1204076690 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f11f400eb70 tx=0x7f11f400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:54.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f11fbe00640 1 -- 172.21.15.40:0/186707982 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f4003b70 con 0x7f1204102860 2024-04-26T23:58:54.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f1208e00640 1 -- 172.21.15.40:0/186707982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12041a2c50 con 0x7f1204102860 2024-04-26T23:58:54.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f11fbe00640 1 -- 172.21.15.40:0/186707982 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11f400ee70 con 0x7f1204102860 2024-04-26T23:58:54.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f11fbe00640 1 -- 172.21.15.40:0/186707982 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f4010650 con 0x7f1204102860 2024-04-26T23:58:54.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.394+0000 7f1208e00640 1 -- 172.21.15.40:0/186707982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12041a3170 con 0x7f1204102860 2024-04-26T23:58:54.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.395+0000 7f1208e00640 1 -- 172.21.15.40:0/186707982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1204109cf0 con 0x7f1204102860 2024-04-26T23:58:54.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.399+0000 7f11fbe00640 1 -- 172.21.15.40:0/186707982 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f11f4004a40 con 0x7f1204102860 2024-04-26T23:58:54.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.401+0000 7f11fbe00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f11e80a3bb0 0x7f11e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:54.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.401+0000 7f11fbe00640 1 -- 172.21.15.40:0/186707982 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f11f40d0e40 con 0x7f1204102860 2024-04-26T23:58:54.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.401+0000 7f1202a00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f11e80a3bb0 0x7f11e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:54.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.402+0000 7f1202a00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f11e80a3bb0 0x7f11e80a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f11ec000c00 tx=0x7f11ec0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:54.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.404+0000 7f11fbe00640 1 -- 172.21.15.40:0/186707982 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f11f409d5a0 con 0x7f1204102860 2024-04-26T23:58:54.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:54 smithi040 ceph-mon[28095]: pgmap v320: 97 pgs: 1 creating+activating, 10 creating+peering, 48 active+clean, 38 unknown; 578 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2024-04-26T23:58:54.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:54 smithi040 ceph-mon[28095]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T23:58:54.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:54 smithi040 ceph-mon[28095]: Cluster is now healthy 2024-04-26T23:58:54.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:54 smithi040 ceph-mon[28095]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T23:58:54.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:54 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1855288653' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T23:58:54.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:54 smithi040 ceph-mon[28095]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T23:58:54.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:54 smithi080 ceph-mon[32362]: pgmap v320: 97 pgs: 1 creating+activating, 10 creating+peering, 48 active+clean, 38 unknown; 578 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2024-04-26T23:58:54.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:54 smithi080 ceph-mon[32362]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T23:58:54.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:54 smithi080 ceph-mon[32362]: Cluster is now healthy 2024-04-26T23:58:54.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:54 smithi080 ceph-mon[32362]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T23:58:54.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:54 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1855288653' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T23:58:54.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:54 smithi080 ceph-mon[32362]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T23:58:54.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:54 smithi103 ceph-mon[32952]: pgmap v320: 97 pgs: 1 creating+activating, 10 creating+peering, 48 active+clean, 38 unknown; 578 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 3 op/s 2024-04-26T23:58:54.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:54 smithi103 ceph-mon[32952]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T23:58:54.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:54 smithi103 ceph-mon[32952]: Cluster is now healthy 2024-04-26T23:58:54.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:54 smithi103 ceph-mon[32952]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T23:58:54.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:54 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1855288653' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T23:58:54.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:54 smithi103 ceph-mon[32952]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T23:58:54.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:54.610+0000 7f1208e00640 1 -- 172.21.15.40:0/186707982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f1204063530 con 0x7f1204102860 2024-04-26T23:58:55.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.164+0000 7f11fbe00640 1 -- 172.21.15.40:0/186707982 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) ==== 169+0+0 (secure 0 0 0) 0x7f11f409cea0 con 0x7f1204102860 2024-04-26T23:58:55.166 INFO:teuthology.orchestra.run.smithi040.stderr:set pool 4 allow_ec_overwrites to true 2024-04-26T23:58:55.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.168+0000 7f1208e00640 1 -- 172.21.15.40:0/186707982 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f11e80a3bb0 msgr2=0x7f11e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:55.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.168+0000 7f1208e00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f11e80a3bb0 0x7f11e80a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f11ec000c00 tx=0x7f11ec0314a0 comp rx=0 tx=0).stop 2024-04-26T23:58:55.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.168+0000 7f1208e00640 1 -- 172.21.15.40:0/186707982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1204102860 msgr2=0x7f1204076690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:55.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.168+0000 7f1208e00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1204102860 0x7f1204076690 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f11f400eb70 tx=0x7f11f400c6a0 comp rx=0 tx=0).stop 2024-04-26T23:58:55.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.168+0000 7f1208e00640 1 -- 172.21.15.40:0/186707982 shutdown_connections 2024-04-26T23:58:55.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.168+0000 7f1208e00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1204104410 0x7f1204075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:55.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.168+0000 7f1208e00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f11e80a3bb0 0x7f11e80a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:55.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.168+0000 7f1208e00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1204103a50 0x7f1204074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:55.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.168+0000 7f1208e00640 1 --2- 172.21.15.40:0/186707982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1204102860 0x7f1204076690 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:55.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.168+0000 7f1208e00640 1 -- 172.21.15.40:0/186707982 >> 172.21.15.40:0/186707982 conn(0x7f12040fddd0 msgr2=0x7f1204106ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:55.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.169+0000 7f1208e00640 1 -- 172.21.15.40:0/186707982 shutdown_connections 2024-04-26T23:58:55.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.169+0000 7f1208e00640 1 -- 172.21.15.40:0/186707982 wait complete. 2024-04-26T23:58:55.203 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-26T23:58:55.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.388+0000 7f5f8b000640 1 -- 172.21.15.40:0/1520957725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f84078530 msgr2=0x7f5f84078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:55.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.388+0000 7f5f8b000640 1 --2- 172.21.15.40:0/1520957725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f84078530 0x7f5f84078990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c009900 tx=0x7f5f6c02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:58:55.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.388+0000 7f5f8b000640 1 -- 172.21.15.40:0/1520957725 shutdown_connections 2024-04-26T23:58:55.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.388+0000 7f5f8b000640 1 --2- 172.21.15.40:0/1520957725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f84078ed0 0x7f5f84079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:55.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.388+0000 7f5f8b000640 1 --2- 172.21.15.40:0/1520957725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f84078530 0x7f5f84078990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:55.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.388+0000 7f5f8b000640 1 --2- 172.21.15.40:0/1520957725 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5f840772f0 0x7f5f84077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:55.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.388+0000 7f5f8b000640 1 -- 172.21.15.40:0/1520957725 >> 172.21.15.40:0/1520957725 conn(0x7f5f841000e0 msgr2=0x7f5f84102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:55.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.389+0000 7f5f8b000640 1 -- 172.21.15.40:0/1520957725 shutdown_connections 2024-04-26T23:58:55.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.389+0000 7f5f8b000640 1 -- 172.21.15.40:0/1520957725 wait complete. 2024-04-26T23:58:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f8b000640 1 Processor -- start 2024-04-26T23:58:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f8b000640 1 -- start start 2024-04-26T23:58:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f8b000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840772f0 0x7f5f8419f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f8b000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5f84078530 0x7f5f8419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f8b000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f84078ed0 0x7f5f841a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f8b000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f84114b10 con 0x7f5f840772f0 2024-04-26T23:58:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f8b000640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f5f84114990 con 0x7f5f84078530 2024-04-26T23:58:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f8b000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5f84114c90 con 0x7f5f84078ed0 2024-04-26T23:58:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f83e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840772f0 0x7f5f8419f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f89000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f84078ed0 0x7f5f841a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:55.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f83e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840772f0 0x7f5f8419f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52864/0 (socket says 172.21.15.40:52864) 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f83e00640 1 -- 172.21.15.40:0/2861391182 learned_addr learned my addr 172.21.15.40:0/2861391182 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.390+0000 7f5f7b400640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5f84078530 0x7f5f8419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.391+0000 7f5f89000640 1 -- 172.21.15.40:0/2861391182 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5f84078530 msgr2=0x7f5f8419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.391+0000 7f5f89000640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5f84078530 0x7f5f8419f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.391+0000 7f5f89000640 1 -- 172.21.15.40:0/2861391182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840772f0 msgr2=0x7f5f8419f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.391+0000 7f5f89000640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840772f0 0x7f5f8419f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.391+0000 7f5f89000640 1 -- 172.21.15.40:0/2861391182 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f841a6df0 con 0x7f5f84078ed0 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.391+0000 7f5f83e00640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840772f0 0x7f5f8419f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.391+0000 7f5f89000640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f84078ed0 0x7f5f841a6780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009fd0 tx=0x7f5f7c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.391+0000 7f5f81600640 1 -- 172.21.15.40:0/2861391182 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c00ce20 con 0x7f5f84078ed0 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.391+0000 7f5f81600640 1 -- 172.21.15.40:0/2861391182 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f7c005020 con 0x7f5f84078ed0 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.391+0000 7f5f8b000640 1 -- 172.21.15.40:0/2861391182 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f841a70e0 con 0x7f5f84078ed0 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.391+0000 7f5f81600640 1 -- 172.21.15.40:0/2861391182 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c010690 con 0x7f5f84078ed0 2024-04-26T23:58:55.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.391+0000 7f5f8b000640 1 -- 172.21.15.40:0/2861391182 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f841a7600 con 0x7f5f84078ed0 2024-04-26T23:58:55.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.392+0000 7f5f8b000640 1 -- 172.21.15.40:0/2861391182 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f84111710 con 0x7f5f84078ed0 2024-04-26T23:58:55.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.394+0000 7f5f81600640 1 -- 172.21.15.40:0/2861391182 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5f7c0054d0 con 0x7f5f84078ed0 2024-04-26T23:58:55.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.396+0000 7f5f81600640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5f680a3bb0 0x7f5f680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:55.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.396+0000 7f5f81600640 1 -- 172.21.15.40:0/2861391182 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7f5f7c0d0d50 con 0x7f5f84078ed0 2024-04-26T23:58:55.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.396+0000 7f5f83e00640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5f680a3bb0 0x7f5f680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:55.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.397+0000 7f5f83e00640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5f680a3bb0 0x7f5f680a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5f740095f0 tx=0x7f5f74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:55.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.400+0000 7f5f81600640 1 -- 172.21.15.40:0/2861391182 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5f7c09d4b0 con 0x7f5f84078ed0 2024-04-26T23:58:55.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:55 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/186707982' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T23:58:55.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:55 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/186707982' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T23:58:55.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:55 smithi040 ceph-mon[28095]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T23:58:55.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:55 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/186707982' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T23:58:55.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:55 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/186707982' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T23:58:55.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:55 smithi080 ceph-mon[32362]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T23:58:55.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:55 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/186707982' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T23:58:55.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:55 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/186707982' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T23:58:55.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:55 smithi103 ceph-mon[32952]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T23:58:55.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:55.647+0000 7f5f8b000640 1 -- 172.21.15.40:0/2861391182 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f5f84063530 con 0x7f5f84078ed0 2024-04-26T23:58:56.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.222+0000 7f5f81600640 1 -- 172.21.15.40:0/2861391182 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7f5f7c09cdb0 con 0x7f5f84078ed0 2024-04-26T23:58:56.225 INFO:teuthology.orchestra.run.smithi040.stderr:added data pool 4 to fsmap 2024-04-26T23:58:56.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.226+0000 7f5f8b000640 1 -- 172.21.15.40:0/2861391182 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5f680a3bb0 msgr2=0x7f5f680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:56.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.226+0000 7f5f8b000640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5f680a3bb0 0x7f5f680a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5f740095f0 tx=0x7f5f74005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:56.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.226+0000 7f5f8b000640 1 -- 172.21.15.40:0/2861391182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f84078ed0 msgr2=0x7f5f841a6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:56.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.226+0000 7f5f8b000640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f84078ed0 0x7f5f841a6780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009fd0 tx=0x7f5f7c00ef10 comp rx=0 tx=0).stop 2024-04-26T23:58:56.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.226+0000 7f5f8b000640 1 -- 172.21.15.40:0/2861391182 shutdown_connections 2024-04-26T23:58:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.226+0000 7f5f8b000640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5f680a3bb0 0x7f5f680a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.226+0000 7f5f8b000640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f84078ed0 0x7f5f841a6780 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.226+0000 7f5f8b000640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5f84078530 0x7f5f8419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.226+0000 7f5f8b000640 1 --2- 172.21.15.40:0/2861391182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840772f0 0x7f5f8419f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.226+0000 7f5f8b000640 1 -- 172.21.15.40:0/2861391182 >> 172.21.15.40:0/2861391182 conn(0x7f5f841000e0 msgr2=0x7f5f841018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.226+0000 7f5f8b000640 1 -- 172.21.15.40:0/2861391182 shutdown_connections 2024-04-26T23:58:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.226+0000 7f5f8b000640 1 -- 172.21.15.40:0/2861391182 wait complete. 2024-04-26T23:58:56.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:56.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.462+0000 7fb56f200640 1 -- 172.21.15.40:0/3083360125 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb568104b90 msgr2=0x7fb568104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:56.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.462+0000 7fb56f200640 1 --2- 172.21.15.40:0/3083360125 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb568104b90 0x7fb568104fb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb558009860 tx=0x7fb55802a340 comp rx=0 tx=0).stop 2024-04-26T23:58:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.462+0000 7fb56f200640 1 -- 172.21.15.40:0/3083360125 shutdown_connections 2024-04-26T23:58:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.462+0000 7fb56f200640 1 --2- 172.21.15.40:0/3083360125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568106740 0x7fb56810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.462+0000 7fb56f200640 1 --2- 172.21.15.40:0/3083360125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb568105d80 0x7fb568106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.462+0000 7fb56f200640 1 --2- 172.21.15.40:0/3083360125 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb568104b90 0x7fb568104fb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.462+0000 7fb56f200640 1 -- 172.21.15.40:0/3083360125 >> 172.21.15.40:0/3083360125 conn(0x7fb568100120 msgr2=0x7fb568102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.462+0000 7fb56f200640 1 -- 172.21.15.40:0/3083360125 shutdown_connections 2024-04-26T23:58:56.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.462+0000 7fb56f200640 1 -- 172.21.15.40:0/3083360125 wait complete. 2024-04-26T23:58:56.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.463+0000 7fb56f200640 1 Processor -- start 2024-04-26T23:58:56.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.463+0000 7fb56f200640 1 -- start start 2024-04-26T23:58:56.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb56f200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb568104b90 0x7fb56819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:56.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb56f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568105d80 0x7fb56819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:56.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb56f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb568106740 0x7fb5681a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:56.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb56f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb5681106b0 con 0x7fb568105d80 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb56f200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb568110530 con 0x7fb568104b90 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb56f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb568110830 con 0x7fb568106740 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb55fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568105d80 0x7fb56819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb567e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb568104b90 0x7fb56819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb55fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568105d80 0x7fb56819b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52878/0 (socket says 172.21.15.40:52878) 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb55fe00640 1 -- 172.21.15.40:0/869098822 learned_addr learned my addr 172.21.15.40:0/869098822 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb56d200640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb568106740 0x7fb5681a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb567e00640 1 -- 172.21.15.40:0/869098822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb568106740 msgr2=0x7fb5681a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb567e00640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb568106740 0x7fb5681a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb567e00640 1 -- 172.21.15.40:0/869098822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568105d80 msgr2=0x7fb56819b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb567e00640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568105d80 0x7fb56819b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb567e00640 1 -- 172.21.15.40:0/869098822 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5681a2a00 con 0x7fb568104b90 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.464+0000 7fb55fe00640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568105d80 0x7fb56819b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:56.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.465+0000 7fb567e00640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb568104b90 0x7fb56819ade0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb55802a860 tx=0x7fb558032dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:56.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:56 smithi040 ceph-mon[28095]: pgmap v323: 129 pgs: 1 creating+activating, 11 creating+peering, 81 active+clean, 36 unknown; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-04-26T23:58:56.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:56 smithi040 ceph-mon[28095]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T23:58:56.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:56 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2861391182' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T23:58:56.469 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:56 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T23:58:56.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.465+0000 7fb565600640 1 -- 172.21.15.40:0/869098822 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb558037040 con 0x7fb568104b90 2024-04-26T23:58:56.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.465+0000 7fb565600640 1 -- 172.21.15.40:0/869098822 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb558030bd0 con 0x7fb568104b90 2024-04-26T23:58:56.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.465+0000 7fb56f200640 1 -- 172.21.15.40:0/869098822 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5681a2c90 con 0x7fb568104b90 2024-04-26T23:58:56.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.465+0000 7fb565600640 1 -- 172.21.15.40:0/869098822 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb558003850 con 0x7fb568104b90 2024-04-26T23:58:56.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.465+0000 7fb56f200640 1 -- 172.21.15.40:0/869098822 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5681a3120 con 0x7fb568104b90 2024-04-26T23:58:56.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.466+0000 7fb56f200640 1 -- 172.21.15.40:0/869098822 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb56810bf30 con 0x7fb568104b90 2024-04-26T23:58:56.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.468+0000 7fb565600640 1 -- 172.21.15.40:0/869098822 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb558043020 con 0x7fb568104b90 2024-04-26T23:58:56.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.469+0000 7fb565600640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb54c0a3bb0 0x7fb54c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:56.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.470+0000 7fb565600640 1 -- 172.21.15.40:0/869098822 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb5580ef880 con 0x7fb568104b90 2024-04-26T23:58:56.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.470+0000 7fb55fe00640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb54c0a3bb0 0x7fb54c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:56.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.470+0000 7fb55fe00640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb54c0a3bb0 0x7fb54c0a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb56819c2f0 tx=0x7fb550005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:56.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.474+0000 7fb565600640 1 -- 172.21.15.40:0/869098822 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb55804a060 con 0x7fb568104b90 2024-04-26T23:58:56.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:56 smithi080 ceph-mon[32362]: pgmap v323: 129 pgs: 1 creating+activating, 11 creating+peering, 81 active+clean, 36 unknown; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-04-26T23:58:56.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:56 smithi080 ceph-mon[32362]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T23:58:56.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:56 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2861391182' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T23:58:56.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:56 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T23:58:56.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:56 smithi103 ceph-mon[32952]: pgmap v323: 129 pgs: 1 creating+activating, 11 creating+peering, 81 active+clean, 36 unknown; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-04-26T23:58:56.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:56 smithi103 ceph-mon[32952]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T23:58:56.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:56 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2861391182' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T23:58:56.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:56 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T23:58:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.723+0000 7fb56f200640 1 -- 172.21.15.40:0/869098822 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb56802dad0 con 0x7fb568104b90 2024-04-26T23:58:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.724+0000 7fb565600640 1 -- 172.21.15.40:0/869098822 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11800 (secure 0 0 0) 0x7fb5580bbfc0 con 0x7fb568104b90 2024-04-26T23:58:56.726 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/2265538751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":2265538751},{"type":"v1","addr":"172.21.15.40:6841","nonce":2265538751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:58:50.616677+0000","modified":"2024-04-26T23:58:55.650180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.80:6839/3536638444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6838","nonce":3536638444},{"type":"v1","addr":"172.21.15.80:6839","nonce":3536638444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24590,"qdb_cluster":[24590]},"id":1}]} 2024-04-26T23:58:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 16 2024-04-26T23:58:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.727+0000 7fb56f200640 1 -- 172.21.15.40:0/869098822 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb54c0a3bb0 msgr2=0x7fb54c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.727+0000 7fb56f200640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb54c0a3bb0 0x7fb54c0a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb56819c2f0 tx=0x7fb550005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.727+0000 7fb56f200640 1 -- 172.21.15.40:0/869098822 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb568104b90 msgr2=0x7fb56819ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.727+0000 7fb56f200640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb568104b90 0x7fb56819ade0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb55802a860 tx=0x7fb558032dc0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.727+0000 7fb56f200640 1 -- 172.21.15.40:0/869098822 shutdown_connections 2024-04-26T23:58:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.727+0000 7fb56f200640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb54c0a3bb0 0x7fb54c0a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.727+0000 7fb56f200640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb568106740 0x7fb5681a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.727+0000 7fb56f200640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568105d80 0x7fb56819b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.727+0000 7fb56f200640 1 --2- 172.21.15.40:0/869098822 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb568104b90 0x7fb56819ade0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.727+0000 7fb56f200640 1 -- 172.21.15.40:0/869098822 >> 172.21.15.40:0/869098822 conn(0x7fb568100120 msgr2=0x7fb568109100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.727+0000 7fb56f200640 1 -- 172.21.15.40:0/869098822 shutdown_connections 2024-04-26T23:58:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.727+0000 7fb56f200640 1 -- 172.21.15.40:0/869098822 wait complete. 2024-04-26T23:58:56.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:58:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.954+0000 7fe2dfe00640 1 -- 172.21.15.40:0/1515192408 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2d8078ed0 msgr2=0x7fe2d8079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.954+0000 7fe2dfe00640 1 --2- 172.21.15.40:0/1515192408 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2d8078ed0 0x7fe2d8079770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009900 tx=0x7fe2d002a400 comp rx=0 tx=0).stop 2024-04-26T23:58:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.954+0000 7fe2dfe00640 1 -- 172.21.15.40:0/1515192408 shutdown_connections 2024-04-26T23:58:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.954+0000 7fe2dfe00640 1 --2- 172.21.15.40:0/1515192408 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2d8078ed0 0x7fe2d8079770 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.954+0000 7fe2dfe00640 1 --2- 172.21.15.40:0/1515192408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d8078530 0x7fe2d8078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.954+0000 7fe2dfe00640 1 --2- 172.21.15.40:0/1515192408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2d80772f0 0x7fe2d8077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.954+0000 7fe2dfe00640 1 -- 172.21.15.40:0/1515192408 >> 172.21.15.40:0/1515192408 conn(0x7fe2d8100120 msgr2=0x7fe2d8102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.954+0000 7fe2dfe00640 1 -- 172.21.15.40:0/1515192408 shutdown_connections 2024-04-26T23:58:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.954+0000 7fe2dfe00640 1 -- 172.21.15.40:0/1515192408 wait complete. 2024-04-26T23:58:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.955+0000 7fe2dfe00640 1 Processor -- start 2024-04-26T23:58:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.956+0000 7fe2dfe00640 1 -- start start 2024-04-26T23:58:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.956+0000 7fe2dfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2d80772f0 0x7fe2d819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.956+0000 7fe2dfe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2d8078530 0x7fe2d819f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.956+0000 7fe2dfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d8078ed0 0x7fe2d81a66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.956+0000 7fe2dfe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d8114af0 con 0x7fe2d80772f0 2024-04-26T23:58:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.956+0000 7fe2dfe00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d8114970 con 0x7fe2d8078530 2024-04-26T23:58:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.956+0000 7fe2dfe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d8114c70 con 0x7fe2d8078ed0 2024-04-26T23:58:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.956+0000 7fe2dde00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d8078ed0 0x7fe2d81a66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.956+0000 7fe2dde00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d8078ed0 0x7fe2d81a66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43430/0 (socket says 172.21.15.40:43430) 2024-04-26T23:58:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.956+0000 7fe2dde00640 1 -- 172.21.15.40:0/2934630496 learned_addr learned my addr 172.21.15.40:0/2934630496 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.956+0000 7fe2dca00640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2d8078530 0x7fe2d819f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.956+0000 7fe2dd400640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2d80772f0 0x7fe2d819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.957+0000 7fe2dde00640 1 -- 172.21.15.40:0/2934630496 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2d8078530 msgr2=0x7fe2d819f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.957+0000 7fe2dde00640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2d8078530 0x7fe2d819f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.957+0000 7fe2dde00640 1 -- 172.21.15.40:0/2934630496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2d80772f0 msgr2=0x7fe2d819f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.957+0000 7fe2dde00640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2d80772f0 0x7fe2d819f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.957+0000 7fe2dde00640 1 -- 172.21.15.40:0/2934630496 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d81a6d40 con 0x7fe2d8078ed0 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.957+0000 7fe2dde00640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d8078ed0 0x7fe2d81a66d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0005ab0 tx=0x7fe2d0003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.957+0000 7fe2d6000640 1 -- 172.21.15.40:0/2934630496 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d003f070 con 0x7fe2d8078ed0 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.957+0000 7fe2d6000640 1 -- 172.21.15.40:0/2934630496 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2d0031c50 con 0x7fe2d8078ed0 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.957+0000 7fe2dfe00640 1 -- 172.21.15.40:0/2934630496 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d81a6fd0 con 0x7fe2d8078ed0 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.957+0000 7fe2d6000640 1 -- 172.21.15.40:0/2934630496 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0035030 con 0x7fe2d8078ed0 2024-04-26T23:58:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.957+0000 7fe2dfe00640 1 -- 172.21.15.40:0/2934630496 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d81a7440 con 0x7fe2d8078ed0 2024-04-26T23:58:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.959+0000 7fe2dfe00640 1 -- 172.21.15.40:0/2934630496 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d81114e0 con 0x7fe2d8078ed0 2024-04-26T23:58:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.961+0000 7fe2d6000640 1 -- 172.21.15.40:0/2934630496 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe2d0042070 con 0x7fe2d8078ed0 2024-04-26T23:58:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.962+0000 7fe2d6000640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe2bc0a3bb0 0x7fe2bc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.962+0000 7fe2d6000640 1 -- 172.21.15.40:0/2934630496 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe2d00f0e00 con 0x7fe2d8078ed0 2024-04-26T23:58:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.962+0000 7fe2dd400640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe2bc0a3bb0 0x7fe2bc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.963+0000 7fe2dd400640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe2bc0a3bb0 0x7fe2bc0a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8005fd0 tx=0x7fe2c8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:56.967+0000 7fe2d6000640 1 -- 172.21.15.40:0/2934630496 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe2d00bcef0 con 0x7fe2d8078ed0 2024-04-26T23:58:57.050 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 26 23:58:56 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b[63150]: 2024-04-26T23:58:56.551+0000 7f1faee6c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T23:58:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.175+0000 7fe2dfe00640 1 -- 172.21.15.40:0/2934630496 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe2d80635d0 con 0x7fe2d8078ed0 2024-04-26T23:58:57.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.177+0000 7fe2d6000640 1 -- 172.21.15.40:0/2934630496 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24090 (secure 0 0 0) 0x7fe2d00c11f0 con 0x7fe2d8078ed0 2024-04-26T23:58:57.179 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:57.180 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:58:55.649955+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:58:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.180+0000 7fe2dfe00640 1 -- 172.21.15.40:0/2934630496 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe2bc0a3bb0 msgr2=0x7fe2bc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.180+0000 7fe2dfe00640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe2bc0a3bb0 0x7fe2bc0a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8005fd0 tx=0x7fe2c8009500 comp rx=0 tx=0).stop 2024-04-26T23:58:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.180+0000 7fe2dfe00640 1 -- 172.21.15.40:0/2934630496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d8078ed0 msgr2=0x7fe2d81a66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.180+0000 7fe2dfe00640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d8078ed0 0x7fe2d81a66d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0005ab0 tx=0x7fe2d0003b90 comp rx=0 tx=0).stop 2024-04-26T23:58:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.180+0000 7fe2dfe00640 1 -- 172.21.15.40:0/2934630496 shutdown_connections 2024-04-26T23:58:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.180+0000 7fe2dfe00640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe2bc0a3bb0 0x7fe2bc0a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.180+0000 7fe2dfe00640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2d8078ed0 0x7fe2d81a66d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.180+0000 7fe2dfe00640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe2d8078530 0x7fe2d819f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.180+0000 7fe2dfe00640 1 --2- 172.21.15.40:0/2934630496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2d80772f0 0x7fe2d819f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.180+0000 7fe2dfe00640 1 -- 172.21.15.40:0/2934630496 >> 172.21.15.40:0/2934630496 conn(0x7fe2d8100120 msgr2=0x7fe2d8101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.180+0000 7fe2dfe00640 1 -- 172.21.15.40:0/2934630496 shutdown_connections 2024-04-26T23:58:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.180+0000 7fe2dfe00640 1 -- 172.21.15.40:0/2934630496 wait complete. 2024-04-26T23:58:57.204 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:58:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.400+0000 7fe10b200640 1 -- 172.21.15.40:0/3467964407 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe10c109f80 msgr2=0x7fe10c10a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.400+0000 7fe10b200640 1 --2- 172.21.15.40:0/3467964407 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe10c109f80 0x7fe10c10a400 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009900 tx=0x7fe0f802a3f0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.400+0000 7fe10b200640 1 -- 172.21.15.40:0/3467964407 shutdown_connections 2024-04-26T23:58:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.400+0000 7fe10b200640 1 --2- 172.21.15.40:0/3467964407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe10c10a940 0x7fe10c1111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.400+0000 7fe10b200640 1 --2- 172.21.15.40:0/3467964407 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe10c109f80 0x7fe10c10a400 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.400+0000 7fe10b200640 1 --2- 172.21.15.40:0/3467964407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10c108d90 0x7fe10c1091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.400+0000 7fe10b200640 1 -- 172.21.15.40:0/3467964407 >> 172.21.15.40:0/3467964407 conn(0x7fe10c077d30 msgr2=0x7fe10c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.400+0000 7fe10b200640 1 -- 172.21.15.40:0/3467964407 shutdown_connections 2024-04-26T23:58:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.400+0000 7fe10b200640 1 -- 172.21.15.40:0/3467964407 wait complete. 2024-04-26T23:58:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.401+0000 7fe10b200640 1 Processor -- start 2024-04-26T23:58:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.401+0000 7fe10b200640 1 -- start start 2024-04-26T23:58:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.401+0000 7fe10b200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe10c108d90 0x7fe10c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.401+0000 7fe10b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10c109f80 0x7fe10c19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.401+0000 7fe10b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe10c10a940 0x7fe10c1a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.401+0000 7fe10b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe10c1148a0 con 0x7fe10c109f80 2024-04-26T23:58:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.401+0000 7fe10b200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fe10c114720 con 0x7fe10c108d90 2024-04-26T23:58:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.401+0000 7fe10b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe10c114a20 con 0x7fe10c10a940 2024-04-26T23:58:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.402+0000 7fe109400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10c109f80 0x7fe10c19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.402+0000 7fe109e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe10c108d90 0x7fe10c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.402+0000 7fe109e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe10c108d90 0x7fe10c19ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:40720/0 (socket says 172.21.15.40:40720) 2024-04-26T23:58:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.402+0000 7fe109e00640 1 -- 172.21.15.40:0/2243859458 learned_addr learned my addr 172.21.15.40:0/2243859458 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.402+0000 7fe10a800640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe10c10a940 0x7fe10c1a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.402+0000 7fe109e00640 1 -- 172.21.15.40:0/2243859458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe10c10a940 msgr2=0x7fe10c1a64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.402+0000 7fe109e00640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe10c10a940 0x7fe10c1a64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.402+0000 7fe109e00640 1 -- 172.21.15.40:0/2243859458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10c109f80 msgr2=0x7fe10c19f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.402+0000 7fe109e00640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10c109f80 0x7fe10c19f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.402+0000 7fe109e00640 1 -- 172.21.15.40:0/2243859458 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10c1a6b50 con 0x7fe10c108d90 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.402+0000 7fe109400640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10c109f80 0x7fe10c19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.402+0000 7fe109e00640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe10c108d90 0x7fe10c19ef30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00e9c0 tx=0x7fe0fc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.403+0000 7fe102a00640 1 -- 172.21.15.40:0/2243859458 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc00cd80 con 0x7fe10c108d90 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.403+0000 7fe102a00640 1 -- 172.21.15.40:0/2243859458 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0fc0035f0 con 0x7fe10c108d90 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.403+0000 7fe10b200640 1 -- 172.21.15.40:0/2243859458 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe10c1a6e40 con 0x7fe10c108d90 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.403+0000 7fe102a00640 1 -- 172.21.15.40:0/2243859458 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc010690 con 0x7fe10c108d90 2024-04-26T23:58:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.403+0000 7fe10b200640 1 -- 172.21.15.40:0/2243859458 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe10c1a72a0 con 0x7fe10c108d90 2024-04-26T23:58:57.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.404+0000 7fe10b200640 1 -- 172.21.15.40:0/2243859458 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe10c110150 con 0x7fe10c108d90 2024-04-26T23:58:57.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.406+0000 7fe102a00640 1 -- 172.21.15.40:0/2243859458 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe0fc0037a0 con 0x7fe10c108d90 2024-04-26T23:58:57.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.407+0000 7fe102a00640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe0f00a3bb0 0x7fe0f00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:57.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.407+0000 7fe102a00640 1 -- 172.21.15.40:0/2243859458 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe0fc0d09e0 con 0x7fe10c108d90 2024-04-26T23:58:57.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.408+0000 7fe109400640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe0f00a3bb0 0x7fe0f00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:57.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.408+0000 7fe109400640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe0f00a3bb0 0x7fe0f00a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8000c00 tx=0x7fe0f80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:57.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.412+0000 7fe102a00640 1 -- 172.21.15.40:0/2243859458 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe0fc09d120 con 0x7fe10c108d90 2024-04-26T23:58:57.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:57 smithi040 ceph-mon[28095]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T23:58:57.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:57 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T23:58:57.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:57 smithi040 ceph-mon[28095]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:58:57.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:57 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/869098822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:57.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:57 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2934630496' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:57.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:57 smithi080 ceph-mon[32362]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T23:58:57.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:57 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T23:58:57.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:57 smithi080 ceph-mon[32362]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:58:57.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:57 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/869098822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:57.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:57 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2934630496' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:57.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:57 smithi103 ceph-mon[32952]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T23:58:57.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:57 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T23:58:57.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:57 smithi103 ceph-mon[32952]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:58:57.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:57 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/869098822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:57.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:57 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2934630496' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.618+0000 7fe10b200640 1 -- 172.21.15.40:0/2243859458 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe10c063630 con 0x7fe10c108d90 2024-04-26T23:58:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.620+0000 7fe102a00640 1 -- 172.21.15.40:0/2243859458 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24090 (secure 0 0 0) 0x7fe0fc09ca20 con 0x7fe10c108d90 2024-04-26T23:58:57.623 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:57.623 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:58:55.649955+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:58:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.623+0000 7fe10b200640 1 -- 172.21.15.40:0/2243859458 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe0f00a3bb0 msgr2=0x7fe0f00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.623+0000 7fe10b200640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe0f00a3bb0 0x7fe0f00a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8000c00 tx=0x7fe0f80314a0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.623+0000 7fe10b200640 1 -- 172.21.15.40:0/2243859458 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe10c108d90 msgr2=0x7fe10c19ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.623+0000 7fe10b200640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe10c108d90 0x7fe10c19ef30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00e9c0 tx=0x7fe0fc00ee80 comp rx=0 tx=0).stop 2024-04-26T23:58:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.623+0000 7fe10b200640 1 -- 172.21.15.40:0/2243859458 shutdown_connections 2024-04-26T23:58:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.623+0000 7fe10b200640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe0f00a3bb0 0x7fe0f00a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.623+0000 7fe10b200640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe10c10a940 0x7fe10c1a64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.623+0000 7fe10b200640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe10c109f80 0x7fe10c19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.623+0000 7fe10b200640 1 --2- 172.21.15.40:0/2243859458 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe10c108d90 0x7fe10c19ef30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.623+0000 7fe10b200640 1 -- 172.21.15.40:0/2243859458 >> 172.21.15.40:0/2243859458 conn(0x7fe10c077d30 msgr2=0x7fe10c10d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.623+0000 7fe10b200640 1 -- 172.21.15.40:0/2243859458 shutdown_connections 2024-04-26T23:58:57.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.623+0000 7fe10b200640 1 -- 172.21.15.40:0/2243859458 wait complete. 2024-04-26T23:58:57.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:58:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.867+0000 7fb89fe00640 1 -- 172.21.15.40:0/2528488988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8a0078530 msgr2=0x7fb8a0078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.867+0000 7fb89fe00640 1 --2- 172.21.15.40:0/2528488988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8a0078530 0x7fb8a0078990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009900 tx=0x7fb88c02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.868+0000 7fb89fe00640 1 -- 172.21.15.40:0/2528488988 shutdown_connections 2024-04-26T23:58:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.868+0000 7fb89fe00640 1 --2- 172.21.15.40:0/2528488988 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8a0078ed0 0x7fb8a0079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.868+0000 7fb89fe00640 1 --2- 172.21.15.40:0/2528488988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8a0078530 0x7fb8a0078990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.868+0000 7fb89fe00640 1 --2- 172.21.15.40:0/2528488988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8a00772f0 0x7fb8a0077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.868+0000 7fb89fe00640 1 -- 172.21.15.40:0/2528488988 >> 172.21.15.40:0/2528488988 conn(0x7fb8a00ffeb0 msgr2=0x7fb8a0102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.868+0000 7fb89fe00640 1 -- 172.21.15.40:0/2528488988 shutdown_connections 2024-04-26T23:58:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.868+0000 7fb89fe00640 1 -- 172.21.15.40:0/2528488988 wait complete. 2024-04-26T23:58:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.869+0000 7fb89fe00640 1 Processor -- start 2024-04-26T23:58:57.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.869+0000 7fb89fe00640 1 -- start start 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.869+0000 7fb89fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8a00772f0 0x7fb8a019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.869+0000 7fb89fe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8a0078530 0x7fb8a019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.869+0000 7fb89fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8a0078ed0 0x7fb8a01a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.869+0000 7fb89fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a01148a0 con 0x7fb8a0078ed0 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.869+0000 7fb89fe00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a0114720 con 0x7fb8a0078530 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.869+0000 7fb89fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a0114a20 con 0x7fb8a00772f0 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.869+0000 7fb89f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8a0078ed0 0x7fb8a01a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.870+0000 7fb89ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8a00772f0 0x7fb8a019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.870+0000 7fb89e000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8a0078530 0x7fb8a019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.870+0000 7fb89ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8a00772f0 0x7fb8a019ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43464/0 (socket says 172.21.15.40:43464) 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.870+0000 7fb89ea00640 1 -- 172.21.15.40:0/1907567388 learned_addr learned my addr 172.21.15.40:0/1907567388 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.870+0000 7fb89f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8a0078ed0 0x7fb8a01a6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52902/0 (socket says 172.21.15.40:52902) 2024-04-26T23:58:57.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.870+0000 7fb89ea00640 1 -- 172.21.15.40:0/1907567388 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8a0078530 msgr2=0x7fb8a019f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.870+0000 7fb89ea00640 1 --2- 172.21.15.40:0/1907567388 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8a0078530 0x7fb8a019f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.870+0000 7fb89ea00640 1 -- 172.21.15.40:0/1907567388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8a0078ed0 msgr2=0x7fb8a01a6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.870+0000 7fb89ea00640 1 --2- 172.21.15.40:0/1907567388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8a0078ed0 0x7fb8a01a6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.870+0000 7fb89ea00640 1 -- 172.21.15.40:0/1907567388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a01a6b90 con 0x7fb8a00772f0 2024-04-26T23:58:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.870+0000 7fb89ea00640 1 --2- 172.21.15.40:0/1907567388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8a00772f0 0x7fb8a019ef70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb89000e9f0 tx=0x7fb89000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.871+0000 7fb897400640 1 -- 172.21.15.40:0/1907567388 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89000cd80 con 0x7fb8a00772f0 2024-04-26T23:58:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.871+0000 7fb897400640 1 -- 172.21.15.40:0/1907567388 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8900108e0 con 0x7fb8a00772f0 2024-04-26T23:58:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.871+0000 7fb89fe00640 1 -- 172.21.15.40:0/1907567388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8a01a6e80 con 0x7fb8a00772f0 2024-04-26T23:58:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.871+0000 7fb897400640 1 -- 172.21.15.40:0/1907567388 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb890018980 con 0x7fb8a00772f0 2024-04-26T23:58:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.871+0000 7fb89fe00640 1 -- 172.21.15.40:0/1907567388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8a01a72e0 con 0x7fb8a00772f0 2024-04-26T23:58:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.872+0000 7fb89fe00640 1 -- 172.21.15.40:0/1907567388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8a01112d0 con 0x7fb8a00772f0 2024-04-26T23:58:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.874+0000 7fb897400640 1 -- 172.21.15.40:0/1907567388 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb890010430 con 0x7fb8a00772f0 2024-04-26T23:58:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.875+0000 7fb897400640 1 --2- 172.21.15.40:0/1907567388 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb8840a3bb0 0x7fb8840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.875+0000 7fb897400640 1 -- 172.21.15.40:0/1907567388 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb8900d0e20 con 0x7fb8a00772f0 2024-04-26T23:58:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.876+0000 7fb89e000640 1 --2- 172.21.15.40:0/1907567388 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb8840a3bb0 0x7fb8840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.876+0000 7fb89e000640 1 --2- 172.21.15.40:0/1907567388 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb8840a3bb0 0x7fb8840a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb88c031500 tx=0x7fb88c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:57.878+0000 7fb897400640 1 -- 172.21.15.40:0/1907567388 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb89009cf10 con 0x7fb8a00772f0 2024-04-26T23:58:58.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.117+0000 7fb89fe00640 1 -- 172.21.15.40:0/1907567388 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8a002dad0 con 0x7fb8a00772f0 2024-04-26T23:58:58.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.118+0000 7fb897400640 1 -- 172.21.15.40:0/1907567388 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11800 (secure 0 0 0) 0x7fb8900a1210 con 0x7fb8a00772f0 2024-04-26T23:58:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:58.121 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/2265538751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":2265538751},{"type":"v1","addr":"172.21.15.40:6841","nonce":2265538751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:58:50.616677+0000","modified":"2024-04-26T23:58:55.650180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.80:6839/3536638444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6838","nonce":3536638444},{"type":"v1","addr":"172.21.15.80:6839","nonce":3536638444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24590,"qdb_cluster":[24590]},"id":1}]} 2024-04-26T23:58:58.121 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 16 2024-04-26T23:58:58.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.121+0000 7fb89fe00640 1 -- 172.21.15.40:0/1907567388 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb8840a3bb0 msgr2=0x7fb8840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.121+0000 7fb89fe00640 1 --2- 172.21.15.40:0/1907567388 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb8840a3bb0 0x7fb8840a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb88c031500 tx=0x7fb88c005a20 comp rx=0 tx=0).stop 2024-04-26T23:58:58.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.121+0000 7fb89fe00640 1 -- 172.21.15.40:0/1907567388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8a00772f0 msgr2=0x7fb8a019ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.121+0000 7fb89fe00640 1 --2- 172.21.15.40:0/1907567388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8a00772f0 0x7fb8a019ef70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb89000e9f0 tx=0x7fb89000eeb0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.121+0000 7fb89fe00640 1 -- 172.21.15.40:0/1907567388 shutdown_connections 2024-04-26T23:58:58.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.121+0000 7fb89fe00640 1 --2- 172.21.15.40:0/1907567388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8a0078ed0 0x7fb8a01a6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.121+0000 7fb89fe00640 1 --2- 172.21.15.40:0/1907567388 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb8a0078530 0x7fb8a019f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.121+0000 7fb89fe00640 1 --2- 172.21.15.40:0/1907567388 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb8840a3bb0 0x7fb8840a6070 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.121+0000 7fb89fe00640 1 --2- 172.21.15.40:0/1907567388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8a00772f0 0x7fb8a019ef70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.121+0000 7fb89fe00640 1 -- 172.21.15.40:0/1907567388 >> 172.21.15.40:0/1907567388 conn(0x7fb8a00ffeb0 msgr2=0x7fb8a01019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:58.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.122+0000 7fb89fe00640 1 -- 172.21.15.40:0/1907567388 shutdown_connections 2024-04-26T23:58:58.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.122+0000 7fb89fe00640 1 -- 172.21.15.40:0/1907567388 wait complete. 2024-04-26T23:58:58.143 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:58:58.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.342+0000 7f37f0c00640 1 -- 172.21.15.40:0/1589453337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37ec104360 msgr2=0x7f37ec10abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.342+0000 7f37f0c00640 1 --2- 172.21.15.40:0/1589453337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37ec104360 0x7f37ec10abf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009ca0 tx=0x7f37e402a8c0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.342+0000 7f37f0c00640 1 -- 172.21.15.40:0/1589453337 shutdown_connections 2024-04-26T23:58:58.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.342+0000 7f37f0c00640 1 --2- 172.21.15.40:0/1589453337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37ec104360 0x7f37ec10abf0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.342+0000 7f37f0c00640 1 --2- 172.21.15.40:0/1589453337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37ec1039a0 0x7f37ec103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.342+0000 7f37f0c00640 1 --2- 172.21.15.40:0/1589453337 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f37ec1027b0 0x7f37ec102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.342+0000 7f37f0c00640 1 -- 172.21.15.40:0/1589453337 >> 172.21.15.40:0/1589453337 conn(0x7f37ec0fdd40 msgr2=0x7f37ec100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:58.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.342+0000 7f37f0c00640 1 -- 172.21.15.40:0/1589453337 shutdown_connections 2024-04-26T23:58:58.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.342+0000 7f37f0c00640 1 -- 172.21.15.40:0/1589453337 wait complete. 2024-04-26T23:58:58.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.343+0000 7f37f0c00640 1 Processor -- start 2024-04-26T23:58:58.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.344+0000 7f37f0c00640 1 -- start start 2024-04-26T23:58:58.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.344+0000 7f37f0c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37ec1027b0 0x7f37ec19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.344+0000 7f37f0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37ec1039a0 0x7f37ec19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.344+0000 7f37f0c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f37ec104360 0x7f37ec1a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.344+0000 7f37f0c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec10e330 con 0x7f37ec1039a0 2024-04-26T23:58:58.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.344+0000 7f37f0c00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec10e1b0 con 0x7f37ec104360 2024-04-26T23:58:58.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.344+0000 7f37f0c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec10e4b0 con 0x7f37ec1027b0 2024-04-26T23:58:58.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.344+0000 7f37e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37ec1039a0 0x7f37ec19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.344+0000 7f37e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37ec1039a0 0x7f37ec19b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52914/0 (socket says 172.21.15.40:52914) 2024-04-26T23:58:58.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.345+0000 7f37e3e00640 1 -- 172.21.15.40:0/1330737537 learned_addr learned my addr 172.21.15.40:0/1330737537 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.344+0000 7f37eb400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37ec1027b0 0x7f37ec19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.345+0000 7f37ebe00640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f37ec104360 0x7f37ec1a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.345+0000 7f37e3e00640 1 -- 172.21.15.40:0/1330737537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37ec1027b0 msgr2=0x7f37ec19ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.345+0000 7f37e3e00640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37ec1027b0 0x7f37ec19ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.345+0000 7f37e3e00640 1 -- 172.21.15.40:0/1330737537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f37ec104360 msgr2=0x7f37ec1a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.345+0000 7f37e3e00640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f37ec104360 0x7f37ec1a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.345+0000 7f37e3e00640 1 -- 172.21.15.40:0/1330737537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ec1a2780 con 0x7f37ec1039a0 2024-04-26T23:58:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.345+0000 7f37e3e00640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37ec1039a0 0x7f37ec19b0a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f37d400e9f0 tx=0x7f37d400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.346+0000 7f37e8c00640 1 -- 172.21.15.40:0/1330737537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d400cd80 con 0x7f37ec1039a0 2024-04-26T23:58:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.346+0000 7f37e8c00640 1 -- 172.21.15.40:0/1330737537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37d40054d0 con 0x7f37ec1039a0 2024-04-26T23:58:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.346+0000 7f37e8c00640 1 -- 172.21.15.40:0/1330737537 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4010690 con 0x7f37ec1039a0 2024-04-26T23:58:58.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.346+0000 7f37f0c00640 1 -- 172.21.15.40:0/1330737537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37ec1a2a70 con 0x7f37ec1039a0 2024-04-26T23:58:58.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.346+0000 7f37f0c00640 1 -- 172.21.15.40:0/1330737537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37ec1a2f70 con 0x7f37ec1039a0 2024-04-26T23:58:58.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.347+0000 7f37f0c00640 1 -- 172.21.15.40:0/1330737537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37ec1a3200 con 0x7f37ec1039a0 2024-04-26T23:58:58.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.349+0000 7f37e8c00640 1 -- 172.21.15.40:0/1330737537 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f37d4004e90 con 0x7f37ec1039a0 2024-04-26T23:58:58.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.350+0000 7f37e8c00640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f37d00a3bb0 0x7f37d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.351+0000 7f37e8c00640 1 -- 172.21.15.40:0/1330737537 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f37d40d0a80 con 0x7f37ec1039a0 2024-04-26T23:58:58.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.351+0000 7f37eb400640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f37d00a3bb0 0x7f37d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.351+0000 7f37eb400640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f37d00a3bb0 0x7f37d00a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f37ec103800 tx=0x7f37dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:58.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.355+0000 7f37e8c00640 1 -- 172.21.15.40:0/1330737537 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f37d40d2e20 con 0x7f37ec1039a0 2024-04-26T23:58:58.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:58 smithi040 ceph-mon[28095]: pgmap v326: 129 pgs: 6 creating+peering, 115 active+clean, 8 unknown; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-04-26T23:58:58.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:58 smithi040 ceph-mon[28095]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T23:58:58.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:58 smithi040 ceph-mon[28095]: Cluster is now healthy 2024-04-26T23:58:58.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:58 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2243859458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:58.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:58 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1907567388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:58.499 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:58 smithi103 ceph-mon[32952]: pgmap v326: 129 pgs: 6 creating+peering, 115 active+clean, 8 unknown; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-04-26T23:58:58.499 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:58 smithi103 ceph-mon[32952]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T23:58:58.500 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:58 smithi103 ceph-mon[32952]: Cluster is now healthy 2024-04-26T23:58:58.500 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:58 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2243859458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:58.500 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:58 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1907567388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:58.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:58 smithi080 ceph-mon[32362]: pgmap v326: 129 pgs: 6 creating+peering, 115 active+clean, 8 unknown; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-04-26T23:58:58.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:58 smithi080 ceph-mon[32362]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T23:58:58.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:58 smithi080 ceph-mon[32362]: Cluster is now healthy 2024-04-26T23:58:58.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:58 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2243859458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:58.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:58 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1907567388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:58:58.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.553+0000 7f37f0c00640 1 -- 172.21.15.40:0/1330737537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37ec1a33d0 con 0x7f37ec1039a0 2024-04-26T23:58:58.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.557+0000 7f37e8c00640 1 -- 172.21.15.40:0/1330737537 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24090 (secure 0 0 0) 0x7f37d409d1c0 con 0x7f37ec1039a0 2024-04-26T23:58:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:58:58.560 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:58:55.649955+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:58:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.560+0000 7f37f0c00640 1 -- 172.21.15.40:0/1330737537 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f37d00a3bb0 msgr2=0x7f37d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.560+0000 7f37f0c00640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f37d00a3bb0 0x7f37d00a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f37ec103800 tx=0x7f37dc005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.560+0000 7f37f0c00640 1 -- 172.21.15.40:0/1330737537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37ec1039a0 msgr2=0x7f37ec19b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.560+0000 7f37f0c00640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37ec1039a0 0x7f37ec19b0a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f37d400e9f0 tx=0x7f37d400edf0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.560+0000 7f37f0c00640 1 -- 172.21.15.40:0/1330737537 shutdown_connections 2024-04-26T23:58:58.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.560+0000 7f37f0c00640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f37ec104360 0x7f37ec1a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.560+0000 7f37f0c00640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37ec1039a0 0x7f37ec19b0a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.560+0000 7f37f0c00640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f37d00a3bb0 0x7f37d00a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.560+0000 7f37f0c00640 1 --2- 172.21.15.40:0/1330737537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37ec1027b0 0x7f37ec19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.560+0000 7f37f0c00640 1 -- 172.21.15.40:0/1330737537 >> 172.21.15.40:0/1330737537 conn(0x7f37ec0fdd40 msgr2=0x7f37ec074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:58.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.560+0000 7f37f0c00640 1 -- 172.21.15.40:0/1330737537 shutdown_connections 2024-04-26T23:58:58.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.560+0000 7f37f0c00640 1 -- 172.21.15.40:0/1330737537 wait complete. 2024-04-26T23:58:58.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:58:50.616677+0000', 'modified': '2024-04-26T23:58:55.650180+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.80:6839/3536638444', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6838', 'nonce': 3536638444}, {'type': 'v1', 'addr': '172.21.15.80:6839', 'nonce': 3536638444}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24590, 'qdb_cluster': [24590]} 2024-04-26T23:58:58.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T23:58:58.581 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T23:58:58.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.789+0000 7f18dfe00640 1 -- 172.21.15.40:0/2816522399 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18e010a940 msgr2=0x7f18e01111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.789+0000 7f18dfe00640 1 --2- 172.21.15.40:0/2816522399 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18e010a940 0x7f18e01111d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009860 tx=0x7f18d802a340 comp rx=0 tx=0).stop 2024-04-26T23:58:58.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.789+0000 7f18dfe00640 1 -- 172.21.15.40:0/2816522399 shutdown_connections 2024-04-26T23:58:58.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.789+0000 7f18dfe00640 1 --2- 172.21.15.40:0/2816522399 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18e010a940 0x7f18e01111d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.789+0000 7f18dfe00640 1 --2- 172.21.15.40:0/2816522399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e0109f80 0x7f18e010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.789+0000 7f18dfe00640 1 --2- 172.21.15.40:0/2816522399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e0108d90 0x7f18e01091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.789+0000 7f18dfe00640 1 -- 172.21.15.40:0/2816522399 >> 172.21.15.40:0/2816522399 conn(0x7f18e0077d30 msgr2=0x7f18e007a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:58.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.789+0000 7f18dfe00640 1 -- 172.21.15.40:0/2816522399 shutdown_connections 2024-04-26T23:58:58.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.789+0000 7f18dfe00640 1 -- 172.21.15.40:0/2816522399 wait complete. 2024-04-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.790+0000 7f18dfe00640 1 Processor -- start 2024-04-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.790+0000 7f18dfe00640 1 -- start start 2024-04-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18dfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e0108d90 0x7f18e019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18dfe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18e0109f80 0x7f18e019f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18dfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e010a940 0x7f18e01a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18dfe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f18e01148a0 con 0x7f18e0108d90 2024-04-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18dfe00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f18e0114720 con 0x7f18e0109f80 2024-04-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18dfe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18e0114a20 con 0x7f18e010a940 2024-04-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18dea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e0108d90 0x7f18e019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18df400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e010a940 0x7f18e01a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18de000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18e0109f80 0x7f18e019f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18df400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e010a940 0x7f18e01a6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43498/0 (socket says 172.21.15.40:43498) 2024-04-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18df400640 1 -- 172.21.15.40:0/3788774894 learned_addr learned my addr 172.21.15.40:0/3788774894 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18de000640 1 -- 172.21.15.40:0/3788774894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e010a940 msgr2=0x7f18e01a6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18de000640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e010a940 0x7f18e01a6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18de000640 1 -- 172.21.15.40:0/3788774894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e0108d90 msgr2=0x7f18e019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18de000640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e0108d90 0x7f18e019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18de000640 1 -- 172.21.15.40:0/3788774894 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e01a6ae0 con 0x7f18e0109f80 2024-04-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.791+0000 7f18df400640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e010a940 0x7f18e01a6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.792+0000 7f18de000640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18e0109f80 0x7f18e019f400 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f18c8009fd0 tx=0x7f18c800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.792+0000 7f18d7400640 1 -- 172.21.15.40:0/3788774894 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c8003970 con 0x7f18e0109f80 2024-04-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.792+0000 7f18d7400640 1 -- 172.21.15.40:0/3788774894 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18c8004020 con 0x7f18e0109f80 2024-04-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.792+0000 7f18dfe00640 1 -- 172.21.15.40:0/3788774894 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18e01a6dd0 con 0x7f18e0109f80 2024-04-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.792+0000 7f18d7400640 1 -- 172.21.15.40:0/3788774894 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c800b9b0 con 0x7f18e0109f80 2024-04-26T23:58:58.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.792+0000 7f18dfe00640 1 -- 172.21.15.40:0/3788774894 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e01a7290 con 0x7f18e0109f80 2024-04-26T23:58:58.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.796+0000 7f18e883c740 1 Processor -- start 2024-04-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.796+0000 7f18e883c740 1 -- start start 2024-04-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18e883c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da7276e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18e883c740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da727cb0 0x5621da72c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18e883c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da72ce20 0x5621da72f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18e883c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5621da68b600 con 0x5621da72ce20 2024-04-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18e883c740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5621da628100 con 0x5621da727cb0 2024-04-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18e883c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5621da6875c0 con 0x5621da7272c0 2024-04-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18d7400640 1 -- 172.21.15.40:0/3788774894 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f18c80041d0 con 0x7f18e0109f80 2024-04-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18df400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da7276e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18df400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da7276e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43512/0 (socket says 172.21.15.40:43512) 2024-04-26T23:58:58.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18df400640 1 -- 172.21.15.40:0/3190694731 learned_addr learned my addr 172.21.15.40:0/3190694731 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18dea00640 1 --2- 172.21.15.40:0/3190694731 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da727cb0 0x5621da72c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18de000640 1 --2- 172.21.15.40:0/3190694731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da72ce20 0x5621da72f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18df400640 1 -- 172.21.15.40:0/3190694731 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da727cb0 msgr2=0x5621da72c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18df400640 1 --2- 172.21.15.40:0/3190694731 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da727cb0 0x5621da72c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18df400640 1 -- 172.21.15.40:0/3190694731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da72ce20 msgr2=0x5621da72f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18df400640 1 --2- 172.21.15.40:0/3190694731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da72ce20 0x5621da72f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18df400640 1 -- 172.21.15.40:0/3190694731 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621da39d2d0 con 0x5621da7272c0 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18dea00640 1 --2- 172.21.15.40:0/3190694731 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da727cb0 0x5621da72c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.797+0000 7f18de000640 1 --2- 172.21.15.40:0/3190694731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da72ce20 0x5621da72f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.798+0000 7f18df400640 1 --2- 172.21.15.40:0/3190694731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da7276e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009490 tx=0x7f18d80094c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4395c0e52b003d1e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.798+0000 7f18cfe00640 1 -- 172.21.15.40:0/3190694731 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d8041020 con 0x5621da7272c0 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.798+0000 7f18cfe00640 1 -- 172.21.15.40:0/3190694731 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18d8004040 con 0x5621da7272c0 2024-04-26T23:58:58.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.798+0000 7f18cfe00640 1 -- 172.21.15.40:0/3190694731 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d8037040 con 0x5621da7272c0 2024-04-26T23:58:58.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18e883c740 1 -- 172.21.15.40:0/3190694731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 msgr2=0x5621da7276e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18e883c740 1 --2- 172.21.15.40:0/3190694731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da7276e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009490 tx=0x7f18d80094c0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18e883c740 1 -- 172.21.15.40:0/3190694731 shutdown_connections 2024-04-26T23:58:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18e883c740 1 --2- 172.21.15.40:0/3190694731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da72ce20 0x5621da72f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18e883c740 1 --2- 172.21.15.40:0/3190694731 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da727cb0 0x5621da72c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18e883c740 1 --2- 172.21.15.40:0/3190694731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da7276e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18e883c740 1 -- 172.21.15.40:0/3190694731 >> 172.21.15.40:0/3190694731 conn(0x5621da6bbd10 msgr2=0x5621da722d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18e883c740 1 -- 172.21.15.40:0/3190694731 shutdown_connections 2024-04-26T23:58:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18e883c740 1 -- 172.21.15.40:0/3190694731 wait complete. 2024-04-26T23:58:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18d7400640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f18c00a3ac0 0x7f18c00a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18d7400640 1 -- 172.21.15.40:0/3788774894 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f18c80d0800 con 0x7f18e0109f80 2024-04-26T23:58:58.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18dea00640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f18c00a3ac0 0x7f18c00a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18e883c740 1 -- start start 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.799+0000 7f18e883c740 1 Processor -- start 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18dea00640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f18c00a3ac0 0x7f18c00a5f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x5621da723bf0 tx=0x7f18d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18e883c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da731630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18e883c740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da727cb0 0x5621da731b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18e883c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da72ce20 0x5621da738bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18e883c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5621da72fd70 con 0x5621da72ce20 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18e883c740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5621da612f70 con 0x5621da727cb0 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18e883c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5621da72fef0 con 0x5621da7272c0 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18dea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da731630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18e883c740 10 client.0 _ll_register_callbacks cb 0x5621da681480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18df400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da72ce20 0x5621da738bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18dea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da731630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43520/0 (socket says 172.21.15.40:43520) 2024-04-26T23:58:58.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18dea00640 1 -- 172.21.15.40:0/2573560084 learned_addr learned my addr 172.21.15.40:0/2573560084 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.800+0000 7f18de000640 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da727cb0 0x5621da731b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18dea00640 1 -- 172.21.15.40:0/2573560084 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da727cb0 msgr2=0x5621da731b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18dea00640 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da727cb0 0x5621da731b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18dea00640 1 -- 172.21.15.40:0/2573560084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da72ce20 msgr2=0x5621da738bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18dea00640 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da72ce20 0x5621da738bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18dea00640 1 -- 172.21.15.40:0/2573560084 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621da4676b0 con 0x5621da7272c0 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18df400640 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da72ce20 0x5621da738bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18dea00640 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da731630 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f18d0013fd0 tx=0x7f18d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3af22c92f6d0db41 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18ccc00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18ccc00640 1 -- 172.21.15.40:0/2573560084 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d002b060 con 0x5621da7272c0 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18ccc00640 1 -- 172.21.15.40:0/2573560084 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f18d001eb90 con 0x5621da7272c0 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18e883c740 1 -- 172.21.15.40:0/2573560084 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18d8004040 con 0x5621da7272c0 2024-04-26T23:58:58.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18ccc00640 1 -- 172.21.15.40:0/2573560084 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f18d0032050 con 0x5621da7272c0 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18e883c740 10 client.24706 fetch_fsmap learned FSMap version 16 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.801+0000 7f18e883c740 1 -- 172.21.15.40:0/2573560084 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x5621da73bbd0 con 0x5621da7272c0 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.802+0000 7f18ccc00640 1 -- 172.21.15.40:0/2573560084 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18d8004040 con 0x5621da7272c0 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.802+0000 7f18ccc00640 1 -- 172.21.15.40:0/2573560084 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7f18d001a040 con 0x5621da7272c0 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.802+0000 7f18e883c740 10 client.24706 fetch_fsmap finished waiting for FSMap version 16 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.802+0000 7f18e883c740 10 client.24706 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.802+0000 7f18e883c740 20 client.24706 populate_metadata read hostname 'smithi040' 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.802+0000 7f18e883c740 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5621da73e2c0 0x5621da73f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.802+0000 7f18e883c740 4 client.24706 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.802+0000 7f18e883c740 1 -- 172.21.15.40:0/2573560084 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621da699210 con 0x5621da73e2c0 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.802+0000 7f18de000640 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5621da73e2c0 0x5621da73f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.803+0000 7f18de000640 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5621da73e2c0 0x5621da73f730 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26c5e5f879e34e67 server_cookie=7c5733601081032f in_seq=0 out_seq=0 2024-04-26T23:58:58.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.803+0000 7f18ccc00640 10 client.24706 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-26T23:58:58.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18ccc00640 1 -- 172.21.15.40:0/2573560084 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621da699210 con 0x5621da73e2c0 2024-04-26T23:58:58.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18ccc00640 10 client.24706 handle_command_reply: tid=0 2024-04-26T23:58:58.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18e883c740 1 client.24706 shutdown 2024-04-26T23:58:58.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18e883c740 1 -- 172.21.15.40:0/2573560084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 msgr2=0x5621da731630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18e883c740 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da731630 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f18d0013fd0 tx=0x7f18d00046c0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18e883c740 1 -- 172.21.15.40:0/2573560084 shutdown_connections 2024-04-26T23:58:58.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18e883c740 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5621da73e2c0 0x5621da73f730 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18e883c740 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da72ce20 0x5621da738bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18e883c740 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da727cb0 0x5621da731b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18e883c740 1 --2- 172.21.15.40:0/2573560084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da731630 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18ccc00640 0 client.24706 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-26T23:58:58.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18e883c740 1 -- 172.21.15.40:0/2573560084 >> 172.21.15.40:0/2573560084 conn(0x5621da6bbd10 msgr2=0x5621da721f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:58.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18e883c740 1 -- 172.21.15.40:0/2573560084 shutdown_connections 2024-04-26T23:58:58.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18e883c740 1 -- 172.21.15.40:0/2573560084 wait complete. 2024-04-26T23:58:58.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.804+0000 7f18e883c740 20 client.24706 trim_cache size 0 max 16384 2024-04-26T23:58:58.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.824+0000 7f18e883c740 1 Processor -- start 2024-04-26T23:58:58.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.824+0000 7f18e883c740 1 -- start start 2024-04-26T23:58:58.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.824+0000 7f18e883c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da729070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.824+0000 7f18e883c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 0x5621da7295b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.824+0000 7f18e883c740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da72ce20 0x5621da738b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.824+0000 7f18e883c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5621da73da70 con 0x5621da727cb0 2024-04-26T23:58:58.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.824+0000 7f18e883c740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5621da73e140 con 0x5621da72ce20 2024-04-26T23:58:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.824+0000 7f18e883c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5621da73e810 con 0x5621da7272c0 2024-04-26T23:58:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.825+0000 7f18dea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da729070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.825+0000 7f18de000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 0x5621da7295b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.825+0000 7f18dea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da729070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43536/0 (socket says 172.21.15.40:43536) 2024-04-26T23:58:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.825+0000 7f18dea00640 1 -- 172.21.15.40:0/793305430 learned_addr learned my addr 172.21.15.40:0/793305430 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.825+0000 7f18df400640 1 --2- 172.21.15.40:0/793305430 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da72ce20 0x5621da738b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.825+0000 7f18de000640 1 -- 172.21.15.40:0/793305430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 msgr2=0x5621da729070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:58:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.825+0000 7f18de000640 1 --2- 172.21.15.40:0/793305430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da729070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.825+0000 7f18de000640 1 -- 172.21.15.40:0/793305430 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da72ce20 msgr2=0x5621da738b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:58:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.825+0000 7f18de000640 1 --2- 172.21.15.40:0/793305430 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da72ce20 0x5621da738b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.825+0000 7f18de000640 1 -- 172.21.15.40:0/793305430 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621da7239f0 con 0x5621da727cb0 2024-04-26T23:58:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.825+0000 7f18de000640 1 --2- 172.21.15.40:0/793305430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 0x5621da7295b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f18c80ce060 tx=0x7f18c80d0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c87fc68f57cb7d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.825+0000 7f18cf400640 1 -- 172.21.15.40:0/793305430 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c809ab40 con 0x5621da727cb0 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.826+0000 7f18cf400640 1 -- 172.21.15.40:0/793305430 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18c80ce170 con 0x5621da727cb0 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.826+0000 7f18cf400640 1 -- 172.21.15.40:0/793305430 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c8099040 con 0x5621da727cb0 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.826+0000 7f18e883c740 1 -- 172.21.15.40:0/793305430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 msgr2=0x5621da7295b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.826+0000 7f18e883c740 1 --2- 172.21.15.40:0/793305430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 0x5621da7295b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f18c80ce060 tx=0x7f18c80d0040 comp rx=0 tx=0).stop 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.826+0000 7f18e883c740 1 -- 172.21.15.40:0/793305430 shutdown_connections 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.826+0000 7f18e883c740 1 --2- 172.21.15.40:0/793305430 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da72ce20 0x5621da738b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.826+0000 7f18e883c740 1 --2- 172.21.15.40:0/793305430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 0x5621da7295b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.826+0000 7f18e883c740 1 --2- 172.21.15.40:0/793305430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da7272c0 0x5621da729070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.826+0000 7f18e883c740 1 -- 172.21.15.40:0/793305430 >> 172.21.15.40:0/793305430 conn(0x5621da6bbd10 msgr2=0x5621da728540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.826+0000 7f18e883c740 1 -- 172.21.15.40:0/793305430 shutdown_connections 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.826+0000 7f18e883c740 1 -- 172.21.15.40:0/793305430 wait complete. 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18e883c740 1 -- start start 2024-04-26T23:58:58.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18e883c740 1 Processor -- start 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18e883c740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da7272c0 0x5621da736b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18e883c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 0x5621da7a0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18e883c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da72ce20 0x5621da7a27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18e883c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5621da73f5b0 con 0x5621da727cb0 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18e883c740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5621da73eee0 con 0x5621da7272c0 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18e883c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5621da73fc80 con 0x5621da72ce20 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18e883c740 10 client.0 _ll_register_callbacks cb 0x7f18d0009950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18de000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 0x5621da7a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18de000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 0x5621da7a0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52958/0 (socket says 172.21.15.40:52958) 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18de000640 1 -- 172.21.15.40:0/3126188299 learned_addr learned my addr 172.21.15.40:0/3126188299 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18df400640 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da72ce20 0x5621da7a27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.827+0000 7f18dea00640 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da7272c0 0x5621da736b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.828+0000 7f18de000640 1 -- 172.21.15.40:0/3126188299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da72ce20 msgr2=0x5621da7a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:58:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.828+0000 7f18de000640 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da72ce20 0x5621da7a27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.828+0000 7f18de000640 1 -- 172.21.15.40:0/3126188299 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da7272c0 msgr2=0x5621da736b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:58:58.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.828+0000 7f18de000640 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da7272c0 0x5621da736b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.828+0000 7f18de000640 1 -- 172.21.15.40:0/3126188299 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621da7a2e90 con 0x5621da727cb0 2024-04-26T23:58:58.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.828+0000 7f18de000640 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 0x5621da7a0250 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f18c80a4f80 tx=0x7f18c80a0d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d998264fc1bbcfc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:58:58.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.828+0000 7f18ccc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-26T23:58:58.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.828+0000 7f18ccc00640 1 -- 172.21.15.40:0/3126188299 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c8099020 con 0x5621da727cb0 2024-04-26T23:58:58.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.828+0000 7f18ccc00640 1 -- 172.21.15.40:0/3126188299 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f18c80870b0 con 0x5621da727cb0 2024-04-26T23:58:58.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.828+0000 7f18e883c740 1 -- 172.21.15.40:0/3126188299 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18c80ce170 con 0x5621da727cb0 2024-04-26T23:58:58.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.828+0000 7f18ccc00640 1 -- 172.21.15.40:0/3126188299 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f18c809db70 con 0x5621da727cb0 2024-04-26T23:58:58.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.829+0000 7f18e883c740 10 client.14574 fetch_fsmap learned FSMap version 16 2024-04-26T23:58:58.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.829+0000 7f18e883c740 1 -- 172.21.15.40:0/3126188299 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x5621da7a4190 con 0x5621da727cb0 2024-04-26T23:58:58.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.829+0000 7f18ccc00640 1 -- 172.21.15.40:0/3126188299 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18c80ce170 con 0x5621da727cb0 2024-04-26T23:58:58.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.829+0000 7f18ccc00640 1 -- 172.21.15.40:0/3126188299 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7f18c800b040 con 0x5621da727cb0 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.829+0000 7f18e883c740 10 client.14574 fetch_fsmap finished waiting for FSMap version 16 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.829+0000 7f18e883c740 10 client.14574 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.829+0000 7f18e883c740 20 client.14574 populate_metadata read hostname 'smithi040' 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.829+0000 7f18e883c740 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5621da7a8920 0x5621da7a7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.829+0000 7f18e883c740 4 client.14574 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.829+0000 7f18e883c740 1 -- 172.21.15.40:0/3126188299 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "status"}) -- 0x5621da693080 con 0x5621da7a8920 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.829+0000 7f18dea00640 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5621da7a8920 0x5621da7a7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18dea00640 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5621da7a8920 0x5621da7a7630 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14178a335a86a3a6 server_cookie=3629e60818d3512b in_seq=0 out_seq=0 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18ccc00640 10 client.14574 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18ccc00640 1 -- 172.21.15.40:0/3126188299 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5621da693080 con 0x5621da7a8920 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18ccc00640 10 client.14574 handle_command_reply: tid=0 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18e883c740 1 client.14574 shutdown 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18e883c740 1 -- 172.21.15.40:0/3126188299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 msgr2=0x5621da7a0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18e883c740 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 0x5621da7a0250 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f18c80a4f80 tx=0x7f18c80a0d70 comp rx=0 tx=0).stop 2024-04-26T23:58:58.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18e883c740 1 -- 172.21.15.40:0/3126188299 shutdown_connections 2024-04-26T23:58:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18e883c740 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5621da7a8920 0x5621da7a7630 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18e883c740 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5621da72ce20 0x5621da7a27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18e883c740 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5621da727cb0 0x5621da7a0250 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18e883c740 1 --2- 172.21.15.40:0/3126188299 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5621da7272c0 0x5621da736b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18ccc00640 0 client.14574 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-26T23:58:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.830+0000 7f18e883c740 1 -- 172.21.15.40:0/3126188299 >> 172.21.15.40:0/3126188299 conn(0x5621da6bbd10 msgr2=0x5621da729070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18e883c740 1 -- 172.21.15.40:0/3126188299 shutdown_connections 2024-04-26T23:58:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18e883c740 1 -- 172.21.15.40:0/3126188299 wait complete. 2024-04-26T23:58:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18e883c740 20 client.14574 trim_cache size 0 max 16384 2024-04-26T23:58:58.837 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:58:58.837 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "78cff6ea-0427-11ef-bc93-c7b262605968", 2024-04-26T23:58:58.838 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-26T23:58:58.838 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24590, 2024-04-26T23:58:58.838 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-26T23:58:58.838 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-26T23:58:58.838 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-26T23:58:58.838 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 7.7166730039999996, 2024-04-26T23:58:58.838 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 16, 2024-04-26T23:58:58.838 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 90, 2024-04-26T23:58:58.838 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-26T23:58:58.838 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 14.124662220999999 2024-04-26T23:58:58.838 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:58:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18dfe00640 1 -- 172.21.15.40:0/3788774894 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f18c00a3ac0 msgr2=0x7f18c00a5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18dfe00640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f18c00a3ac0 0x7f18c00a5f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x5621da723bf0 tx=0x7f18d0005f90 comp rx=0 tx=0).stop 2024-04-26T23:58:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18dfe00640 1 -- 172.21.15.40:0/3788774894 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18e0109f80 msgr2=0x7f18e019f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:58:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18dfe00640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18e0109f80 0x7f18e019f400 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f18c8009fd0 tx=0x7f18c800c880 comp rx=0 tx=0).stop 2024-04-26T23:58:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18dfe00640 1 -- 172.21.15.40:0/3788774894 shutdown_connections 2024-04-26T23:58:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18dfe00640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f18c00a3ac0 0x7f18c00a5f80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18dfe00640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e010a940 0x7f18e01a6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18dfe00640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f18e0109f80 0x7f18e019f400 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18dfe00640 1 --2- 172.21.15.40:0/3788774894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e0108d90 0x7f18e019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:58:58.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.831+0000 7f18dfe00640 1 -- 172.21.15.40:0/3788774894 >> 172.21.15.40:0/3788774894 conn(0x7f18e0077d30 msgr2=0x7f18e010d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:58:58.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.832+0000 7f18dfe00640 1 -- 172.21.15.40:0/3788774894 shutdown_connections 2024-04-26T23:58:58.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:58:58.832+0000 7f18dfe00640 1 -- 172.21.15.40:0/3788774894 wait complete. 2024-04-26T23:58:58.849 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T23:58:58.849 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:58:58.850 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T23:58:59.302 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:58:59.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:58:59 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1330737537' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:59.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:58:59 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1330737537' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:58:59.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:58:59 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1330737537' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:00.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:00 smithi080 ceph-mon[32362]: pgmap v327: 129 pgs: 6 creating+peering, 123 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 554 B/s wr, 1 op/s 2024-04-26T23:59:00.583 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:00 smithi040 ceph-mon[28095]: pgmap v327: 129 pgs: 6 creating+peering, 123 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 554 B/s wr, 1 op/s 2024-04-26T23:59:00.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:00 smithi103 ceph-mon[32952]: pgmap v327: 129 pgs: 6 creating+peering, 123 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 554 B/s wr, 1 op/s 2024-04-26T23:59:01.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.983+0000 7f2e11800700 1 -- 172.21.15.40:0/2637223453 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e0c103e70 msgr2=0x7f2e0c110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.983+0000 7f2e11800700 1 --2- 172.21.15.40:0/2637223453 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e0c103e70 0x7f2e0c110320 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2e040099c0 tx=0x7f2e04009cd0 comp rx=0 tx=0).stop 2024-04-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.983+0000 7f2e11800700 1 -- 172.21.15.40:0/2637223453 shutdown_connections 2024-04-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.983+0000 7f2e11800700 1 --2- 172.21.15.40:0/2637223453 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e0c103e70 0x7f2e0c110320 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.983+0000 7f2e11800700 1 --2- 172.21.15.40:0/2637223453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c109550 0x7f2e0c109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.983+0000 7f2e11800700 1 --2- 172.21.15.40:0/2637223453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e0c1034b0 0x7f2e0c103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.983+0000 7f2e11800700 1 -- 172.21.15.40:0/2637223453 >> 172.21.15.40:0/2637223453 conn(0x7f2e0c07a610 msgr2=0x7f2e0c07aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.983+0000 7f2e11800700 1 -- 172.21.15.40:0/2637223453 shutdown_connections 2024-04-26T23:59:01.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.983+0000 7f2e11800700 1 -- 172.21.15.40:0/2637223453 wait complete. 2024-04-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.985+0000 7f2e11800700 1 Processor -- start 2024-04-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.985+0000 7f2e11800700 1 -- start start 2024-04-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.985+0000 7f2e11800700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e0c1034b0 0x7f2e0c1a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.985+0000 7f2e11800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c103e70 0x7f2e0c1a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.985+0000 7f2e11800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e0c109550 0x7f2e0c19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.985+0000 7f2e11800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0c113630 con 0x7f2e0c103e70 2024-04-26T23:59:01.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.985+0000 7f2e11800700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0c1134b0 con 0x7f2e0c1034b0 2024-04-26T23:59:01.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.985+0000 7f2e11800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0c1137b0 con 0x7f2e0c109550 2024-04-26T23:59:01.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.986+0000 7f2e02000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c103e70 0x7f2e0c1a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.986+0000 7f2e02000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c103e70 0x7f2e0c1a1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32836/0 (socket says 172.21.15.40:32836) 2024-04-26T23:59:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.986+0000 7f2e02000700 1 -- 172.21.15.40:0/3420306928 learned_addr learned my addr 172.21.15.40:0/3420306928 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.986+0000 7f2e0aa00700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e0c1034b0 0x7f2e0c1a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.986+0000 7f2e0b400700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e0c109550 0x7f2e0c19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.986+0000 7f2e0aa00700 1 -- 172.21.15.40:0/3420306928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e0c109550 msgr2=0x7f2e0c19bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.986+0000 7f2e0aa00700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e0c109550 0x7f2e0c19bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.986+0000 7f2e0aa00700 1 -- 172.21.15.40:0/3420306928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c103e70 msgr2=0x7f2e0c1a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.986+0000 7f2e0aa00700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c103e70 0x7f2e0c1a1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.986+0000 7f2e0aa00700 1 -- 172.21.15.40:0/3420306928 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0c19c410 con 0x7f2e0c1034b0 2024-04-26T23:59:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.986+0000 7f2e02000700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c103e70 0x7f2e0c1a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.987+0000 7f2e0aa00700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e0c1034b0 0x7f2e0c1a1a60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00e9f0 tx=0x7f2dfc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:01.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.987+0000 7f2e03e00700 1 -- 172.21.15.40:0/3420306928 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc00c9c0 con 0x7f2e0c1034b0 2024-04-26T23:59:01.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.987+0000 7f2e11800700 1 -- 172.21.15.40:0/3420306928 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e0c19c700 con 0x7f2e0c1034b0 2024-04-26T23:59:01.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.987+0000 7f2e03e00700 1 -- 172.21.15.40:0/3420306928 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dfc00cb60 con 0x7f2e0c1034b0 2024-04-26T23:59:01.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.987+0000 7f2e11800700 1 -- 172.21.15.40:0/3420306928 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e0c1a89e0 con 0x7f2e0c1034b0 2024-04-26T23:59:01.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.987+0000 7f2e03e00700 1 -- 172.21.15.40:0/3420306928 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc018420 con 0x7f2e0c1034b0 2024-04-26T23:59:01.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.988+0000 7f2e11800700 1 -- 172.21.15.40:0/3420306928 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de8005330 con 0x7f2e0c1034b0 2024-04-26T23:59:01.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.990+0000 7f2e03e00700 1 -- 172.21.15.40:0/3420306928 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2dfc010430 con 0x7f2e0c1034b0 2024-04-26T23:59:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.992+0000 7f2e03e00700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2df00a3c40 0x7f2df00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.992+0000 7f2e03e00700 1 -- 172.21.15.40:0/3420306928 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f2dfc0d0810 con 0x7f2e0c1034b0 2024-04-26T23:59:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.992+0000 7f2e02000700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2df00a3c40 0x7f2df00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:01.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.993+0000 7f2e02000700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2df00a3c40 0x7f2df00a6100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c19d280 tx=0x7f2df400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:01.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:01.994+0000 7f2e03e00700 1 -- 172.21.15.40:0/3420306928 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2dfc09cf50 con 0x7f2e0c1034b0 2024-04-26T23:59:02.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.275+0000 7f2e11800700 1 -- 172.21.15.40:0/3420306928 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2de80058f0 con 0x7f2e0c1034b0 2024-04-26T23:59:02.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.276+0000 7f2e03e00700 1 -- 172.21.15.40:0/3420306928 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7f2dfc09c850 con 0x7f2e0c1034b0 2024-04-26T23:59:02.278 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T23:59:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.279+0000 7f2e11800700 1 -- 172.21.15.40:0/3420306928 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2df00a3c40 msgr2=0x7f2df00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.279+0000 7f2e11800700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2df00a3c40 0x7f2df00a6100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c19d280 tx=0x7f2df400b040 comp rx=0 tx=0).stop 2024-04-26T23:59:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.279+0000 7f2e11800700 1 -- 172.21.15.40:0/3420306928 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e0c1034b0 msgr2=0x7f2e0c1a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.279+0000 7f2e11800700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e0c1034b0 0x7f2e0c1a1a60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00e9f0 tx=0x7f2dfc00ed00 comp rx=0 tx=0).stop 2024-04-26T23:59:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.279+0000 7f2e11800700 1 -- 172.21.15.40:0/3420306928 shutdown_connections 2024-04-26T23:59:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.279+0000 7f2e11800700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2df00a3c40 0x7f2df00a6100 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.279+0000 7f2e11800700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e0c109550 0x7f2e0c19bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.279+0000 7f2e11800700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2e0c1034b0 0x7f2e0c1a1a60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.279+0000 7f2e11800700 1 --2- 172.21.15.40:0/3420306928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c103e70 0x7f2e0c1a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.279+0000 7f2e11800700 1 -- 172.21.15.40:0/3420306928 >> 172.21.15.40:0/3420306928 conn(0x7f2e0c07a610 msgr2=0x7f2e0c0758e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:02.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.279+0000 7f2e11800700 1 -- 172.21.15.40:0/3420306928 shutdown_connections 2024-04-26T23:59:02.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:02.279+0000 7f2e11800700 1 -- 172.21.15.40:0/3420306928 wait complete. 2024-04-26T23:59:02.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:02 smithi080 ceph-mon[32362]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 470 B/s wr, 1 op/s 2024-04-26T23:59:02.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:02 smithi103 ceph-mon[32952]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 470 B/s wr, 1 op/s 2024-04-26T23:59:02.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:02 smithi040 ceph-mon[28095]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 470 B/s wr, 1 op/s 2024-04-26T23:59:03.324 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-26T23:59:03.324 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:59:03.324 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-26T23:59:03.324 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-26T23:59:03.324 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:59:03.324 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-26T23:59:03.324 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T23:59:03.324 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:59:03.324 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-26T23:59:03.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:03.325 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-26T23:59:03.384 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:59:03.384 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:59:03.384 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:59:03.384 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:59:03.384 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-26T23:59:03.384 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:59:03.384 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:59:03.384 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:59:03.384 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-26T23:59:03.384 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:59:03.384 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:59:03.385 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-26T23:59:03.385 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:59:03.385 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-26T23:59:03.385 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-26T23:59:03.385 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T23:59:03.385 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42103sec preferred_lft 42103sec 2024-04-26T23:59:03.385 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-26T23:59:03.385 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:59:03.385 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:59:03.385 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-26T23:59:03.385 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-26T23:59:03.387 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T23:59:03.387 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:59:03.387 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:59:03.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add name ceph-brx type bridge 2024-04-26T23:59:03.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr flush dev ceph-brx 2024-04-26T23:59:03.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set ceph-brx up 2024-04-26T23:59:03.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T23:59:03.387 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:59:03.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:59:03.524 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:03 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3420306928' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T23:59:03.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:03 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3420306928' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T23:59:03.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:03 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3420306928' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T23:59:03.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:59:03.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:03.890 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:59:03.962 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-26T23:59:03.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:03.966 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-26T23:59:04.027 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-26T23:59:04.027 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:59:04.027 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T23:59:04.028 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T23:59:04.028 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:59:04.028 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:59:04.028 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:59:04.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T23:59:04.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T23:59:04.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T23:59:04.028 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:59:04.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:59:04.434 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:04 smithi040 ceph-mon[28095]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-26T23:59:04.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:04 smithi080 ceph-mon[32362]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-26T23:59:04.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:04 smithi103 ceph-mon[32952]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-26T23:59:04.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:59:04.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:04.786 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-26T23:59:04.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:04.846 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list-id 2024-04-26T23:59:04.905 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:59:04.905 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:59:04.905 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:59:04.905 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-26T23:59:04.905 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:59:05.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:59:05.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:59:05.104 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-26T23:59:05.104 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:59:05.104 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:59:05.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-26T23:59:05.105 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T23:59:05.105 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-26T23:59:05.105 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-26T23:59:05.105 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-26T23:59:05.105 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:59:05.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:59:05.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:59:05.404 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:59:05.404 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-26T23:59:05.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-04-26T23:59:05.405 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T23:59:05.405 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-26T23:59:05.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:59:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:59:05.619 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-26T23:59:05.619 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:59:05.619 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:59:05.669 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-26T23:59:05.670 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:59:05.670 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:59:05.722 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-04-26T23:59:05.803 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T23:59:05.854 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-04-26T23:59:05.854 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-04-26T23:59:05.854 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-04-26T23:59:05.854 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-04-26T23:59:05.854 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-04-26T23:59:05.854 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-04-26T23:59:05.855 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c2a5e0ecabbffbe50daf79bb4dd9b1f97a63fbba6f26ffb80d80aa92c95199a2/merged 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/9e2144da9c4933f0bf9eb05807da0e4d67dea8b3b9d230ffb61e68288323f6f6/merged 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/0 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/004f1f504a36ab5a96264d9b5d45ba49a8d34bbb8bd760784195321717d5f765/merged 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/3f98da9bee0d280dbac377691d1f29d1ce583aeb32d99c6c35e7e32292430ef3/merged 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/34dd8ddf84282c0de78be344446370c99f7548daa0b16bbab5cd5694f902df7a/merged 2024-04-26T23:59:05.856 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/e39e83c1a5c5cd4a2a3f22e4d176b41c3d91a9e35bf37754178c0cc5e14c2453/merged 2024-04-26T23:59:05.857 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/8b12b67f9470e662019f45e3e3150907d8f72c9edb9de7f3ec41ae36c7d9a290/merged 2024-04-26T23:59:05.857 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c9ee414d1145c64f444b9fbb4b5f96c4acbf78ce8d7416b6535f673f03c3eab5/merged 2024-04-26T23:59:05.857 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/946c2232edbce6b0a8a5c05cae364723d3031108e21066c685df03abea323aea/merged 2024-04-26T23:59:05.857 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/2bea66009b5851fe5a379f55124689887726bfa30b845a36212994fe00d968e8/merged 2024-04-26T23:59:05.857 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-04-26T23:59:05.857 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:59:05.857 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:59:05.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:05.858 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-04-26T23:59:05.905 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-26T23:59:05.905 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-26T23:59:05.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-04-26T23:59:05.983 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T23:59:06.030 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-04-26T23:59:06.030 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-04-26T23:59:06.030 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-04-26T23:59:06.030 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-04-26T23:59:06.030 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-04-26T23:59:06.030 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-04-26T23:59:06.031 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-04-26T23:59:06.031 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-04-26T23:59:06.031 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-04-26T23:59:06.031 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-04-26T23:59:06.031 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-04-26T23:59:06.031 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-04-26T23:59:06.031 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:59:06.031 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-04-26T23:59:06.031 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-04-26T23:59:06.031 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-04-26T23:59:06.031 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-04-26T23:59:06.031 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-04-26T23:59:06.032 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-04-26T23:59:06.032 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T23:59:06.032 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T23:59:06.032 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T23:59:06.032 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T23:59:06.032 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-04-26T23:59:06.032 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-04-26T23:59:06.032 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay 2024-04-26T23:59:06.032 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c2a5e0ecabbffbe50daf79bb4dd9b1f97a63fbba6f26ffb80d80aa92c95199a2/merged 2024-04-26T23:59:06.032 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/9e2144da9c4933f0bf9eb05807da0e4d67dea8b3b9d230ffb61e68288323f6f6/merged 2024-04-26T23:59:06.032 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/0 2024-04-26T23:59:06.032 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:59:06.033 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/004f1f504a36ab5a96264d9b5d45ba49a8d34bbb8bd760784195321717d5f765/merged 2024-04-26T23:59:06.033 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/3f98da9bee0d280dbac377691d1f29d1ce583aeb32d99c6c35e7e32292430ef3/merged 2024-04-26T23:59:06.033 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/34dd8ddf84282c0de78be344446370c99f7548daa0b16bbab5cd5694f902df7a/merged 2024-04-26T23:59:06.033 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/e39e83c1a5c5cd4a2a3f22e4d176b41c3d91a9e35bf37754178c0cc5e14c2453/merged 2024-04-26T23:59:06.033 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/8b12b67f9470e662019f45e3e3150907d8f72c9edb9de7f3ec41ae36c7d9a290/merged 2024-04-26T23:59:06.033 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c9ee414d1145c64f444b9fbb4b5f96c4acbf78ce8d7416b6535f673f03c3eab5/merged 2024-04-26T23:59:06.033 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/946c2232edbce6b0a8a5c05cae364723d3031108e21066c685df03abea323aea/merged 2024-04-26T23:59:06.033 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/2bea66009b5851fe5a379f55124689887726bfa30b845a36212994fe00d968e8/merged 2024-04-26T23:59:06.033 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-04-26T23:59:06.033 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:59:06.033 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:59:06.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:06.035 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-04-26T23:59:06.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:2024-04-26T23:59:06.108+0000 7fd3a66f7480 -1 init, newargv = 0x5641a48b8860 newargc=15 2024-04-26T23:59:06.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:ceph-fuse[69638]: starting ceph client 2024-04-26T23:59:06.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:ceph-fuse[69638]: starting fuse 2024-04-26T23:59:06.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:06 smithi080 ceph-mon[32362]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 214 B/s wr, 0 op/s 2024-04-26T23:59:06.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:06 smithi103 ceph-mon[32952]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 214 B/s wr, 0 op/s 2024-04-26T23:59:06.680 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:06 smithi040 ceph-mon[28095]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 214 B/s wr, 0 op/s 2024-04-26T23:59:07.222 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-04-26T23:59:07.305 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T23:59:07.354 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-04-26T23:59:07.354 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-04-26T23:59:07.354 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-04-26T23:59:07.354 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-04-26T23:59:07.355 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c2a5e0ecabbffbe50daf79bb4dd9b1f97a63fbba6f26ffb80d80aa92c95199a2/merged 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/9e2144da9c4933f0bf9eb05807da0e4d67dea8b3b9d230ffb61e68288323f6f6/merged 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/0 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/004f1f504a36ab5a96264d9b5d45ba49a8d34bbb8bd760784195321717d5f765/merged 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/3f98da9bee0d280dbac377691d1f29d1ce583aeb32d99c6c35e7e32292430ef3/merged 2024-04-26T23:59:07.356 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/34dd8ddf84282c0de78be344446370c99f7548daa0b16bbab5cd5694f902df7a/merged 2024-04-26T23:59:07.357 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/e39e83c1a5c5cd4a2a3f22e4d176b41c3d91a9e35bf37754178c0cc5e14c2453/merged 2024-04-26T23:59:07.357 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/8b12b67f9470e662019f45e3e3150907d8f72c9edb9de7f3ec41ae36c7d9a290/merged 2024-04-26T23:59:07.357 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c9ee414d1145c64f444b9fbb4b5f96c4acbf78ce8d7416b6535f673f03c3eab5/merged 2024-04-26T23:59:07.357 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/946c2232edbce6b0a8a5c05cae364723d3031108e21066c685df03abea323aea/merged 2024-04-26T23:59:07.357 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/2bea66009b5851fe5a379f55124689887726bfa30b845a36212994fe00d968e8/merged 2024-04-26T23:59:07.357 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-04-26T23:59:07.357 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:59:07.357 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:59:07.357 INFO:teuthology.orchestra.run.smithi040.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-26T23:59:07.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:07.358 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-04-26T23:59:07.404 INFO:teuthology.orchestra.run.smithi040.stdout:114 2024-04-26T23:59:07.405 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-04-26T23:59:07.405 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-04-26T23:59:07.405 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-04-26T23:59:07.405 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-04-26T23:59:07.405 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-04-26T23:59:07.405 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-04-26T23:59:07.405 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:59:07.405 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-04-26T23:59:07.405 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-26T23:59:07.405 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-04-26T23:59:07.405 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> 2024-04-26T23:59:07.406 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.admin")) 2024-04-26T23:59:07.407 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-04-26T23:59:07.614 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.admin.69638.asok 2024-04-26T23:59:07.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:59:07.633 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.69638.asok 2024-04-26T23:59:07.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:07.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.69638.asok status 2024-04-26T23:59:07.850 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-26T23:59:07.851 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-04-26T23:59:07.851 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "bc612ffb9c1d5bc27536bc917a705262a36d4387", 2024-04-26T23:59:07.851 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T23:59:07.851 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "admin", 2024-04-26T23:59:07.851 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-04-26T23:59:07.852 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-26T23:59:07.852 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "69638", 2024-04-26T23:59:07.852 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-04-26T23:59:07.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:59:07.852 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-04-26T23:59:07.853 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-04-26T23:59:07.853 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24724, 2024-04-26T23:59:07.853 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-04-26T23:59:07.853 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-04-26T23:59:07.853 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-04-26T23:59:07.854 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 24724 2024-04-26T23:59:07.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:59:07.854 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-04-26T23:59:07.854 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-26T23:59:07.854 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-26T23:59:07.855 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3332661269 2024-04-26T23:59:07.855 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-26T23:59:07.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:59:07.855 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-04-26T23:59:07.855 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-26T23:59:07.856 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-26T23:59:07.856 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3332661269 2024-04-26T23:59:07.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-26T23:59:07.856 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.24724 172.21.15.40:0/3332661269", 2024-04-26T23:59:07.856 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "172.21.15.40:0/3332661269", 2024-04-26T23:59:07.857 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-04-26T23:59:07.857 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 16, 2024-04-26T23:59:07.857 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 90, 2024-04-26T23:59:07.857 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-04-26T23:59:07.857 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-04-26T23:59:07.857 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-04-26T23:59:07.858 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-26T23:59:07.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:07.864 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-04-26T23:59:07.864 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:59:07.916 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-04-26T23:59:07.916 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:59:07.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:07.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:59:08.006 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-26T23:59:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:59:08.154 INFO:teuthology.orchestra.run.smithi040.stdout:# file: . 2024-04-26T23:59:08.154 INFO:teuthology.orchestra.run.smithi040.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-26T23:59:08.154 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:59:08.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:59:08.164 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:59:08.164 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:59:08.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:59:08.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.425+0000 7fa479a00640 1 -- 172.21.15.40:0/3294108531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa474108f90 msgr2=0x7fa4741093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.425+0000 7fa479a00640 1 --2- 172.21.15.40:0/3294108531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa474108f90 0x7fa4741093b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa464009860 tx=0x7fa46402a340 comp rx=0 tx=0).stop 2024-04-26T23:59:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.425+0000 7fa479a00640 1 -- 172.21.15.40:0/3294108531 shutdown_connections 2024-04-26T23:59:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.425+0000 7fa479a00640 1 --2- 172.21.15.40:0/3294108531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa47410ab40 0x7fa4741113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.425+0000 7fa479a00640 1 --2- 172.21.15.40:0/3294108531 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa47410a180 0x7fa47410a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.425+0000 7fa479a00640 1 --2- 172.21.15.40:0/3294108531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa474108f90 0x7fa4741093b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.425+0000 7fa479a00640 1 -- 172.21.15.40:0/3294108531 >> 172.21.15.40:0/3294108531 conn(0x7fa474077f40 msgr2=0x7fa47407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:08.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.425+0000 7fa479a00640 1 -- 172.21.15.40:0/3294108531 shutdown_connections 2024-04-26T23:59:08.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.425+0000 7fa479a00640 1 -- 172.21.15.40:0/3294108531 wait complete. 2024-04-26T23:59:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.426+0000 7fa479a00640 1 Processor -- start 2024-04-26T23:59:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.426+0000 7fa479a00640 1 -- start start 2024-04-26T23:59:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa479a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa474108f90 0x7fa47419f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa479a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa47410a180 0x7fa47419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa479a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa47410ab40 0x7fa4741a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa479a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa474114ad0 con 0x7fa474108f90 2024-04-26T23:59:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa479a00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa474114950 con 0x7fa47410ab40 2024-04-26T23:59:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa479a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa474114c50 con 0x7fa47410a180 2024-04-26T23:59:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa472a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa474108f90 0x7fa47419f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa472a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa474108f90 0x7fa47419f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32866/0 (socket says 172.21.15.40:32866) 2024-04-26T23:59:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa472a00640 1 -- 172.21.15.40:0/2873306641 learned_addr learned my addr 172.21.15.40:0/2873306641 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa46be00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa47410a180 0x7fa47419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa473400640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa47410ab40 0x7fa4741a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa472a00640 1 -- 172.21.15.40:0/2873306641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa47410a180 msgr2=0x7fa47419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa472a00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa47410a180 0x7fa47419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.427+0000 7fa472a00640 1 -- 172.21.15.40:0/2873306641 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa47410ab40 msgr2=0x7fa4741a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.428+0000 7fa472a00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa47410ab40 0x7fa4741a6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.428+0000 7fa472a00640 1 -- 172.21.15.40:0/2873306641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4741a6e00 con 0x7fa474108f90 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.428+0000 7fa46be00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa47410a180 0x7fa47419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.428+0000 7fa473400640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa47410ab40 0x7fa4741a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.428+0000 7fa472a00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa474108f90 0x7fa47419f1e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa464005940 tx=0x7fa464032dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.428+0000 7fa46b400640 1 -- 172.21.15.40:0/2873306641 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464040020 con 0x7fa474108f90 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.428+0000 7fa46b400640 1 -- 172.21.15.40:0/2873306641 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa464004040 con 0x7fa474108f90 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.428+0000 7fa46b400640 1 -- 172.21.15.40:0/2873306641 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464037040 con 0x7fa474108f90 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.428+0000 7fa479a00640 1 -- 172.21.15.40:0/2873306641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4741a7090 con 0x7fa474108f90 2024-04-26T23:59:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.428+0000 7fa479a00640 1 -- 172.21.15.40:0/2873306641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4741a7530 con 0x7fa474108f90 2024-04-26T23:59:08.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.429+0000 7fa479a00640 1 -- 172.21.15.40:0/2873306641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa474110330 con 0x7fa474108f90 2024-04-26T23:59:08.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.432+0000 7fa46b400640 1 -- 172.21.15.40:0/2873306641 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa464043020 con 0x7fa474108f90 2024-04-26T23:59:08.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.434+0000 7fa46b400640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa4580a3c30 0x7fa4580a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:08.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.434+0000 7fa46b400640 1 -- 172.21.15.40:0/2873306641 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa4640f0b90 con 0x7fa474108f90 2024-04-26T23:59:08.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.434+0000 7fa46be00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa4580a3c30 0x7fa4580a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:08.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.435+0000 7fa46be00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa4580a3c30 0x7fa4580a60f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa4741a06f0 tx=0x7fa45c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.437+0000 7fa46b400640 1 -- 172.21.15.40:0/2873306641 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa46404a020 con 0x7fa474108f90 2024-04-26T23:59:08.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:08 smithi080 ceph-mon[32362]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 195 B/s wr, 0 op/s 2024-04-26T23:59:08.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:08 smithi103 ceph-mon[32952]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 195 B/s wr, 0 op/s 2024-04-26T23:59:08.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.647+0000 7fa479a00640 1 -- 172.21.15.40:0/2873306641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa474063530 con 0x7fa474108f90 2024-04-26T23:59:08.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.651+0000 7fa46b400640 1 -- 172.21.15.40:0/2873306641 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24090 (secure 0 0 0) 0x7fa4640bcd00 con 0x7fa474108f90 2024-04-26T23:59:08.654 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:59:08.654 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:58:55.649955+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:59:08.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.654+0000 7fa479a00640 1 -- 172.21.15.40:0/2873306641 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa4580a3c30 msgr2=0x7fa4580a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:08.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.654+0000 7fa479a00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa4580a3c30 0x7fa4580a60f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa4741a06f0 tx=0x7fa45c005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:08.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.654+0000 7fa479a00640 1 -- 172.21.15.40:0/2873306641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa474108f90 msgr2=0x7fa47419f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:08.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.654+0000 7fa479a00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa474108f90 0x7fa47419f1e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa464005940 tx=0x7fa464032dc0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.654+0000 7fa479a00640 1 -- 172.21.15.40:0/2873306641 shutdown_connections 2024-04-26T23:59:08.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.654+0000 7fa479a00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa47410ab40 0x7fa4741a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.654+0000 7fa479a00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa4580a3c30 0x7fa4580a60f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.654+0000 7fa479a00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa47410a180 0x7fa47419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.654+0000 7fa479a00640 1 --2- 172.21.15.40:0/2873306641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa474108f90 0x7fa47419f1e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.654+0000 7fa479a00640 1 -- 172.21.15.40:0/2873306641 >> 172.21.15.40:0/2873306641 conn(0x7fa474077f40 msgr2=0x7fa47410d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:08.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.655+0000 7fa479a00640 1 -- 172.21.15.40:0/2873306641 shutdown_connections 2024-04-26T23:59:08.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.655+0000 7fa479a00640 1 -- 172.21.15.40:0/2873306641 wait complete. 2024-04-26T23:59:08.673 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:08 smithi040 ceph-mon[28095]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 195 B/s wr, 0 op/s 2024-04-26T23:59:08.678 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:53:32.708818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:58:48.506601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:58:49.540141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:58:53.179820+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:59:08.678 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:59:08.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.889+0000 7f3e9b200640 1 -- 172.21.15.40:0/523512112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c075390 msgr2=0x7f3e9c075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:08.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.889+0000 7f3e9b200640 1 --2- 172.21.15.40:0/523512112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c075390 0x7f3e9c075810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3e88009930 tx=0x7f3e8802a440 comp rx=0 tx=0).stop 2024-04-26T23:59:08.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.889+0000 7f3e9b200640 1 -- 172.21.15.40:0/523512112 shutdown_connections 2024-04-26T23:59:08.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.889+0000 7f3e9b200640 1 --2- 172.21.15.40:0/523512112 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3e9c10ee90 0x7f3e9c111280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.889+0000 7f3e9b200640 1 --2- 172.21.15.40:0/523512112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c075390 0x7f3e9c075810 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.889+0000 7f3e9b200640 1 --2- 172.21.15.40:0/523512112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e9c076970 0x7f3e9c074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.890+0000 7f3e9b200640 1 -- 172.21.15.40:0/523512112 >> 172.21.15.40:0/523512112 conn(0x7f3e9c0fde30 msgr2=0x7f3e9c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:08.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.890+0000 7f3e9b200640 1 -- 172.21.15.40:0/523512112 shutdown_connections 2024-04-26T23:59:08.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.890+0000 7f3e9b200640 1 -- 172.21.15.40:0/523512112 wait complete. 2024-04-26T23:59:08.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.891+0000 7f3e9b200640 1 Processor -- start 2024-04-26T23:59:08.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.891+0000 7f3e9b200640 1 -- start start 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.891+0000 7f3e9b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c075390 0x7f3e9c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.891+0000 7f3e9b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e9c076970 0x7f3e9c19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.891+0000 7f3e9b200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3e9c10ee90 0x7f3e9c1a6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.891+0000 7f3e9b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c1149a0 con 0x7f3e9c075390 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.891+0000 7f3e9b200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c114820 con 0x7f3e9c10ee90 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.891+0000 7f3e9b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c114b20 con 0x7f3e9c076970 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.891+0000 7f3e99e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c075390 0x7f3e9c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.891+0000 7f3e99e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c075390 0x7f3e9c19f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32890/0 (socket says 172.21.15.40:32890) 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.891+0000 7f3e99e00640 1 -- 172.21.15.40:0/655319673 learned_addr learned my addr 172.21.15.40:0/655319673 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.891+0000 7f3e9a800640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3e9c10ee90 0x7f3e9c1a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.892+0000 7f3e99400640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e9c076970 0x7f3e9c19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.892+0000 7f3e99e00640 1 -- 172.21.15.40:0/655319673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e9c076970 msgr2=0x7f3e9c19f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:08.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.892+0000 7f3e99e00640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e9c076970 0x7f3e9c19f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.892+0000 7f3e99e00640 1 -- 172.21.15.40:0/655319673 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3e9c10ee90 msgr2=0x7f3e9c1a6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:08.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.892+0000 7f3e99e00640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3e9c10ee90 0x7f3e9c1a6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:08.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.892+0000 7f3e99e00640 1 -- 172.21.15.40:0/655319673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9c1a6cf0 con 0x7f3e9c075390 2024-04-26T23:59:08.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.892+0000 7f3e99e00640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c075390 0x7f3e9c19f0d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00e9f0 tx=0x7f3e8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:08.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.892+0000 7f3e92a00640 1 -- 172.21.15.40:0/655319673 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c00cd80 con 0x7f3e9c075390 2024-04-26T23:59:08.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.893+0000 7f3e92a00640 1 -- 172.21.15.40:0/655319673 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e8c0108e0 con 0x7f3e9c075390 2024-04-26T23:59:08.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.893+0000 7f3e9b200640 1 -- 172.21.15.40:0/655319673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9c1a6fe0 con 0x7f3e9c075390 2024-04-26T23:59:08.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.893+0000 7f3e92a00640 1 -- 172.21.15.40:0/655319673 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c018980 con 0x7f3e9c075390 2024-04-26T23:59:08.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.893+0000 7f3e9b200640 1 -- 172.21.15.40:0/655319673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e9c1a7440 con 0x7f3e9c075390 2024-04-26T23:59:08.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.894+0000 7f3e9b200640 1 -- 172.21.15.40:0/655319673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e9c105060 con 0x7f3e9c075390 2024-04-26T23:59:08.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.896+0000 7f3e92a00640 1 -- 172.21.15.40:0/655319673 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3e8c010430 con 0x7f3e9c075390 2024-04-26T23:59:08.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.898+0000 7f3e92a00640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3e800a3bb0 0x7f3e800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:08.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.898+0000 7f3e92a00640 1 -- 172.21.15.40:0/655319673 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f3e8c0d0950 con 0x7f3e9c075390 2024-04-26T23:59:08.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.898+0000 7f3e99400640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3e800a3bb0 0x7f3e800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:08.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.899+0000 7f3e99400640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3e800a3bb0 0x7f3e800a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3e88005ab0 tx=0x7f3e88005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:08.902+0000 7f3e92a00640 1 -- 172.21.15.40:0/655319673 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3e8c09ca40 con 0x7f3e9c075390 2024-04-26T23:59:09.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.113+0000 7f3e9b200640 1 -- 172.21.15.40:0/655319673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e9c063560 con 0x7f3e9c075390 2024-04-26T23:59:09.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.114+0000 7f3e92a00640 1 -- 172.21.15.40:0/655319673 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f3e8c0a0d40 con 0x7f3e9c075390 2024-04-26T23:59:09.116 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:59:09.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.117+0000 7f3e9b200640 1 -- 172.21.15.40:0/655319673 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3e800a3bb0 msgr2=0x7f3e800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:09.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.117+0000 7f3e9b200640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3e800a3bb0 0x7f3e800a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3e88005ab0 tx=0x7f3e88005a20 comp rx=0 tx=0).stop 2024-04-26T23:59:09.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.117+0000 7f3e9b200640 1 -- 172.21.15.40:0/655319673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c075390 msgr2=0x7f3e9c19f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:09.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.117+0000 7f3e9b200640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c075390 0x7f3e9c19f0d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00e9f0 tx=0x7f3e8c00eeb0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.117+0000 7f3e9b200640 1 -- 172.21.15.40:0/655319673 shutdown_connections 2024-04-26T23:59:09.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.117+0000 7f3e9b200640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3e9c10ee90 0x7f3e9c1a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.117+0000 7f3e9b200640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3e800a3bb0 0x7f3e800a6070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.117+0000 7f3e9b200640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e9c076970 0x7f3e9c19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.117+0000 7f3e9b200640 1 --2- 172.21.15.40:0/655319673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c075390 0x7f3e9c19f0d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.117+0000 7f3e9b200640 1 -- 172.21.15.40:0/655319673 >> 172.21.15.40:0/655319673 conn(0x7f3e9c0fde30 msgr2=0x7f3e9c102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:09.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.117+0000 7f3e9b200640 1 -- 172.21.15.40:0/655319673 shutdown_connections 2024-04-26T23:59:09.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.117+0000 7f3e9b200640 1 -- 172.21.15.40:0/655319673 wait complete. 2024-04-26T23:59:09.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:59:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.350+0000 7febfd800640 1 -- 172.21.15.40:0/993041075 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7febf80772f0 msgr2=0x7febf8077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.350+0000 7febfd800640 1 --2- 172.21.15.40:0/993041075 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7febf80772f0 0x7febf8077710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7febe8009900 tx=0x7febe802a3f0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.350+0000 7febfd800640 1 -- 172.21.15.40:0/993041075 shutdown_connections 2024-04-26T23:59:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.350+0000 7febfd800640 1 --2- 172.21.15.40:0/993041075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febf8078ed0 0x7febf8079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.350+0000 7febfd800640 1 --2- 172.21.15.40:0/993041075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febf8078530 0x7febf8078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.350+0000 7febfd800640 1 --2- 172.21.15.40:0/993041075 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7febf80772f0 0x7febf8077710 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.350+0000 7febfd800640 1 -- 172.21.15.40:0/993041075 >> 172.21.15.40:0/993041075 conn(0x7febf80fffe0 msgr2=0x7febf8102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.350+0000 7febfd800640 1 -- 172.21.15.40:0/993041075 shutdown_connections 2024-04-26T23:59:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.350+0000 7febfd800640 1 -- 172.21.15.40:0/993041075 wait complete. 2024-04-26T23:59:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.351+0000 7febfd800640 1 Processor -- start 2024-04-26T23:59:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.351+0000 7febfd800640 1 -- start start 2024-04-26T23:59:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.351+0000 7febfd800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febf80772f0 0x7febf819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febfd800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7febf8078530 0x7febf819f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febfd800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febf8078ed0 0x7febf81a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febfd800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7febf8114970 con 0x7febf80772f0 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febfd800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7febf81147f0 con 0x7febf8078530 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febfd800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7febf8114af0 con 0x7febf8078ed0 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febf7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febf80772f0 0x7febf819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febf7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febf80772f0 0x7febf819f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32908/0 (socket says 172.21.15.40:32908) 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febf7e00640 1 -- 172.21.15.40:0/3447587363 learned_addr learned my addr 172.21.15.40:0/3447587363 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febfce00640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febf8078ed0 0x7febf81a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febefe00640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7febf8078530 0x7febf819f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febf7e00640 1 -- 172.21.15.40:0/3447587363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febf8078ed0 msgr2=0x7febf81a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febf7e00640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febf8078ed0 0x7febf81a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febf7e00640 1 -- 172.21.15.40:0/3447587363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7febf8078530 msgr2=0x7febf819f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febf7e00640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7febf8078530 0x7febf819f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.352+0000 7febf7e00640 1 -- 172.21.15.40:0/3447587363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf81a6cc0 con 0x7febf80772f0 2024-04-26T23:59:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.353+0000 7febf7e00640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febf80772f0 0x7febf819f0a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7febe8005940 tx=0x7febe80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.353+0000 7febf5600640 1 -- 172.21.15.40:0/3447587363 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe803f030 con 0x7febf80772f0 2024-04-26T23:59:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.353+0000 7febf5600640 1 -- 172.21.15.40:0/3447587363 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febe8003cb0 con 0x7febf80772f0 2024-04-26T23:59:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.353+0000 7febf5600640 1 -- 172.21.15.40:0/3447587363 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe8035070 con 0x7febf80772f0 2024-04-26T23:59:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.353+0000 7febfd800640 1 -- 172.21.15.40:0/3447587363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf81a6f50 con 0x7febf80772f0 2024-04-26T23:59:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.353+0000 7febfd800640 1 -- 172.21.15.40:0/3447587363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febf81a7360 con 0x7febf80772f0 2024-04-26T23:59:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.355+0000 7febfd800640 1 -- 172.21.15.40:0/3447587363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febf8111630 con 0x7febf80772f0 2024-04-26T23:59:09.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.357+0000 7febf5600640 1 -- 172.21.15.40:0/3447587363 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7febe8042030 con 0x7febf80772f0 2024-04-26T23:59:09.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.359+0000 7febf5600640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7febdc0a3bb0 0x7febdc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:09.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.359+0000 7febf5600640 1 -- 172.21.15.40:0/3447587363 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7febe80f1080 con 0x7febf80772f0 2024-04-26T23:59:09.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.360+0000 7febefe00640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7febdc0a3bb0 0x7febdc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:09.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.361+0000 7febefe00640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7febdc0a3bb0 0x7febdc0a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7febf81a05b0 tx=0x7febe0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:09.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.362+0000 7febf5600640 1 -- 172.21.15.40:0/3447587363 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7febe8003e60 con 0x7febf80772f0 2024-04-26T23:59:09.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:09 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2873306641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:09.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:09 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/655319673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:59:09.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:09 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2873306641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:09.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:09 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/655319673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:59:09.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.571+0000 7febfd800640 1 -- 172.21.15.40:0/3447587363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7febf802dd20 con 0x7febf80772f0 2024-04-26T23:59:09.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.572+0000 7febf5600640 1 -- 172.21.15.40:0/3447587363 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7febe80bd170 con 0x7febf80772f0 2024-04-26T23:59:09.574 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:59:09.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.575+0000 7febfd800640 1 -- 172.21.15.40:0/3447587363 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7febdc0a3bb0 msgr2=0x7febdc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:09.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.575+0000 7febfd800640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7febdc0a3bb0 0x7febdc0a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7febf81a05b0 tx=0x7febe0005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:09.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.575+0000 7febfd800640 1 -- 172.21.15.40:0/3447587363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febf80772f0 msgr2=0x7febf819f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:09.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.575+0000 7febfd800640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febf80772f0 0x7febf819f0a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7febe8005940 tx=0x7febe80042f0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.575+0000 7febfd800640 1 -- 172.21.15.40:0/3447587363 shutdown_connections 2024-04-26T23:59:09.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.575+0000 7febfd800640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7febdc0a3bb0 0x7febdc0a6070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.575+0000 7febfd800640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febf8078ed0 0x7febf81a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.575+0000 7febfd800640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7febf8078530 0x7febf819f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.575+0000 7febfd800640 1 --2- 172.21.15.40:0/3447587363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febf80772f0 0x7febf819f0a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.575+0000 7febfd800640 1 -- 172.21.15.40:0/3447587363 >> 172.21.15.40:0/3447587363 conn(0x7febf80fffe0 msgr2=0x7febf81017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:09.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.575+0000 7febfd800640 1 -- 172.21.15.40:0/3447587363 shutdown_connections 2024-04-26T23:59:09.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.575+0000 7febfd800640 1 -- 172.21.15.40:0/3447587363 wait complete. 2024-04-26T23:59:09.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:59:09.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:09 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2873306641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:09.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:09 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/655319673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:59:09.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.811+0000 7f94e5600640 1 -- 172.21.15.40:0/2448713376 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94e0103200 msgr2=0x7f94e0109a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:09.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.811+0000 7f94e5600640 1 --2- 172.21.15.40:0/2448713376 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94e0103200 0x7f94e0109a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f94d8009900 tx=0x7f94d802a410 comp rx=0 tx=0).stop 2024-04-26T23:59:09.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.811+0000 7f94e5600640 1 -- 172.21.15.40:0/2448713376 shutdown_connections 2024-04-26T23:59:09.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.811+0000 7f94e5600640 1 --2- 172.21.15.40:0/2448713376 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94e0103200 0x7f94e0109a90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.811+0000 7f94e5600640 1 --2- 172.21.15.40:0/2448713376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e0102840 0x7f94e0102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.811+0000 7f94e5600640 1 --2- 172.21.15.40:0/2448713376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94e0101650 0x7f94e0101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.811+0000 7f94e5600640 1 -- 172.21.15.40:0/2448713376 >> 172.21.15.40:0/2448713376 conn(0x7f94e00fcc00 msgr2=0x7f94e00ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:09.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.811+0000 7f94e5600640 1 -- 172.21.15.40:0/2448713376 shutdown_connections 2024-04-26T23:59:09.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.811+0000 7f94e5600640 1 -- 172.21.15.40:0/2448713376 wait complete. 2024-04-26T23:59:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.812+0000 7f94e5600640 1 Processor -- start 2024-04-26T23:59:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.812+0000 7f94e5600640 1 -- start start 2024-04-26T23:59:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.812+0000 7f94e5600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e0101650 0x7f94e0198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.812+0000 7f94e5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94e0102840 0x7f94e0198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94e5600640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94e0103200 0x7f94e019fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94e5600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94e010d1d0 con 0x7f94e0102840 2024-04-26T23:59:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94e5600640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f94e010d050 con 0x7f94e0103200 2024-04-26T23:59:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94e5600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f94e010d350 con 0x7f94e0101650 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94df400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94e0102840 0x7f94e0198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94dfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e0101650 0x7f94e0198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94dfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e0101650 0x7f94e0198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37396/0 (socket says 172.21.15.40:37396) 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94dfe00640 1 -- 172.21.15.40:0/362820760 learned_addr learned my addr 172.21.15.40:0/362820760 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94e4c00640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94e0103200 0x7f94e019fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94dfe00640 1 -- 172.21.15.40:0/362820760 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94e0103200 msgr2=0x7f94e019fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94dfe00640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94e0103200 0x7f94e019fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94dfe00640 1 -- 172.21.15.40:0/362820760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94e0102840 msgr2=0x7f94e0198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94dfe00640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94e0102840 0x7f94e0198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94dfe00640 1 -- 172.21.15.40:0/362820760 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e01a0550 con 0x7f94e0101650 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.813+0000 7f94df400640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94e0102840 0x7f94e0198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.814+0000 7f94dfe00640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e0101650 0x7f94e0198930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f94d000ebc0 tx=0x7f94d000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.814+0000 7f94dcc00640 1 -- 172.21.15.40:0/362820760 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d0003b70 con 0x7f94e0101650 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.814+0000 7f94e5600640 1 -- 172.21.15.40:0/362820760 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e01a0840 con 0x7f94e0101650 2024-04-26T23:59:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.814+0000 7f94e5600640 1 -- 172.21.15.40:0/362820760 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e01a0de0 con 0x7f94e0101650 2024-04-26T23:59:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.814+0000 7f94dcc00640 1 -- 172.21.15.40:0/362820760 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94d0003d10 con 0x7f94e0101650 2024-04-26T23:59:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.814+0000 7f94dcc00640 1 -- 172.21.15.40:0/362820760 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d0019800 con 0x7f94e0101650 2024-04-26T23:59:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.815+0000 7f94e5600640 1 -- 172.21.15.40:0/362820760 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94bc005290 con 0x7f94e0101650 2024-04-26T23:59:09.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.818+0000 7f94dcc00640 1 -- 172.21.15.40:0/362820760 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f94d00199a0 con 0x7f94e0101650 2024-04-26T23:59:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.819+0000 7f94dcc00640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f94c40a3c30 0x7f94c40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.819+0000 7f94dcc00640 1 -- 172.21.15.40:0/362820760 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f94d00d0e00 con 0x7f94e0101650 2024-04-26T23:59:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.819+0000 7f94df400640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f94c40a3c30 0x7f94c40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.820+0000 7f94df400640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f94c40a3c30 0x7f94c40a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f94e0199e40 tx=0x7f94c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:09.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:09.824+0000 7f94dcc00640 1 -- 172.21.15.40:0/362820760 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f94d009d540 con 0x7f94e0101650 2024-04-26T23:59:10.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.030+0000 7f94e5600640 1 -- 172.21.15.40:0/362820760 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f94bc005d90 con 0x7f94e0101650 2024-04-26T23:59:10.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.031+0000 7f94dcc00640 1 -- 172.21.15.40:0/362820760 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f94d009ce40 con 0x7f94e0101650 2024-04-26T23:59:10.033 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:59:10.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.033+0000 7f94e5600640 1 -- 172.21.15.40:0/362820760 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f94c40a3c30 msgr2=0x7f94c40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:10.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.033+0000 7f94e5600640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f94c40a3c30 0x7f94c40a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f94e0199e40 tx=0x7f94c8005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:10.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.033+0000 7f94e5600640 1 -- 172.21.15.40:0/362820760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e0101650 msgr2=0x7f94e0198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:10.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.033+0000 7f94e5600640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e0101650 0x7f94e0198930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f94d000ebc0 tx=0x7f94d000c6a0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.033+0000 7f94e5600640 1 -- 172.21.15.40:0/362820760 shutdown_connections 2024-04-26T23:59:10.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.033+0000 7f94e5600640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f94e0103200 0x7f94e019fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.033+0000 7f94e5600640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94e0102840 0x7f94e0198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.033+0000 7f94e5600640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f94c40a3c30 0x7f94c40a60f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.033+0000 7f94e5600640 1 --2- 172.21.15.40:0/362820760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e0101650 0x7f94e0198930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.033+0000 7f94e5600640 1 -- 172.21.15.40:0/362820760 >> 172.21.15.40:0/362820760 conn(0x7f94e00fcc00 msgr2=0x7f94e00fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:10.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.033+0000 7f94e5600640 1 -- 172.21.15.40:0/362820760 shutdown_connections 2024-04-26T23:59:10.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.034+0000 7f94e5600640 1 -- 172.21.15.40:0/362820760 wait complete. 2024-04-26T23:59:10.054 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:59:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.271+0000 7fd9f9200640 1 -- 172.21.15.40:0/706312514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f4075390 msgr2=0x7fd9f4075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.271+0000 7fd9f9200640 1 --2- 172.21.15.40:0/706312514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f4075390 0x7fd9f4075810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc009900 tx=0x7fd9dc02a400 comp rx=0 tx=0).stop 2024-04-26T23:59:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.272+0000 7fd9f9200640 1 -- 172.21.15.40:0/706312514 shutdown_connections 2024-04-26T23:59:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.272+0000 7fd9f9200640 1 --2- 172.21.15.40:0/706312514 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd9f410a840 0x7fd9f410ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.272+0000 7fd9f9200640 1 --2- 172.21.15.40:0/706312514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f4075390 0x7fd9f4075810 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.272+0000 7fd9f9200640 1 --2- 172.21.15.40:0/706312514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f4076970 0x7fd9f4074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.272+0000 7fd9f9200640 1 -- 172.21.15.40:0/706312514 >> 172.21.15.40:0/706312514 conn(0x7fd9f40fdf10 msgr2=0x7fd9f4100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.272+0000 7fd9f9200640 1 -- 172.21.15.40:0/706312514 shutdown_connections 2024-04-26T23:59:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.272+0000 7fd9f9200640 1 -- 172.21.15.40:0/706312514 wait complete. 2024-04-26T23:59:10.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.273+0000 7fd9f9200640 1 Processor -- start 2024-04-26T23:59:10.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.273+0000 7fd9f9200640 1 -- start start 2024-04-26T23:59:10.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.273+0000 7fd9f9200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f4075390 0x7fd9f419ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:10.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.273+0000 7fd9f9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f4076970 0x7fd9f419b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:10.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.273+0000 7fd9f9200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd9f410a840 0x7fd9f41a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:10.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.273+0000 7fd9f9200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f4110690 con 0x7fd9f4076970 2024-04-26T23:59:10.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.273+0000 7fd9f9200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f4110510 con 0x7fd9f410a840 2024-04-26T23:59:10.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.273+0000 7fd9f9200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f4110810 con 0x7fd9f4075390 2024-04-26T23:59:10.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.274+0000 7fd9ebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f4076970 0x7fd9f419b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:10.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.274+0000 7fd9f1e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f4075390 0x7fd9f419ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:10.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.274+0000 7fd9f2800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd9f410a840 0x7fd9f41a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:10.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.274+0000 7fd9f1e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f4075390 0x7fd9f419ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50766/0 (socket says 172.21.15.40:50766) 2024-04-26T23:59:10.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.274+0000 7fd9f1e00640 1 -- 172.21.15.40:0/1807999858 learned_addr learned my addr 172.21.15.40:0/1807999858 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:10.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.274+0000 7fd9f2800640 1 -- 172.21.15.40:0/1807999858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f4075390 msgr2=0x7fd9f419ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:10.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.274+0000 7fd9f2800640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f4075390 0x7fd9f419ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.274+0000 7fd9f2800640 1 -- 172.21.15.40:0/1807999858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f4076970 msgr2=0x7fd9f419b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:10.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.274+0000 7fd9f2800640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f4076970 0x7fd9f419b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.274+0000 7fd9f2800640 1 -- 172.21.15.40:0/1807999858 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f41a2a00 con 0x7fd9f410a840 2024-04-26T23:59:10.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.274+0000 7fd9ebe00640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f4076970 0x7fd9f419b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:10.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.274+0000 7fd9f2800640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd9f410a840 0x7fd9f41a2390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec009fd0 tx=0x7fd9ec00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:10.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.275+0000 7fd9eaa00640 1 -- 172.21.15.40:0/1807999858 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec00ce20 con 0x7fd9f410a840 2024-04-26T23:59:10.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.275+0000 7fd9eaa00640 1 -- 172.21.15.40:0/1807999858 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9ec005020 con 0x7fd9f410a840 2024-04-26T23:59:10.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.275+0000 7fd9eaa00640 1 -- 172.21.15.40:0/1807999858 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec010690 con 0x7fd9f410a840 2024-04-26T23:59:10.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.275+0000 7fd9f9200640 1 -- 172.21.15.40:0/1807999858 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f41a2cf0 con 0x7fd9f410a840 2024-04-26T23:59:10.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.275+0000 7fd9f9200640 1 -- 172.21.15.40:0/1807999858 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f41a31c0 con 0x7fd9f410a840 2024-04-26T23:59:10.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.276+0000 7fd9f9200640 1 -- 172.21.15.40:0/1807999858 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f4109570 con 0x7fd9f410a840 2024-04-26T23:59:10.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.278+0000 7fd9eaa00640 1 -- 172.21.15.40:0/1807999858 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd9ec0054d0 con 0x7fd9f410a840 2024-04-26T23:59:10.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.279+0000 7fd9eaa00640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd9d80a3c30 0x7fd9d80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:10.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.279+0000 7fd9eaa00640 1 -- 172.21.15.40:0/1807999858 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd9ec0d0d10 con 0x7fd9f410a840 2024-04-26T23:59:10.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.279+0000 7fd9f1e00640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd9d80a3c30 0x7fd9d80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:10.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.280+0000 7fd9f1e00640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd9d80a3c30 0x7fd9d80a60f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009cd0 tx=0x7fd9e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.284+0000 7fd9eaa00640 1 -- 172.21.15.40:0/1807999858 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd9ec0d6020 con 0x7fd9f410a840 2024-04-26T23:59:10.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:10 smithi040 ceph-mon[28095]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-26T23:59:10.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:10 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3447587363' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:59:10.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:10 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/362820760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:59:10.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.490+0000 7fd9f9200640 1 -- 172.21.15.40:0/1807999858 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd9f4063530 con 0x7fd9f410a840 2024-04-26T23:59:10.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.491+0000 7fd9eaa00640 1 -- 172.21.15.40:0/1807999858 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7fd9ec09d450 con 0x7fd9f410a840 2024-04-26T23:59:10.493 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:59:10.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.494+0000 7fd9f9200640 1 -- 172.21.15.40:0/1807999858 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd9d80a3c30 msgr2=0x7fd9d80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:10.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.494+0000 7fd9f9200640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd9d80a3c30 0x7fd9d80a60f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009cd0 tx=0x7fd9e4005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:10.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.494+0000 7fd9f9200640 1 -- 172.21.15.40:0/1807999858 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd9f410a840 msgr2=0x7fd9f41a2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:10.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.494+0000 7fd9f9200640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd9f410a840 0x7fd9f41a2390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec009fd0 tx=0x7fd9ec00ef10 comp rx=0 tx=0).stop 2024-04-26T23:59:10.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.494+0000 7fd9f9200640 1 -- 172.21.15.40:0/1807999858 shutdown_connections 2024-04-26T23:59:10.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.494+0000 7fd9f9200640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd9f410a840 0x7fd9f41a2390 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.494+0000 7fd9f9200640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f4076970 0x7fd9f419b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.494+0000 7fd9f9200640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd9d80a3c30 0x7fd9d80a60f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.494+0000 7fd9f9200640 1 --2- 172.21.15.40:0/1807999858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f4075390 0x7fd9f419ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.494+0000 7fd9f9200640 1 -- 172.21.15.40:0/1807999858 >> 172.21.15.40:0/1807999858 conn(0x7fd9f40fdf10 msgr2=0x7fd9f4106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:10.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.494+0000 7fd9f9200640 1 -- 172.21.15.40:0/1807999858 shutdown_connections 2024-04-26T23:59:10.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.494+0000 7fd9f9200640 1 -- 172.21.15.40:0/1807999858 wait complete. 2024-04-26T23:59:10.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:59:10.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:10 smithi080 ceph-mon[32362]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-26T23:59:10.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:10 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3447587363' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:59:10.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:10 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/362820760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:59:10.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:10 smithi103 ceph-mon[32952]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-26T23:59:10.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:10 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3447587363' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:59:10.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:10 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/362820760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:59:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.732+0000 7f0f29400640 1 -- 172.21.15.40:0/71429899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f24078530 msgr2=0x7f0f24078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.732+0000 7f0f29400640 1 --2- 172.21.15.40:0/71429899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f24078530 0x7f0f24078990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c009900 tx=0x7f0f0c02a400 comp rx=0 tx=0).stop 2024-04-26T23:59:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.732+0000 7f0f29400640 1 -- 172.21.15.40:0/71429899 shutdown_connections 2024-04-26T23:59:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.732+0000 7f0f29400640 1 --2- 172.21.15.40:0/71429899 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f24078ed0 0x7f0f24079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.732+0000 7f0f29400640 1 --2- 172.21.15.40:0/71429899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f24078530 0x7f0f24078990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.732+0000 7f0f29400640 1 --2- 172.21.15.40:0/71429899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f240772f0 0x7f0f24077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.732+0000 7f0f29400640 1 -- 172.21.15.40:0/71429899 >> 172.21.15.40:0/71429899 conn(0x7f0f24100120 msgr2=0x7f0f24102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:10.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.732+0000 7f0f29400640 1 -- 172.21.15.40:0/71429899 shutdown_connections 2024-04-26T23:59:10.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.732+0000 7f0f29400640 1 -- 172.21.15.40:0/71429899 wait complete. 2024-04-26T23:59:10.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.733+0000 7f0f29400640 1 Processor -- start 2024-04-26T23:59:10.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f29400640 1 -- start start 2024-04-26T23:59:10.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f29400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f240772f0 0x7f0f2419f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:10.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f29400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f24078530 0x7f0f2419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:10.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f29400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f24078ed0 0x7f0f241a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f29400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0f24114af0 con 0x7f0f24078ed0 2024-04-26T23:59:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f29400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0f24114970 con 0x7f0f24078530 2024-04-26T23:59:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f29400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f24114c70 con 0x7f0f240772f0 2024-04-26T23:59:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f22800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f240772f0 0x7f0f2419f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f23200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f24078ed0 0x7f0f241a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f21e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f24078530 0x7f0f2419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f21e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f24078530 0x7f0f2419f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:57790/0 (socket says 172.21.15.40:57790) 2024-04-26T23:59:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f21e00640 1 -- 172.21.15.40:0/723794424 learned_addr learned my addr 172.21.15.40:0/723794424 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.734+0000 7f0f23200640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f24078ed0 0x7f0f241a67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39074/0 (socket says 172.21.15.40:39074) 2024-04-26T23:59:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.735+0000 7f0f21e00640 1 -- 172.21.15.40:0/723794424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f240772f0 msgr2=0x7f0f2419f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.735+0000 7f0f21e00640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f240772f0 0x7f0f2419f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.735+0000 7f0f21e00640 1 -- 172.21.15.40:0/723794424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f24078ed0 msgr2=0x7f0f241a67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.735+0000 7f0f21e00640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f24078ed0 0x7f0f241a67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.735+0000 7f0f21e00640 1 -- 172.21.15.40:0/723794424 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f241a6e20 con 0x7f0f24078530 2024-04-26T23:59:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.735+0000 7f0f23200640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f24078ed0 0x7f0f241a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.735+0000 7f0f22800640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f240772f0 0x7f0f2419f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.735+0000 7f0f21e00640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f24078530 0x7f0f2419f740 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c005ab0 tx=0x7f0f0c003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.735+0000 7f0f1b400640 1 -- 172.21.15.40:0/723794424 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c03f070 con 0x7f0f24078530 2024-04-26T23:59:10.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.735+0000 7f0f29400640 1 -- 172.21.15.40:0/723794424 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f241a70b0 con 0x7f0f24078530 2024-04-26T23:59:10.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.736+0000 7f0f1b400640 1 -- 172.21.15.40:0/723794424 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f0c031c50 con 0x7f0f24078530 2024-04-26T23:59:10.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.736+0000 7f0f1b400640 1 -- 172.21.15.40:0/723794424 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c035030 con 0x7f0f24078530 2024-04-26T23:59:10.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.736+0000 7f0f29400640 1 -- 172.21.15.40:0/723794424 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f241a75a0 con 0x7f0f24078530 2024-04-26T23:59:10.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.737+0000 7f0f29400640 1 -- 172.21.15.40:0/723794424 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f24111760 con 0x7f0f24078530 2024-04-26T23:59:10.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.739+0000 7f0f1b400640 1 -- 172.21.15.40:0/723794424 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0f0c042070 con 0x7f0f24078530 2024-04-26T23:59:10.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.740+0000 7f0f1b400640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f080a3bb0 0x7f0f080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:10.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.740+0000 7f0f1b400640 1 -- 172.21.15.40:0/723794424 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f0f0c0f1530 con 0x7f0f24078530 2024-04-26T23:59:10.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.741+0000 7f0f22800640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f080a3bb0 0x7f0f080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:10.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.741+0000 7f0f22800640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f080a3bb0 0x7f0f080a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c009fc0 tx=0x7f0f1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:10.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.743+0000 7f0f1b400640 1 -- 172.21.15.40:0/723794424 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0f0c0bdc70 con 0x7f0f24078530 2024-04-26T23:59:10.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.963+0000 7f0f29400640 1 -- 172.21.15.40:0/723794424 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0f2402dad0 con 0x7f0f24078530 2024-04-26T23:59:10.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.964+0000 7f0f1b400640 1 -- 172.21.15.40:0/723794424 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+588 (secure 0 0 0) 0x7f0f0c0bd570 con 0x7f0f24078530 2024-04-26T23:59:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1959424465 2024-04-27T23:50:44.546469+0000 2024-04-26T23:59:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/1568598447 2024-04-27T23:50:44.546469+0000 2024-04-26T23:59:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/29889962 2024-04-27T23:50:44.546469+0000 2024-04-26T23:59:10.966 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/673764112 2024-04-27T23:50:09.399879+0000 2024-04-26T23:59:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3782482730 2024-04-27T23:50:09.399879+0000 2024-04-26T23:59:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/1568598447 2024-04-27T23:50:44.546469+0000 2024-04-26T23:59:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1079307273 2024-04-27T23:50:09.399879+0000 2024-04-26T23:59:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/673764112 2024-04-27T23:50:09.399879+0000 2024-04-26T23:59:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4238592841 2024-04-27T23:50:44.546469+0000 2024-04-26T23:59:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4067589404 2024-04-27T23:50:09.399879+0000 2024-04-26T23:59:10.967 INFO:teuthology.orchestra.run.smithi040.stderr:listed 10 entries 2024-04-26T23:59:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.967+0000 7f0f29400640 1 -- 172.21.15.40:0/723794424 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f080a3bb0 msgr2=0x7f0f080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.967+0000 7f0f29400640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f080a3bb0 0x7f0f080a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c009fc0 tx=0x7f0f1c005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.967+0000 7f0f29400640 1 -- 172.21.15.40:0/723794424 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f24078530 msgr2=0x7f0f2419f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.967+0000 7f0f29400640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f24078530 0x7f0f2419f740 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c005ab0 tx=0x7f0f0c003b90 comp rx=0 tx=0).stop 2024-04-26T23:59:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.967+0000 7f0f29400640 1 -- 172.21.15.40:0/723794424 shutdown_connections 2024-04-26T23:59:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.967+0000 7f0f29400640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f24078ed0 0x7f0f241a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.967+0000 7f0f29400640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f24078530 0x7f0f2419f740 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.967+0000 7f0f29400640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f080a3bb0 0x7f0f080a6070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.967+0000 7f0f29400640 1 --2- 172.21.15.40:0/723794424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f240772f0 0x7f0f2419f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:10.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.967+0000 7f0f29400640 1 -- 172.21.15.40:0/723794424 >> 172.21.15.40:0/723794424 conn(0x7f0f24100120 msgr2=0x7f0f241018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:10.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.967+0000 7f0f29400640 1 -- 172.21.15.40:0/723794424 shutdown_connections 2024-04-26T23:59:10.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:10.967+0000 7f0f29400640 1 -- 172.21.15.40:0/723794424 wait complete. 2024-04-26T23:59:10.988 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:59:10.988 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:59:11.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:59:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.251+0000 7f9b08600640 1 -- 172.21.15.40:0/4213299360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b0010ab40 msgr2=0x7f9b001113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.251+0000 7f9b08600640 1 --2- 172.21.15.40:0/4213299360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b0010ab40 0x7f9b001113d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009ca0 tx=0x7f9af802a8c0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.251+0000 7f9b08600640 1 -- 172.21.15.40:0/4213299360 shutdown_connections 2024-04-26T23:59:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.251+0000 7f9b08600640 1 --2- 172.21.15.40:0/4213299360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b0010ab40 0x7f9b001113d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.251+0000 7f9b08600640 1 --2- 172.21.15.40:0/4213299360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b0010a180 0x7f9b0010a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.251+0000 7f9b08600640 1 --2- 172.21.15.40:0/4213299360 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9b00108f90 0x7f9b001093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.251+0000 7f9b08600640 1 -- 172.21.15.40:0/4213299360 >> 172.21.15.40:0/4213299360 conn(0x7f9b00077f40 msgr2=0x7f9b0007a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.251+0000 7f9b08600640 1 -- 172.21.15.40:0/4213299360 shutdown_connections 2024-04-26T23:59:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.251+0000 7f9b08600640 1 -- 172.21.15.40:0/4213299360 wait complete. 2024-04-26T23:59:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.252+0000 7f9b08600640 1 Processor -- start 2024-04-26T23:59:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.252+0000 7f9b08600640 1 -- start start 2024-04-26T23:59:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.252+0000 7f9b08600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b00108f90 0x7f9b0019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.252+0000 7f9b08600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b0010a180 0x7f9b0019f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.252+0000 7f9b08600640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9b0010ab40 0x7f9b001a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.252+0000 7f9b08600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9b00114ae0 con 0x7f9b0010a180 2024-04-26T23:59:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.252+0000 7f9b08600640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f9b00114960 con 0x7f9b0010ab40 2024-04-26T23:59:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.252+0000 7f9b08600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b00114c60 con 0x7f9b00108f90 2024-04-26T23:59:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.252+0000 7f9b05200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b0010a180 0x7f9b0019f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:11.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.253+0000 7f9b06600640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9b0010ab40 0x7f9b001a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.253+0000 7f9b06600640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9b0010ab40 0x7f9b001a66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:57804/0 (socket says 172.21.15.40:57804) 2024-04-26T23:59:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.253+0000 7f9b05200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b0010a180 0x7f9b0019f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39102/0 (socket says 172.21.15.40:39102) 2024-04-26T23:59:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.253+0000 7f9b06600640 1 -- 172.21.15.40:0/799045848 learned_addr learned my addr 172.21.15.40:0/799045848 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.253+0000 7f9b05c00640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b00108f90 0x7f9b0019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.253+0000 7f9b05200640 1 -- 172.21.15.40:0/799045848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b00108f90 msgr2=0x7f9b0019f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.253+0000 7f9b05200640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b00108f90 0x7f9b0019f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.253+0000 7f9b05200640 1 -- 172.21.15.40:0/799045848 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9b0010ab40 msgr2=0x7f9b001a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.253+0000 7f9b05200640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9b0010ab40 0x7f9b001a66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.253+0000 7f9b05200640 1 -- 172.21.15.40:0/799045848 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b001a6d20 con 0x7f9b0010a180 2024-04-26T23:59:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.253+0000 7f9b05200640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b0010a180 0x7f9b0019f640 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00e9f0 tx=0x7f9aec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.254+0000 7f9afe000640 1 -- 172.21.15.40:0/799045848 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec00ccc0 con 0x7f9b0010a180 2024-04-26T23:59:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.254+0000 7f9afe000640 1 -- 172.21.15.40:0/799045848 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9aec0108e0 con 0x7f9b0010a180 2024-04-26T23:59:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.254+0000 7f9b08600640 1 -- 172.21.15.40:0/799045848 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b001a6fb0 con 0x7f9b0010a180 2024-04-26T23:59:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.254+0000 7f9afe000640 1 -- 172.21.15.40:0/799045848 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec018880 con 0x7f9b0010a180 2024-04-26T23:59:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.254+0000 7f9b08600640 1 -- 172.21.15.40:0/799045848 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b001a7520 con 0x7f9b0010a180 2024-04-26T23:59:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.255+0000 7f9b08600640 1 -- 172.21.15.40:0/799045848 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b001102b0 con 0x7f9b0010a180 2024-04-26T23:59:11.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.257+0000 7f9afe000640 1 -- 172.21.15.40:0/799045848 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9aec010430 con 0x7f9b0010a180 2024-04-26T23:59:11.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.259+0000 7f9afe000640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9ae80a3c30 0x7f9ae80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:11.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.259+0000 7f9afe000640 1 -- 172.21.15.40:0/799045848 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f9aec0d08e0 con 0x7f9b0010a180 2024-04-26T23:59:11.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.259+0000 7f9b05c00640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9ae80a3c30 0x7f9ae80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:11.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.260+0000 7f9b05c00640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9ae80a3c30 0x7f9ae80a60f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9af40095f0 tx=0x7f9af4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:11.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.263+0000 7f9afe000640 1 -- 172.21.15.40:0/799045848 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9aec09c9d0 con 0x7f9b0010a180 2024-04-26T23:59:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.463+0000 7f9b08600640 1 -- 172.21.15.40:0/799045848 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b000635d0 con 0x7f9b0010a180 2024-04-26T23:59:11.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.465+0000 7f9afe000640 1 -- 172.21.15.40:0/799045848 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24090 (secure 0 0 0) 0x7f9aec0a0cd0 con 0x7f9b0010a180 2024-04-26T23:59:11.467 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:59:11.467 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:58:55.649955+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:59:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.467+0000 7f9b08600640 1 -- 172.21.15.40:0/799045848 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9ae80a3c30 msgr2=0x7f9ae80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.467+0000 7f9b08600640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9ae80a3c30 0x7f9ae80a60f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9af40095f0 tx=0x7f9af4005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.467+0000 7f9b08600640 1 -- 172.21.15.40:0/799045848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b0010a180 msgr2=0x7f9b0019f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.467+0000 7f9b08600640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b0010a180 0x7f9b0019f640 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00e9f0 tx=0x7f9aec00eeb0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.467+0000 7f9b08600640 1 -- 172.21.15.40:0/799045848 shutdown_connections 2024-04-26T23:59:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.467+0000 7f9b08600640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9b0010ab40 0x7f9b001a66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.467+0000 7f9b08600640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b0010a180 0x7f9b0019f640 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.467+0000 7f9b08600640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9ae80a3c30 0x7f9ae80a60f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.467+0000 7f9b08600640 1 --2- 172.21.15.40:0/799045848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b00108f90 0x7f9b0019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.467+0000 7f9b08600640 1 -- 172.21.15.40:0/799045848 >> 172.21.15.40:0/799045848 conn(0x7f9b00077f40 msgr2=0x7f9b0010d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.468+0000 7f9b08600640 1 -- 172.21.15.40:0/799045848 shutdown_connections 2024-04-26T23:59:11.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.468+0000 7f9b08600640 1 -- 172.21.15.40:0/799045848 wait complete. 2024-04-26T23:59:11.488 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:53:32.708818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:58:48.506601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:58:49.540141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:58:53.179820+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:59:11.489 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:59:11.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:11 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1807999858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:59:11.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:11 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/723794424' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:59:11.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:11 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1807999858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:59:11.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:11 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/723794424' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:59:11.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:11 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1807999858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:59:11.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:11 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/723794424' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:59:11.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.701+0000 7f824d400640 1 -- 172.21.15.40:0/563948668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248109fc0 msgr2=0x7f824810a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:11.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.701+0000 7f824d400640 1 --2- 172.21.15.40:0/563948668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248109fc0 0x7f824810a440 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8230009900 tx=0x7f823002a410 comp rx=0 tx=0).stop 2024-04-26T23:59:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.701+0000 7f824d400640 1 -- 172.21.15.40:0/563948668 shutdown_connections 2024-04-26T23:59:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.701+0000 7f824d400640 1 --2- 172.21.15.40:0/563948668 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f824810a980 0x7f82481112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.701+0000 7f824d400640 1 --2- 172.21.15.40:0/563948668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248109fc0 0x7f824810a440 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.701+0000 7f824d400640 1 --2- 172.21.15.40:0/563948668 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8248108e70 0x7f8248109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.701+0000 7f824d400640 1 -- 172.21.15.40:0/563948668 >> 172.21.15.40:0/563948668 conn(0x7f8248077e00 msgr2=0x7f824807a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.701+0000 7f824d400640 1 -- 172.21.15.40:0/563948668 shutdown_connections 2024-04-26T23:59:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.702+0000 7f824d400640 1 -- 172.21.15.40:0/563948668 wait complete. 2024-04-26T23:59:11.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.702+0000 7f824d400640 1 Processor -- start 2024-04-26T23:59:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.703+0000 7f824d400640 1 -- start start 2024-04-26T23:59:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.703+0000 7f824d400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8248108e70 0x7f824819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.703+0000 7f824d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248109fc0 0x7f824819f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.703+0000 7f824d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f824810a980 0x7f82481a6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.703+0000 7f824d400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f82481149d0 con 0x7f8248109fc0 2024-04-26T23:59:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.703+0000 7f824d400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f8248114850 con 0x7f8248108e70 2024-04-26T23:59:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.703+0000 7f824d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8248114b50 con 0x7f824810a980 2024-04-26T23:59:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.703+0000 7f823fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248109fc0 0x7f824819f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.703+0000 7f823fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248109fc0 0x7f824819f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39114/0 (socket says 172.21.15.40:39114) 2024-04-26T23:59:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.703+0000 7f823fe00640 1 -- 172.21.15.40:0/3166226666 learned_addr learned my addr 172.21.15.40:0/3166226666 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.703+0000 7f824ca00640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f824810a980 0x7f82481a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.703+0000 7f8247e00640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8248108e70 0x7f824819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.704+0000 7f823fe00640 1 -- 172.21.15.40:0/3166226666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f824810a980 msgr2=0x7f82481a6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.704+0000 7f823fe00640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f824810a980 0x7f82481a6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.704+0000 7f823fe00640 1 -- 172.21.15.40:0/3166226666 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8248108e70 msgr2=0x7f824819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.704+0000 7f823fe00640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8248108e70 0x7f824819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.704+0000 7f823fe00640 1 -- 172.21.15.40:0/3166226666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82481a6cf0 con 0x7f8248109fc0 2024-04-26T23:59:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.704+0000 7f823fe00640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248109fc0 0x7f824819f610 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8230006010 tx=0x7f82300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.704+0000 7f8245600640 1 -- 172.21.15.40:0/3166226666 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823003f030 con 0x7f8248109fc0 2024-04-26T23:59:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.704+0000 7f8245600640 1 -- 172.21.15.40:0/3166226666 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8230031910 con 0x7f8248109fc0 2024-04-26T23:59:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.704+0000 7f8245600640 1 -- 172.21.15.40:0/3166226666 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8230035070 con 0x7f8248109fc0 2024-04-26T23:59:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.704+0000 7f824d400640 1 -- 172.21.15.40:0/3166226666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82481a6f80 con 0x7f8248109fc0 2024-04-26T23:59:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.705+0000 7f824d400640 1 -- 172.21.15.40:0/3166226666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82481a73e0 con 0x7f8248109fc0 2024-04-26T23:59:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.706+0000 7f824d400640 1 -- 172.21.15.40:0/3166226666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82481101f0 con 0x7f8248109fc0 2024-04-26T23:59:11.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.708+0000 7f8245600640 1 -- 172.21.15.40:0/3166226666 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8230042030 con 0x7f8248109fc0 2024-04-26T23:59:11.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.710+0000 7f8245600640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f822c0a3c30 0x7f822c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:11.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.710+0000 7f8245600640 1 -- 172.21.15.40:0/3166226666 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f82300f0b10 con 0x7f8248109fc0 2024-04-26T23:59:11.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.710+0000 7f8247e00640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f822c0a3c30 0x7f822c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:11.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.711+0000 7f8247e00640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f822c0a3c30 0x7f822c0a60f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f82380095f0 tx=0x7f8238005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.715+0000 7f8245600640 1 -- 172.21.15.40:0/3166226666 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f82300bcc00 con 0x7f8248109fc0 2024-04-26T23:59:11.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.928+0000 7f824d400640 1 -- 172.21.15.40:0/3166226666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f82480635d0 con 0x7f8248109fc0 2024-04-26T23:59:11.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.929+0000 7f8245600640 1 -- 172.21.15.40:0/3166226666 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f82300c0f00 con 0x7f8248109fc0 2024-04-26T23:59:11.931 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-26T23:59:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.932+0000 7f824d400640 1 -- 172.21.15.40:0/3166226666 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f822c0a3c30 msgr2=0x7f822c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.932+0000 7f824d400640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f822c0a3c30 0x7f822c0a60f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f82380095f0 tx=0x7f8238005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.932+0000 7f824d400640 1 -- 172.21.15.40:0/3166226666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248109fc0 msgr2=0x7f824819f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.932+0000 7f824d400640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248109fc0 0x7f824819f610 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8230006010 tx=0x7f82300042f0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.932+0000 7f824d400640 1 -- 172.21.15.40:0/3166226666 shutdown_connections 2024-04-26T23:59:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.932+0000 7f824d400640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f822c0a3c30 0x7f822c0a60f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.932+0000 7f824d400640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f824810a980 0x7f82481a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.932+0000 7f824d400640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8248109fc0 0x7f824819f610 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.932+0000 7f824d400640 1 --2- 172.21.15.40:0/3166226666 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8248108e70 0x7f824819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.932+0000 7f824d400640 1 -- 172.21.15.40:0/3166226666 >> 172.21.15.40:0/3166226666 conn(0x7f8248077e00 msgr2=0x7f824810d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.932+0000 7f824d400640 1 -- 172.21.15.40:0/3166226666 shutdown_connections 2024-04-26T23:59:11.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:11.932+0000 7f824d400640 1 -- 172.21.15.40:0/3166226666 wait complete. 2024-04-26T23:59:11.952 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:59:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.166+0000 7f0076e00640 1 -- 172.21.15.40:0/2880790192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0070105c60 msgr2=0x7f00701060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.166+0000 7f0076e00640 1 --2- 172.21.15.40:0/2880790192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0070105c60 0x7f00701060e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0060009900 tx=0x7f006002a3f0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.166+0000 7f0076e00640 1 -- 172.21.15.40:0/2880790192 shutdown_connections 2024-04-26T23:59:12.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.166+0000 7f0076e00640 1 --2- 172.21.15.40:0/2880790192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0070106620 0x7f007010ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.166+0000 7f0076e00640 1 --2- 172.21.15.40:0/2880790192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0070105c60 0x7f00701060e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.166+0000 7f0076e00640 1 --2- 172.21.15.40:0/2880790192 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0070104a70 0x7f0070104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.166+0000 7f0076e00640 1 -- 172.21.15.40:0/2880790192 >> 172.21.15.40:0/2880790192 conn(0x7f00700fffe0 msgr2=0x7f0070102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:12.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.166+0000 7f0076e00640 1 -- 172.21.15.40:0/2880790192 shutdown_connections 2024-04-26T23:59:12.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.166+0000 7f0076e00640 1 -- 172.21.15.40:0/2880790192 wait complete. 2024-04-26T23:59:12.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.167+0000 7f0076e00640 1 Processor -- start 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.168+0000 7f0076e00640 1 -- start start 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.168+0000 7f0076e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0070104a70 0x7f007019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.168+0000 7f0076e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0070105c60 0x7f007019b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.168+0000 7f0076e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0070106620 0x7f00701a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.168+0000 7f0076e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00701105d0 con 0x7f0070104a70 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.168+0000 7f0076e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0070110450 con 0x7f0070105c60 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.168+0000 7f0076e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0070110750 con 0x7f0070106620 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.168+0000 7f0075000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0070105c60 0x7f007019b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.168+0000 7f0075000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0070105c60 0x7f007019b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:57840/0 (socket says 172.21.15.40:57840) 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.169+0000 7f0075a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0070104a70 0x7f007019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.169+0000 7f0075000640 1 -- 172.21.15.40:0/2389647818 learned_addr learned my addr 172.21.15.40:0/2389647818 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.169+0000 7f0076400640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0070106620 0x7f00701a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.169+0000 7f0075000640 1 -- 172.21.15.40:0/2389647818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0070106620 msgr2=0x7f00701a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:12.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.169+0000 7f0075000640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0070106620 0x7f00701a2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.169+0000 7f0075000640 1 -- 172.21.15.40:0/2389647818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0070104a70 msgr2=0x7f007019aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:12.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.169+0000 7f0075000640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0070104a70 0x7f007019aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.169+0000 7f0075000640 1 -- 172.21.15.40:0/2389647818 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00701a28e0 con 0x7f0070105c60 2024-04-26T23:59:12.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.169+0000 7f0076400640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0070106620 0x7f00701a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.169+0000 7f0075000640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0070105c60 0x7f007019b1e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0060005ab0 tx=0x7f0060003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.170+0000 7f006e000640 1 -- 172.21.15.40:0/2389647818 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006003f070 con 0x7f0070105c60 2024-04-26T23:59:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.170+0000 7f006e000640 1 -- 172.21.15.40:0/2389647818 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f006003b460 con 0x7f0070105c60 2024-04-26T23:59:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.170+0000 7f0076e00640 1 -- 172.21.15.40:0/2389647818 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00701a2b70 con 0x7f0070105c60 2024-04-26T23:59:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.170+0000 7f006e000640 1 -- 172.21.15.40:0/2389647818 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060035030 con 0x7f0070105c60 2024-04-26T23:59:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.170+0000 7f0076e00640 1 -- 172.21.15.40:0/2389647818 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00701a3060 con 0x7f0070105c60 2024-04-26T23:59:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.171+0000 7f0076e00640 1 -- 172.21.15.40:0/2389647818 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f007010beb0 con 0x7f0070105c60 2024-04-26T23:59:12.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.173+0000 7f006e000640 1 -- 172.21.15.40:0/2389647818 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0060042070 con 0x7f0070105c60 2024-04-26T23:59:12.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.175+0000 7f006e000640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f00580a3bb0 0x7f00580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:12.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.175+0000 7f006e000640 1 -- 172.21.15.40:0/2389647818 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f00600f12a0 con 0x7f0070105c60 2024-04-26T23:59:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.175+0000 7f0075a00640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f00580a3bb0 0x7f00580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.176+0000 7f0075a00640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f00580a3bb0 0x7f00580a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f007019bb70 tx=0x7f006400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:12.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.177+0000 7f006e000640 1 -- 172.21.15.40:0/2389647818 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f00600bd390 con 0x7f0070105c60 2024-04-26T23:59:12.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.386+0000 7f0076e00640 1 -- 172.21.15.40:0/2389647818 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f007002dc60 con 0x7f0070105c60 2024-04-26T23:59:12.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.387+0000 7f006e000640 1 -- 172.21.15.40:0/2389647818 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f00600c1690 con 0x7f0070105c60 2024-04-26T23:59:12.389 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:59:12.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.390+0000 7f0076e00640 1 -- 172.21.15.40:0/2389647818 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f00580a3bb0 msgr2=0x7f00580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:12.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.390+0000 7f0076e00640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f00580a3bb0 0x7f00580a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f007019bb70 tx=0x7f006400b390 comp rx=0 tx=0).stop 2024-04-26T23:59:12.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.390+0000 7f0076e00640 1 -- 172.21.15.40:0/2389647818 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0070105c60 msgr2=0x7f007019b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:12.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.390+0000 7f0076e00640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0070105c60 0x7f007019b1e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0060005ab0 tx=0x7f0060003cd0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.390+0000 7f0076e00640 1 -- 172.21.15.40:0/2389647818 shutdown_connections 2024-04-26T23:59:12.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.390+0000 7f0076e00640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f00580a3bb0 0x7f00580a6070 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.390+0000 7f0076e00640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0070106620 0x7f00701a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.390+0000 7f0076e00640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0070105c60 0x7f007019b1e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.390+0000 7f0076e00640 1 --2- 172.21.15.40:0/2389647818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0070104a70 0x7f007019aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.390+0000 7f0076e00640 1 -- 172.21.15.40:0/2389647818 >> 172.21.15.40:0/2389647818 conn(0x7f00700fffe0 msgr2=0x7f0070109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:12.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.390+0000 7f0076e00640 1 -- 172.21.15.40:0/2389647818 shutdown_connections 2024-04-26T23:59:12.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.391+0000 7f0076e00640 1 -- 172.21.15.40:0/2389647818 wait complete. 2024-04-26T23:59:12.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:59:12.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:12 smithi080 ceph-mon[32362]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-26T23:59:12.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:12 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/799045848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:12.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:12 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3166226666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:59:12.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:12 smithi103 ceph-mon[32952]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-26T23:59:12.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:12 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/799045848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:12.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:12 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3166226666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:59:12.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.622+0000 7f8443e00640 1 -- 172.21.15.40:0/1030061204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8444108430 msgr2=0x7f844410a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:12.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.622+0000 7f8443e00640 1 --2- 172.21.15.40:0/1030061204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8444108430 0x7f844410a880 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f842c009900 tx=0x7f842c02a410 comp rx=0 tx=0).stop 2024-04-26T23:59:12.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.623+0000 7f8443e00640 1 -- 172.21.15.40:0/1030061204 shutdown_connections 2024-04-26T23:59:12.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.623+0000 7f8443e00640 1 --2- 172.21.15.40:0/1030061204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8444108430 0x7f844410a880 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.623+0000 7f8443e00640 1 --2- 172.21.15.40:0/1030061204 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8444105b00 0x7f8444107ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.623+0000 7f8443e00640 1 --2- 172.21.15.40:0/1030061204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84441031d0 0x7f84441055c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.623+0000 7f8443e00640 1 -- 172.21.15.40:0/1030061204 >> 172.21.15.40:0/1030061204 conn(0x7f84440fcc00 msgr2=0x7f84440ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:12.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.623+0000 7f8443e00640 1 -- 172.21.15.40:0/1030061204 shutdown_connections 2024-04-26T23:59:12.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.623+0000 7f8443e00640 1 -- 172.21.15.40:0/1030061204 wait complete. 2024-04-26T23:59:12.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.624+0000 7f8443e00640 1 Processor -- start 2024-04-26T23:59:12.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.624+0000 7f8443e00640 1 -- start start 2024-04-26T23:59:12.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.624+0000 7f8443e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84441031d0 0x7f8444102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:12.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.624+0000 7f8443e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8444105b00 0x7f8444101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:12.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.624+0000 7f8443e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8444108430 0x7f84441016c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:12.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.624+0000 7f8443e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f844410e280 con 0x7f84441031d0 2024-04-26T23:59:12.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.624+0000 7f8443e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f844410e100 con 0x7f8444108430 2024-04-26T23:59:12.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.624+0000 7f8443e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f844410e400 con 0x7f8444105b00 2024-04-26T23:59:12.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.625+0000 7f8442a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84441031d0 0x7f8444102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:12.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.625+0000 7f8442a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84441031d0 0x7f8444102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39150/0 (socket says 172.21.15.40:39150) 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.625+0000 7f8442a00640 1 -- 172.21.15.40:0/2471747757 learned_addr learned my addr 172.21.15.40:0/2471747757 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.625+0000 7f8442000640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8444105b00 0x7f8444101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.625+0000 7f8443400640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8444108430 0x7f84441016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.625+0000 7f8442a00640 1 -- 172.21.15.40:0/2471747757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8444105b00 msgr2=0x7f8444101180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.625+0000 7f8442a00640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8444105b00 0x7f8444101180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.625+0000 7f8442a00640 1 -- 172.21.15.40:0/2471747757 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8444108430 msgr2=0x7f84441016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.625+0000 7f8442a00640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8444108430 0x7f84441016c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.625+0000 7f8442a00640 1 -- 172.21.15.40:0/2471747757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8444101f50 con 0x7f84441031d0 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.625+0000 7f8443400640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8444108430 0x7f84441016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.626+0000 7f8442a00640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84441031d0 0x7f8444102b40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f843c00ebc0 tx=0x7f843c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.626+0000 7f843b400640 1 -- 172.21.15.40:0/2471747757 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c010630 con 0x7f84441031d0 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.626+0000 7f843b400640 1 -- 172.21.15.40:0/2471747757 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f843c010c80 con 0x7f84441031d0 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.626+0000 7f8443e00640 1 -- 172.21.15.40:0/2471747757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84441a2a40 con 0x7f84441031d0 2024-04-26T23:59:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.626+0000 7f843b400640 1 -- 172.21.15.40:0/2471747757 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c018b60 con 0x7f84441031d0 2024-04-26T23:59:12.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.626+0000 7f8443e00640 1 -- 172.21.15.40:0/2471747757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84441a2f90 con 0x7f84441031d0 2024-04-26T23:59:12.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.627+0000 7f8443e00640 1 -- 172.21.15.40:0/2471747757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8444069670 con 0x7f84441031d0 2024-04-26T23:59:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.629+0000 7f843b400640 1 -- 172.21.15.40:0/2471747757 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f843c018d00 con 0x7f84441031d0 2024-04-26T23:59:12.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.631+0000 7f843b400640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f84280a3bb0 0x7f84280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:12.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.631+0000 7f843b400640 1 -- 172.21.15.40:0/2471747757 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f843c0d0cc0 con 0x7f84441031d0 2024-04-26T23:59:12.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.631+0000 7f8442000640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f84280a3bb0 0x7f84280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:12.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.632+0000 7f8442000640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f84280a3bb0 0x7f84280a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f84441028b0 tx=0x7f8434005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:12.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.635+0000 7f843b400640 1 -- 172.21.15.40:0/2471747757 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f843c09d460 con 0x7f84441031d0 2024-04-26T23:59:12.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:12 smithi040 ceph-mon[28095]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-26T23:59:12.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:12 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/799045848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:12.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:12 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3166226666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:59:12.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.841+0000 7f8443e00640 1 -- 172.21.15.40:0/2471747757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8444063530 con 0x7f84441031d0 2024-04-26T23:59:12.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.843+0000 7f843b400640 1 -- 172.21.15.40:0/2471747757 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f843c09cd60 con 0x7f84441031d0 2024-04-26T23:59:12.845 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-26T23:59:12.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.845+0000 7f8443e00640 1 -- 172.21.15.40:0/2471747757 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f84280a3bb0 msgr2=0x7f84280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:12.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.845+0000 7f8443e00640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f84280a3bb0 0x7f84280a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f84441028b0 tx=0x7f8434005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:12.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.845+0000 7f8443e00640 1 -- 172.21.15.40:0/2471747757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84441031d0 msgr2=0x7f8444102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:12.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.845+0000 7f8443e00640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84441031d0 0x7f8444102b40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f843c00ebc0 tx=0x7f843c00c6a0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.845+0000 7f8443e00640 1 -- 172.21.15.40:0/2471747757 shutdown_connections 2024-04-26T23:59:12.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.845+0000 7f8443e00640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8444108430 0x7f84441016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.845+0000 7f8443e00640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f84280a3bb0 0x7f84280a6070 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.845+0000 7f8443e00640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8444105b00 0x7f8444101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.845+0000 7f8443e00640 1 --2- 172.21.15.40:0/2471747757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84441031d0 0x7f8444102b40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:12.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.845+0000 7f8443e00640 1 -- 172.21.15.40:0/2471747757 >> 172.21.15.40:0/2471747757 conn(0x7f84440fcc00 msgr2=0x7f8444103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:12.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.845+0000 7f8443e00640 1 -- 172.21.15.40:0/2471747757 shutdown_connections 2024-04-26T23:59:12.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:12.846+0000 7f8443e00640 1 -- 172.21.15.40:0/2471747757 wait complete. 2024-04-26T23:59:12.867 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:59:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.080+0000 7efc7f200640 1 -- 172.21.15.40:0/603787974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc80104a70 msgr2=0x7efc80104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.080+0000 7efc7f200640 1 --2- 172.21.15.40:0/603787974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc80104a70 0x7efc80104e90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efc70009900 tx=0x7efc7002a400 comp rx=0 tx=0).stop 2024-04-26T23:59:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.080+0000 7efc7f200640 1 -- 172.21.15.40:0/603787974 shutdown_connections 2024-04-26T23:59:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.080+0000 7efc7f200640 1 --2- 172.21.15.40:0/603787974 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc80106620 0x7efc8010ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.080+0000 7efc7f200640 1 --2- 172.21.15.40:0/603787974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc80105c60 0x7efc801060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.080+0000 7efc7f200640 1 --2- 172.21.15.40:0/603787974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc80104a70 0x7efc80104e90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.080+0000 7efc7f200640 1 -- 172.21.15.40:0/603787974 >> 172.21.15.40:0/603787974 conn(0x7efc800fffe0 msgr2=0x7efc80102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:13.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.080+0000 7efc7f200640 1 -- 172.21.15.40:0/603787974 shutdown_connections 2024-04-26T23:59:13.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.080+0000 7efc7f200640 1 -- 172.21.15.40:0/603787974 wait complete. 2024-04-26T23:59:13.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.081+0000 7efc7f200640 1 Processor -- start 2024-04-26T23:59:13.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.081+0000 7efc7f200640 1 -- start start 2024-04-26T23:59:13.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.082+0000 7efc7f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc80104a70 0x7efc8019ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:13.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.082+0000 7efc7f200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc80105c60 0x7efc8019b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:13.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.082+0000 7efc7f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc80106620 0x7efc801a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:13.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.082+0000 7efc7f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc80110570 con 0x7efc80104a70 2024-04-26T23:59:13.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.082+0000 7efc7f200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7efc801103f0 con 0x7efc80105c60 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.082+0000 7efc7f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc801106f0 con 0x7efc80106620 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.082+0000 7efc7de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc80104a70 0x7efc8019ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.082+0000 7efc7d400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc80105c60 0x7efc8019b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.082+0000 7efc7e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc80106620 0x7efc801a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.082+0000 7efc7d400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc80105c60 0x7efc8019b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:57872/0 (socket says 172.21.15.40:57872) 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.082+0000 7efc7d400640 1 -- 172.21.15.40:0/3075044832 learned_addr learned my addr 172.21.15.40:0/3075044832 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.082+0000 7efc7de00640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc80104a70 0x7efc8019ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39178/0 (socket says 172.21.15.40:39178) 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.083+0000 7efc7de00640 1 -- 172.21.15.40:0/3075044832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc80106620 msgr2=0x7efc801a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.083+0000 7efc7de00640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc80106620 0x7efc801a2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.083+0000 7efc7de00640 1 -- 172.21.15.40:0/3075044832 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc80105c60 msgr2=0x7efc8019b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.083+0000 7efc7de00640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc80105c60 0x7efc8019b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.083+0000 7efc7de00640 1 -- 172.21.15.40:0/3075044832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc801a28d0 con 0x7efc80104a70 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.083+0000 7efc7d400640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc80105c60 0x7efc8019b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.083+0000 7efc7de00640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc80104a70 0x7efc8019ac90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efc70006010 tx=0x7efc700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.083+0000 7efc76a00640 1 -- 172.21.15.40:0/3075044832 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7003f030 con 0x7efc80104a70 2024-04-26T23:59:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.083+0000 7efc76a00640 1 -- 172.21.15.40:0/3075044832 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc70031910 con 0x7efc80104a70 2024-04-26T23:59:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.083+0000 7efc7f200640 1 -- 172.21.15.40:0/3075044832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc801a2b60 con 0x7efc80104a70 2024-04-26T23:59:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.083+0000 7efc76a00640 1 -- 172.21.15.40:0/3075044832 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70035070 con 0x7efc80104a70 2024-04-26T23:59:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.083+0000 7efc7f200640 1 -- 172.21.15.40:0/3075044832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc801a30a0 con 0x7efc80104a70 2024-04-26T23:59:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.085+0000 7efc7f200640 1 -- 172.21.15.40:0/3075044832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc8010beb0 con 0x7efc80104a70 2024-04-26T23:59:13.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.087+0000 7efc76a00640 1 -- 172.21.15.40:0/3075044832 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efc70042030 con 0x7efc80104a70 2024-04-26T23:59:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.088+0000 7efc76a00640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7efc640a3c30 0x7efc640a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.088+0000 7efc76a00640 1 -- 172.21.15.40:0/3075044832 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7efc700f0b10 con 0x7efc80104a70 2024-04-26T23:59:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.088+0000 7efc7d400640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7efc640a3c30 0x7efc640a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.089+0000 7efc7d400640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7efc640a3c30 0x7efc640a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efc8019c190 tx=0x7efc6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:13.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.093+0000 7efc76a00640 1 -- 172.21.15.40:0/3075044832 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efc700bcc00 con 0x7efc80104a70 2024-04-26T23:59:13.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.301+0000 7efc7f200640 1 -- 172.21.15.40:0/3075044832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efc80063530 con 0x7efc80104a70 2024-04-26T23:59:13.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.302+0000 7efc76a00640 1 -- 172.21.15.40:0/3075044832 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7efc700c0f00 con 0x7efc80104a70 2024-04-26T23:59:13.304 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-26T23:59:13.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.304+0000 7efc7f200640 1 -- 172.21.15.40:0/3075044832 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7efc640a3c30 msgr2=0x7efc640a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:13.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.304+0000 7efc7f200640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7efc640a3c30 0x7efc640a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efc8019c190 tx=0x7efc6c005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:13.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.304+0000 7efc7f200640 1 -- 172.21.15.40:0/3075044832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc80104a70 msgr2=0x7efc8019ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:13.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.304+0000 7efc7f200640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc80104a70 0x7efc8019ac90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efc70006010 tx=0x7efc700042f0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.304+0000 7efc7f200640 1 -- 172.21.15.40:0/3075044832 shutdown_connections 2024-04-26T23:59:13.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.304+0000 7efc7f200640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7efc640a3c30 0x7efc640a60f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.304+0000 7efc7f200640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc80106620 0x7efc801a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.304+0000 7efc7f200640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efc80105c60 0x7efc8019b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.305+0000 7efc7f200640 1 --2- 172.21.15.40:0/3075044832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc80104a70 0x7efc8019ac90 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.305+0000 7efc7f200640 1 -- 172.21.15.40:0/3075044832 >> 172.21.15.40:0/3075044832 conn(0x7efc800fffe0 msgr2=0x7efc80109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:13.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.305+0000 7efc7f200640 1 -- 172.21.15.40:0/3075044832 shutdown_connections 2024-04-26T23:59:13.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.305+0000 7efc7f200640 1 -- 172.21.15.40:0/3075044832 wait complete. 2024-04-26T23:59:13.324 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:59:13.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.536+0000 7f032f200640 1 -- 172.21.15.40:0/2292509780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0330100fd0 msgr2=0x7f0330103400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:13.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.536+0000 7f032f200640 1 --2- 172.21.15.40:0/2292509780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0330100fd0 0x7f0330103400 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0320009900 tx=0x7f032002a410 comp rx=0 tx=0).stop 2024-04-26T23:59:13.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.536+0000 7f032f200640 1 -- 172.21.15.40:0/2292509780 shutdown_connections 2024-04-26T23:59:13.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.536+0000 7f032f200640 1 --2- 172.21.15.40:0/2292509780 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f03301062f0 0x7f0330106770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.536+0000 7f032f200640 1 --2- 172.21.15.40:0/2292509780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0330103940 0x7f0330105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.536+0000 7f032f200640 1 --2- 172.21.15.40:0/2292509780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0330100fd0 0x7f0330103400 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.536+0000 7f032f200640 1 -- 172.21.15.40:0/2292509780 >> 172.21.15.40:0/2292509780 conn(0x7f03300fcbb0 msgr2=0x7f03300ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:13.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.536+0000 7f032f200640 1 -- 172.21.15.40:0/2292509780 shutdown_connections 2024-04-26T23:59:13.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.536+0000 7f032f200640 1 -- 172.21.15.40:0/2292509780 wait complete. 2024-04-26T23:59:13.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.537+0000 7f032f200640 1 Processor -- start 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.537+0000 7f032f200640 1 -- start start 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.537+0000 7f032f200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0330100fd0 0x7f033019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.537+0000 7f032f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0330103940 0x7f033019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.537+0000 7f032f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03301062f0 0x7f03301a6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.537+0000 7f032f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0330114860 con 0x7f0330103940 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.537+0000 7f032f200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f03301146e0 con 0x7f0330100fd0 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.537+0000 7f032f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f03301149e0 con 0x7f03301062f0 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0330103940 0x7f033019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0330103940 0x7f033019f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39196/0 (socket says 172.21.15.40:39196) 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032d400640 1 -- 172.21.15.40:0/898792374 learned_addr learned my addr 172.21.15.40:0/898792374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032de00640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0330100fd0 0x7f033019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032e800640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03301062f0 0x7f03301a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032d400640 1 -- 172.21.15.40:0/898792374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03301062f0 msgr2=0x7f03301a6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:13.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032d400640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03301062f0 0x7f03301a6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032d400640 1 -- 172.21.15.40:0/898792374 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0330100fd0 msgr2=0x7f033019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:13.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032d400640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0330100fd0 0x7f033019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032d400640 1 -- 172.21.15.40:0/898792374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03301a6b80 con 0x7f0330103940 2024-04-26T23:59:13.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032e800640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03301062f0 0x7f03301a6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:59:13.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032de00640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0330100fd0 0x7f033019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:13.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.538+0000 7f032d400640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0330103940 0x7f033019f4a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0318009d50 tx=0x7f031800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:13.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.539+0000 7f0326a00640 1 -- 172.21.15.40:0/898792374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318003b70 con 0x7f0330103940 2024-04-26T23:59:13.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.539+0000 7f0326a00640 1 -- 172.21.15.40:0/898792374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0318003d10 con 0x7f0330103940 2024-04-26T23:59:13.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.539+0000 7f0326a00640 1 -- 172.21.15.40:0/898792374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03180108a0 con 0x7f0330103940 2024-04-26T23:59:13.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.539+0000 7f032f200640 1 -- 172.21.15.40:0/898792374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03301a6e70 con 0x7f0330103940 2024-04-26T23:59:13.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.539+0000 7f032f200640 1 -- 172.21.15.40:0/898792374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03301a72e0 con 0x7f0330103940 2024-04-26T23:59:13.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.540+0000 7f032f200640 1 -- 172.21.15.40:0/898792374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03300fe6a0 con 0x7f0330103940 2024-04-26T23:59:13.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.542+0000 7f0326a00640 1 -- 172.21.15.40:0/898792374 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0318004000 con 0x7f0330103940 2024-04-26T23:59:13.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.544+0000 7f0326a00640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f03100a3c30 0x7f03100a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:13.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.544+0000 7f0326a00640 1 -- 172.21.15.40:0/898792374 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f03180d0d50 con 0x7f0330103940 2024-04-26T23:59:13.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.544+0000 7f032de00640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f03100a3c30 0x7f03100a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:13.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.545+0000 7f032de00640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f03100a3c30 0x7f03100a60f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0320000c00 tx=0x7f03200314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:13.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:13 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2389647818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:59:13.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:13 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2471747757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:59:13.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.548+0000 7f0326a00640 1 -- 172.21.15.40:0/898792374 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f031809d490 con 0x7f0330103940 2024-04-26T23:59:13.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:13 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2389647818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:59:13.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:13 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2471747757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:59:13.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:13 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2389647818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:59:13.746 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:13 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2471747757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:59:13.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.768+0000 7f032f200640 1 -- 172.21.15.40:0/898792374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f033002dad0 con 0x7f0330103940 2024-04-26T23:59:13.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.769+0000 7f0326a00640 1 -- 172.21.15.40:0/898792374 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+588 (secure 0 0 0) 0x7f031809cd90 con 0x7f0330103940 2024-04-26T23:59:13.771 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1959424465 2024-04-27T23:50:44.546469+0000 2024-04-26T23:59:13.771 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/1568598447 2024-04-27T23:50:44.546469+0000 2024-04-26T23:59:13.771 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/29889962 2024-04-27T23:50:44.546469+0000 2024-04-26T23:59:13.771 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/673764112 2024-04-27T23:50:09.399879+0000 2024-04-26T23:59:13.772 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3782482730 2024-04-27T23:50:09.399879+0000 2024-04-26T23:59:13.772 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/1568598447 2024-04-27T23:50:44.546469+0000 2024-04-26T23:59:13.772 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1079307273 2024-04-27T23:50:09.399879+0000 2024-04-26T23:59:13.772 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/673764112 2024-04-27T23:50:09.399879+0000 2024-04-26T23:59:13.772 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4238592841 2024-04-27T23:50:44.546469+0000 2024-04-26T23:59:13.772 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4067589404 2024-04-27T23:50:09.399879+0000 2024-04-26T23:59:13.772 INFO:teuthology.orchestra.run.smithi040.stderr:listed 10 entries 2024-04-26T23:59:13.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.772+0000 7f032f200640 1 -- 172.21.15.40:0/898792374 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f03100a3c30 msgr2=0x7f03100a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:13.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.772+0000 7f032f200640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f03100a3c30 0x7f03100a60f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0320000c00 tx=0x7f03200314a0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.772+0000 7f032f200640 1 -- 172.21.15.40:0/898792374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0330103940 msgr2=0x7f033019f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:13.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.772+0000 7f032f200640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0330103940 0x7f033019f4a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0318009d50 tx=0x7f031800c6a0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.772+0000 7f032f200640 1 -- 172.21.15.40:0/898792374 shutdown_connections 2024-04-26T23:59:13.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.772+0000 7f032f200640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f03100a3c30 0x7f03100a60f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.772+0000 7f032f200640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03301062f0 0x7f03301a6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.772+0000 7f032f200640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0330103940 0x7f033019f4a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.772+0000 7f032f200640 1 --2- 172.21.15.40:0/898792374 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0330100fd0 0x7f033019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:13.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.772+0000 7f032f200640 1 -- 172.21.15.40:0/898792374 >> 172.21.15.40:0/898792374 conn(0x7f03300fcbb0 msgr2=0x7f03301042f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:13.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.772+0000 7f032f200640 1 -- 172.21.15.40:0/898792374 shutdown_connections 2024-04-26T23:59:13.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:13.772+0000 7f032f200640 1 -- 172.21.15.40:0/898792374 wait complete. 2024-04-26T23:59:13.793 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-04-26T23:59:13.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:13.793 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:59:13.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:ceph-fuse[69638]: fuse finished with error 0 and tester_r 0 2024-04-26T23:59:13.883 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:59:14.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:14 smithi040 ceph-mon[28095]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:59:14.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:14 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3075044832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:59:14.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:14 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/898792374' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:59:14.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:14 smithi080 ceph-mon[32362]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:59:14.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:14 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3075044832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:59:14.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:14 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/898792374' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:59:14.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:14 smithi103 ceph-mon[32952]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:59:14.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:14 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3075044832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:59:14.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:14 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/898792374' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:59:16.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:16 smithi080 ceph-mon[32362]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:59:16.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:16 smithi103 ceph-mon[32952]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:59:16.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:16 smithi040 ceph-mon[28095]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:59:18.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:18 smithi080 ceph-mon[32362]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-26T23:59:18.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:18 smithi103 ceph-mon[32952]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-26T23:59:18.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:18 smithi040 ceph-mon[28095]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-26T23:59:19.984 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-26T23:59:19.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:59:19.985 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-26T23:59:20.058 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:59:20.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.278+0000 7f8802800640 1 -- 172.21.15.40:0/1370641623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87fc108d90 msgr2=0x7f87fc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:20.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.278+0000 7f8802800640 1 --2- 172.21.15.40:0/1370641623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87fc108d90 0x7f87fc1091b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009860 tx=0x7f87f002a340 comp rx=0 tx=0).stop 2024-04-26T23:59:20.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.278+0000 7f8802800640 1 -- 172.21.15.40:0/1370641623 shutdown_connections 2024-04-26T23:59:20.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.278+0000 7f8802800640 1 --2- 172.21.15.40:0/1370641623 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f87fc10a940 0x7f87fc1111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.278+0000 7f8802800640 1 --2- 172.21.15.40:0/1370641623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87fc109f80 0x7f87fc10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.278+0000 7f8802800640 1 --2- 172.21.15.40:0/1370641623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87fc108d90 0x7f87fc1091b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.278+0000 7f8802800640 1 -- 172.21.15.40:0/1370641623 >> 172.21.15.40:0/1370641623 conn(0x7f87fc077d30 msgr2=0x7f87fc07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.278+0000 7f8802800640 1 -- 172.21.15.40:0/1370641623 shutdown_connections 2024-04-26T23:59:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.278+0000 7f8802800640 1 -- 172.21.15.40:0/1370641623 wait complete. 2024-04-26T23:59:20.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.279+0000 7f8802800640 1 Processor -- start 2024-04-26T23:59:20.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.279+0000 7f8802800640 1 -- start start 2024-04-26T23:59:20.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8802800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f87fc108d90 0x7f87fc19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:20.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8802800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87fc109f80 0x7f87fc19f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:20.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8802800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87fc10a940 0x7f87fc1a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8802800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87fc114890 con 0x7f87fc10a940 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8802800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f87fc114710 con 0x7f87fc108d90 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8802800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87fc114a10 con 0x7f87fc109f80 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8801e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87fc10a940 0x7f87fc1a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8800a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87fc109f80 0x7f87fc19f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8801e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87fc10a940 0x7f87fc1a6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49296/0 (socket says 172.21.15.40:49296) 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8801e00640 1 -- 172.21.15.40:0/2659596820 learned_addr learned my addr 172.21.15.40:0/2659596820 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8801400640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f87fc108d90 0x7f87fc19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8800a00640 1 -- 172.21.15.40:0/2659596820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f87fc108d90 msgr2=0x7f87fc19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.280+0000 7f8800a00640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f87fc108d90 0x7f87fc19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.281+0000 7f8800a00640 1 -- 172.21.15.40:0/2659596820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87fc10a940 msgr2=0x7f87fc1a6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.281+0000 7f8800a00640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87fc10a940 0x7f87fc1a6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.281+0000 7f8800a00640 1 -- 172.21.15.40:0/2659596820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87fc1a6bc0 con 0x7f87fc109f80 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.281+0000 7f8801e00640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87fc10a940 0x7f87fc1a6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.281+0000 7f8800a00640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87fc109f80 0x7f87fc19f580 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f87e8005fd0 tx=0x7f87e800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:20.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.281+0000 7f87fa000640 1 -- 172.21.15.40:0/2659596820 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e800cd00 con 0x7f87fc109f80 2024-04-26T23:59:20.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.281+0000 7f87fa000640 1 -- 172.21.15.40:0/2659596820 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87e80035f0 con 0x7f87fc109f80 2024-04-26T23:59:20.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.281+0000 7f8802800640 1 -- 172.21.15.40:0/2659596820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87fc1a6eb0 con 0x7f87fc109f80 2024-04-26T23:59:20.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.281+0000 7f87fa000640 1 -- 172.21.15.40:0/2659596820 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e8010690 con 0x7f87fc109f80 2024-04-26T23:59:20.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.281+0000 7f8802800640 1 -- 172.21.15.40:0/2659596820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87fc1a7370 con 0x7f87fc109f80 2024-04-26T23:59:20.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.282+0000 7f8802800640 1 -- 172.21.15.40:0/2659596820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87fc110180 con 0x7f87fc109f80 2024-04-26T23:59:20.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.285+0000 7f87fa000640 1 -- 172.21.15.40:0/2659596820 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f87e8003a80 con 0x7f87fc109f80 2024-04-26T23:59:20.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.286+0000 7f87fa000640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f87e40a3bb0 0x7f87e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:20.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.286+0000 7f87fa000640 1 -- 172.21.15.40:0/2659596820 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f87e80d0810 con 0x7f87fc109f80 2024-04-26T23:59:20.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.286+0000 7f8801400640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f87e40a3bb0 0x7f87e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:20.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.287+0000 7f8801400640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f87e40a3bb0 0x7f87e40a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87f002a930 tx=0x7f87f0030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:20.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.290+0000 7f87fa000640 1 -- 172.21.15.40:0/2659596820 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f87e809cf50 con 0x7f87fc109f80 2024-04-26T23:59:20.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:20 smithi040 ceph-mon[28095]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-26T23:59:20.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.496+0000 7f8802800640 1 -- 172.21.15.40:0/2659596820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87fc0635d0 con 0x7f87fc109f80 2024-04-26T23:59:20.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.498+0000 7f87fa000640 1 -- 172.21.15.40:0/2659596820 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24090 (secure 0 0 0) 0x7f87e809c850 con 0x7f87fc109f80 2024-04-26T23:59:20.500 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:59:20.500 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:58:55.649955+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:59:20.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.501+0000 7f8802800640 1 -- 172.21.15.40:0/2659596820 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f87e40a3bb0 msgr2=0x7f87e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:20.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.501+0000 7f8802800640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f87e40a3bb0 0x7f87e40a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87f002a930 tx=0x7f87f0030470 comp rx=0 tx=0).stop 2024-04-26T23:59:20.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.501+0000 7f8802800640 1 -- 172.21.15.40:0/2659596820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87fc109f80 msgr2=0x7f87fc19f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:20.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.501+0000 7f8802800640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87fc109f80 0x7f87fc19f580 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f87e8005fd0 tx=0x7f87e800ee50 comp rx=0 tx=0).stop 2024-04-26T23:59:20.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.501+0000 7f8802800640 1 -- 172.21.15.40:0/2659596820 shutdown_connections 2024-04-26T23:59:20.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.501+0000 7f8802800640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87fc10a940 0x7f87fc1a6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.501+0000 7f8802800640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f87e40a3bb0 0x7f87e40a6070 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.501+0000 7f8802800640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87fc109f80 0x7f87fc19f580 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.501+0000 7f8802800640 1 --2- 172.21.15.40:0/2659596820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f87fc108d90 0x7f87fc19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.501+0000 7f8802800640 1 -- 172.21.15.40:0/2659596820 >> 172.21.15.40:0/2659596820 conn(0x7f87fc077d30 msgr2=0x7f87fc10d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:20.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.501+0000 7f8802800640 1 -- 172.21.15.40:0/2659596820 shutdown_connections 2024-04-26T23:59:20.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.501+0000 7f8802800640 1 -- 172.21.15.40:0/2659596820 wait complete. 2024-04-26T23:59:20.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:59:20.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:20 smithi103 ceph-mon[32952]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-26T23:59:20.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.738+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3304659821 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0abc10ab40 msgr2=0x7f0abc1113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:20.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.738+0000 7f0ac3a00640 1 --2- 172.21.15.40:0/3304659821 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0abc10ab40 0x7f0abc1113d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4009ca0 tx=0x7f0ab402a8c0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.738+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3304659821 shutdown_connections 2024-04-26T23:59:20.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.738+0000 7f0ac3a00640 1 --2- 172.21.15.40:0/3304659821 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0abc10ab40 0x7f0abc1113d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.738+0000 7f0ac3a00640 1 --2- 172.21.15.40:0/3304659821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0abc10a180 0x7f0abc10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.738+0000 7f0ac3a00640 1 --2- 172.21.15.40:0/3304659821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0abc108f90 0x7f0abc1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.738+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3304659821 >> 172.21.15.40:0/3304659821 conn(0x7f0abc077f40 msgr2=0x7f0abc07a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:20.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.738+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3304659821 shutdown_connections 2024-04-26T23:59:20.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.738+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3304659821 wait complete. 2024-04-26T23:59:20.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.739+0000 7f0ac3a00640 1 Processor -- start 2024-04-26T23:59:20.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.740+0000 7f0ac3a00640 1 -- start start 2024-04-26T23:59:20.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.740+0000 7f0ac3a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0abc108f90 0x7f0abc19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.740+0000 7f0ac3a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0abc10a180 0x7f0abc19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.740+0000 7f0ac3a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0abc10ab40 0x7f0abc1a6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.740+0000 7f0ac3a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0abc114b20 con 0x7f0abc10ab40 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.740+0000 7f0ac3a00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0abc1149a0 con 0x7f0abc108f90 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.740+0000 7f0ac3a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0abc114ca0 con 0x7f0abc10a180 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.740+0000 7f0ac1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0abc10ab40 0x7f0abc1a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.740+0000 7f0ab3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0abc10a180 0x7f0abc19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.740+0000 7f0ac1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0abc10ab40 0x7f0abc1a6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49314/0 (socket says 172.21.15.40:49314) 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.740+0000 7f0ac1a00640 1 -- 172.21.15.40:0/3655018628 learned_addr learned my addr 172.21.15.40:0/3655018628 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.740+0000 7f0ac1000640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0abc108f90 0x7f0abc19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.741+0000 7f0ab3e00640 1 -- 172.21.15.40:0/3655018628 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0abc108f90 msgr2=0x7f0abc19f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.741+0000 7f0ab3e00640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0abc108f90 0x7f0abc19f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.741+0000 7f0ab3e00640 1 -- 172.21.15.40:0/3655018628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0abc10ab40 msgr2=0x7f0abc1a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:20.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.741+0000 7f0ab3e00640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0abc10ab40 0x7f0abc1a6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.741+0000 7f0ab3e00640 1 -- 172.21.15.40:0/3655018628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0abc1a6ce0 con 0x7f0abc10a180 2024-04-26T23:59:20.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.741+0000 7f0ab3e00640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0abc10a180 0x7f0abc19f600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0aa400e9f0 tx=0x7f0aa400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:20.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.741+0000 7f0aba000640 1 -- 172.21.15.40:0/3655018628 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa400ccc0 con 0x7f0abc10a180 2024-04-26T23:59:20.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.741+0000 7f0aba000640 1 -- 172.21.15.40:0/3655018628 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0aa40108e0 con 0x7f0abc10a180 2024-04-26T23:59:20.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.741+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3655018628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0abc1a6fd0 con 0x7f0abc10a180 2024-04-26T23:59:20.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.742+0000 7f0aba000640 1 -- 172.21.15.40:0/3655018628 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa4018880 con 0x7f0abc10a180 2024-04-26T23:59:20.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.742+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3655018628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0abc1a74e0 con 0x7f0abc10a180 2024-04-26T23:59:20.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.743+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3655018628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0abc110250 con 0x7f0abc10a180 2024-04-26T23:59:20.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.745+0000 7f0aba000640 1 -- 172.21.15.40:0/3655018628 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0aa4010430 con 0x7f0abc10a180 2024-04-26T23:59:20.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.746+0000 7f0aba000640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0aa00a3c30 0x7f0aa00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:20.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.747+0000 7f0aba000640 1 -- 172.21.15.40:0/3655018628 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f0aa40d08a0 con 0x7f0abc10a180 2024-04-26T23:59:20.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.747+0000 7f0ac1000640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0aa00a3c30 0x7f0aa00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:20.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.747+0000 7f0ac1000640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0aa00a3c30 0x7f0aa00a60f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0abc109fe0 tx=0x7f0aac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:20.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.749+0000 7f0aba000640 1 -- 172.21.15.40:0/3655018628 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0aa409c990 con 0x7f0abc10a180 2024-04-26T23:59:20.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:20 smithi080 ceph-mon[32362]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-26T23:59:20.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.958+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3655018628 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0abc02dd20 con 0x7f0abc10a180 2024-04-26T23:59:20.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.960+0000 7f0aba000640 1 -- 172.21.15.40:0/3655018628 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24090 (secure 0 0 0) 0x7f0aa40a0c90 con 0x7f0abc10a180 2024-04-26T23:59:20.963 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:59:20.963 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:58:55.649955+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:59:20.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.963+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3655018628 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0aa00a3c30 msgr2=0x7f0aa00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:20.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.963+0000 7f0ac3a00640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0aa00a3c30 0x7f0aa00a60f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0abc109fe0 tx=0x7f0aac005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:20.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.963+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3655018628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0abc10a180 msgr2=0x7f0abc19f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:20.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.963+0000 7f0ac3a00640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0abc10a180 0x7f0abc19f600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0aa400e9f0 tx=0x7f0aa400eeb0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.963+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3655018628 shutdown_connections 2024-04-26T23:59:20.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.963+0000 7f0ac3a00640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0abc10ab40 0x7f0abc1a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.963+0000 7f0ac3a00640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0aa00a3c30 0x7f0aa00a60f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.963+0000 7f0ac3a00640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0abc10a180 0x7f0abc19f600 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.963+0000 7f0ac3a00640 1 --2- 172.21.15.40:0/3655018628 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0abc108f90 0x7f0abc19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:20.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.963+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3655018628 >> 172.21.15.40:0/3655018628 conn(0x7f0abc077f40 msgr2=0x7f0abc10d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:20.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.964+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3655018628 shutdown_connections 2024-04-26T23:59:20.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:20.964+0000 7f0ac3a00640 1 -- 172.21.15.40:0/3655018628 wait complete. 2024-04-26T23:59:20.984 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T23:59:21.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.197+0000 7f46aa200640 1 -- 172.21.15.40:0/343763311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a4076970 msgr2=0x7f46a4074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:21.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.197+0000 7f46aa200640 1 --2- 172.21.15.40:0/343763311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a4076970 0x7f46a4074dc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4698009930 tx=0x7f469802a420 comp rx=0 tx=0).stop 2024-04-26T23:59:21.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.197+0000 7f46aa200640 1 -- 172.21.15.40:0/343763311 shutdown_connections 2024-04-26T23:59:21.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.197+0000 7f46aa200640 1 --2- 172.21.15.40:0/343763311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46a410ee10 0x7f46a4111200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:21.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.197+0000 7f46aa200640 1 --2- 172.21.15.40:0/343763311 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f46a4075390 0x7f46a4075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:21.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.197+0000 7f46aa200640 1 --2- 172.21.15.40:0/343763311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a4076970 0x7f46a4074dc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:21.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.197+0000 7f46aa200640 1 -- 172.21.15.40:0/343763311 >> 172.21.15.40:0/343763311 conn(0x7f46a40fddd0 msgr2=0x7f46a41001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:21.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.197+0000 7f46aa200640 1 -- 172.21.15.40:0/343763311 shutdown_connections 2024-04-26T23:59:21.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.197+0000 7f46aa200640 1 -- 172.21.15.40:0/343763311 wait complete. 2024-04-26T23:59:21.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.198+0000 7f46aa200640 1 Processor -- start 2024-04-26T23:59:21.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.198+0000 7f46aa200640 1 -- start start 2024-04-26T23:59:21.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.198+0000 7f46aa200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46a4075390 0x7f46a419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:21.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.198+0000 7f46aa200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f46a4076970 0x7f46a419f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:21.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.198+0000 7f46aa200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a410ee10 0x7f46a41a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:21.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46aa200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f46a41149b0 con 0x7f46a410ee10 2024-04-26T23:59:21.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46aa200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f46a4114830 con 0x7f46a4076970 2024-04-26T23:59:21.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46aa200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46a4114b30 con 0x7f46a4075390 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46a9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a410ee10 0x7f46a41a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46a3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f46a4076970 0x7f46a419f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46a9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a410ee10 0x7f46a41a6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49342/0 (socket says 172.21.15.40:49342) 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46a8e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46a4075390 0x7f46a419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46a9800640 1 -- 172.21.15.40:0/4278567246 learned_addr learned my addr 172.21.15.40:0/4278567246 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46a3e00640 1 -- 172.21.15.40:0/4278567246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46a4075390 msgr2=0x7f46a419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46a3e00640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46a4075390 0x7f46a419f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46a3e00640 1 -- 172.21.15.40:0/4278567246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a410ee10 msgr2=0x7f46a41a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46a3e00640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a410ee10 0x7f46a41a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46a3e00640 1 -- 172.21.15.40:0/4278567246 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a41a6cc0 con 0x7f46a4076970 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.199+0000 7f46a9800640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a410ee10 0x7f46a41a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.200+0000 7f46a3e00640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f46a4076970 0x7f46a419f5e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f469000e9f0 tx=0x7f469000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:21.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.200+0000 7f46a1600640 1 -- 172.21.15.40:0/4278567246 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469000ccc0 con 0x7f46a4076970 2024-04-26T23:59:21.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.200+0000 7f46aa200640 1 -- 172.21.15.40:0/4278567246 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a41a6fb0 con 0x7f46a4076970 2024-04-26T23:59:21.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.200+0000 7f46a1600640 1 -- 172.21.15.40:0/4278567246 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46900108e0 con 0x7f46a4076970 2024-04-26T23:59:21.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.200+0000 7f46a1600640 1 -- 172.21.15.40:0/4278567246 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690018880 con 0x7f46a4076970 2024-04-26T23:59:21.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.200+0000 7f46aa200640 1 -- 172.21.15.40:0/4278567246 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a41a74c0 con 0x7f46a4076970 2024-04-26T23:59:21.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.201+0000 7f46aa200640 1 -- 172.21.15.40:0/4278567246 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a4104fe0 con 0x7f46a4076970 2024-04-26T23:59:21.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.204+0000 7f46a1600640 1 -- 172.21.15.40:0/4278567246 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4690010430 con 0x7f46a4076970 2024-04-26T23:59:21.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.206+0000 7f46a1600640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f468c0a3c30 0x7f468c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:21.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.206+0000 7f46a1600640 1 -- 172.21.15.40:0/4278567246 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f46900d0d70 con 0x7f46a4076970 2024-04-26T23:59:21.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.206+0000 7f46a8e00640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f468c0a3c30 0x7f468c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:21.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.207+0000 7f46a8e00640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f468c0a3c30 0x7f468c0a60f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4698005ab0 tx=0x7f4698005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:21.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.207+0000 7f46a1600640 1 -- 172.21.15.40:0/4278567246 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f469009ce60 con 0x7f46a4076970 2024-04-26T23:59:21.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:21.457+0000 7f46aa200640 1 -- 172.21.15.40:0/4278567246 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f46a402dad0 con 0x7f46a4076970 2024-04-26T23:59:21.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:21 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2659596820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:21.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:21 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3655018628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:21.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:21 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2659596820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:21.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:21 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3655018628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:21.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:21 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2659596820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:21.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:21 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3655018628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:22.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.320+0000 7f46a1600640 1 -- 172.21.15.40:0/4278567246 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f46900a1160 con 0x7f46a4076970 2024-04-26T23:59:22.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.323+0000 7f46aa200640 1 -- 172.21.15.40:0/4278567246 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f468c0a3c30 msgr2=0x7f468c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:22.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.323+0000 7f46aa200640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f468c0a3c30 0x7f468c0a60f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4698005ab0 tx=0x7f4698005a20 comp rx=0 tx=0).stop 2024-04-26T23:59:22.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.323+0000 7f46aa200640 1 -- 172.21.15.40:0/4278567246 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f46a4076970 msgr2=0x7f46a419f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:22.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.323+0000 7f46aa200640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f46a4076970 0x7f46a419f5e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f469000e9f0 tx=0x7f469000eeb0 comp rx=0 tx=0).stop 2024-04-26T23:59:22.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.323+0000 7f46aa200640 1 -- 172.21.15.40:0/4278567246 shutdown_connections 2024-04-26T23:59:22.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.323+0000 7f46aa200640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46a410ee10 0x7f46a41a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:22.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.323+0000 7f46aa200640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f46a4076970 0x7f46a419f5e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:22.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.323+0000 7f46aa200640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f468c0a3c30 0x7f468c0a60f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:22.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.323+0000 7f46aa200640 1 --2- 172.21.15.40:0/4278567246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46a4075390 0x7f46a419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:22.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.323+0000 7f46aa200640 1 -- 172.21.15.40:0/4278567246 >> 172.21.15.40:0/4278567246 conn(0x7f46a40fddd0 msgr2=0x7f46a41021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:22.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.324+0000 7f46aa200640 1 -- 172.21.15.40:0/4278567246 shutdown_connections 2024-04-26T23:59:22.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.324+0000 7f46aa200640 1 -- 172.21.15.40:0/4278567246 wait complete. 2024-04-26T23:59:22.353 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-26T23:59:22.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-26T23:59:22.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.572+0000 7f4cbf200640 1 -- 172.21.15.40:0/2002516643 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cc0105c60 msgr2=0x7f4cc01060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:22.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.572+0000 7f4cbf200640 1 --2- 172.21.15.40:0/2002516643 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cc0105c60 0x7f4cc01060e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009900 tx=0x7f4cac02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:59:22.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.572+0000 7f4cbf200640 1 -- 172.21.15.40:0/2002516643 shutdown_connections 2024-04-26T23:59:22.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.572+0000 7f4cbf200640 1 --2- 172.21.15.40:0/2002516643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cc0106620 0x7f4cc010ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:22.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.572+0000 7f4cbf200640 1 --2- 172.21.15.40:0/2002516643 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cc0105c60 0x7f4cc01060e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:22.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.572+0000 7f4cbf200640 1 --2- 172.21.15.40:0/2002516643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cc0104a70 0x7f4cc0104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:22.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.572+0000 7f4cbf200640 1 -- 172.21.15.40:0/2002516643 >> 172.21.15.40:0/2002516643 conn(0x7f4cc00fffe0 msgr2=0x7f4cc0102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:22.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.572+0000 7f4cbf200640 1 -- 172.21.15.40:0/2002516643 shutdown_connections 2024-04-26T23:59:22.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.572+0000 7f4cbf200640 1 -- 172.21.15.40:0/2002516643 wait complete. 2024-04-26T23:59:22.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.573+0000 7f4cbf200640 1 Processor -- start 2024-04-26T23:59:22.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.573+0000 7f4cbf200640 1 -- start start 2024-04-26T23:59:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.573+0000 7f4cbf200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cc0104a70 0x7f4cc019ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.573+0000 7f4cbf200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cc0105c60 0x7f4cc019b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbf200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cc0106620 0x7f4cc01a2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbf200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc01105d0 con 0x7f4cc0106620 2024-04-26T23:59:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbf200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc0110450 con 0x7f4cc0105c60 2024-04-26T23:59:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbf200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc0110750 con 0x7f4cc0104a70 2024-04-26T23:59:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbde00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cc0104a70 0x7f4cc019ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cc0106620 0x7f4cc01a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbde00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cc0104a70 0x7f4cc019ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47958/0 (socket says 172.21.15.40:47958) 2024-04-26T23:59:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbde00640 1 -- 172.21.15.40:0/2030862569 learned_addr learned my addr 172.21.15.40:0/2030862569 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbd400640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cc0105c60 0x7f4cc019b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:22.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbe800640 1 -- 172.21.15.40:0/2030862569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cc0104a70 msgr2=0x7f4cc019ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:22.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbe800640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cc0104a70 0x7f4cc019ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:22.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbe800640 1 -- 172.21.15.40:0/2030862569 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cc0105c60 msgr2=0x7f4cc019b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:22.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbe800640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cc0105c60 0x7f4cc019b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:22.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbe800640 1 -- 172.21.15.40:0/2030862569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc01a28b0 con 0x7f4cc0106620 2024-04-26T23:59:22.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.574+0000 7f4cbde00640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cc0104a70 0x7f4cc019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:22.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.575+0000 7f4cbe800640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cc0106620 0x7f4cc01a2210 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800e970 tx=0x7f4cb800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:22.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.575+0000 7f4cb6a00640 1 -- 172.21.15.40:0/2030862569 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb800ccc0 con 0x7f4cc0106620 2024-04-26T23:59:22.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.575+0000 7f4cb6a00640 1 -- 172.21.15.40:0/2030862569 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cb8005020 con 0x7f4cc0106620 2024-04-26T23:59:22.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.575+0000 7f4cbf200640 1 -- 172.21.15.40:0/2030862569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc01a2ba0 con 0x7f4cc0106620 2024-04-26T23:59:22.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.575+0000 7f4cb6a00640 1 -- 172.21.15.40:0/2030862569 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8004a00 con 0x7f4cc0106620 2024-04-26T23:59:22.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.575+0000 7f4cbf200640 1 -- 172.21.15.40:0/2030862569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc01a3070 con 0x7f4cc0106620 2024-04-26T23:59:22.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.576+0000 7f4cbf200640 1 -- 172.21.15.40:0/2030862569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc010be10 con 0x7f4cc0106620 2024-04-26T23:59:22.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.578+0000 7f4cb6a00640 1 -- 172.21.15.40:0/2030862569 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4cb8004ba0 con 0x7f4cc0106620 2024-04-26T23:59:22.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.580+0000 7f4cb6a00640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4ca40a3bb0 0x7f4ca40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:22.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.580+0000 7f4cb6a00640 1 -- 172.21.15.40:0/2030862569 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4cb80d05e0 con 0x7f4cc0106620 2024-04-26T23:59:22.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.580+0000 7f4cbde00640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4ca40a3bb0 0x7f4ca40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:22.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.581+0000 7f4cbde00640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4ca40a3bb0 0x7f4ca40a6070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4cc019bb40 tx=0x7f4cb0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:22.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.584+0000 7f4cb6a00640 1 -- 172.21.15.40:0/2030862569 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4cb809cd20 con 0x7f4cc0106620 2024-04-26T23:59:22.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:22 smithi103 ceph-mon[32952]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-26T23:59:22.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:22 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/4278567246' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T23:59:22.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:22 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T23:59:22.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:22 smithi040 ceph-mon[28095]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-26T23:59:22.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:22 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/4278567246' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T23:59:22.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:22 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T23:59:22.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:22 smithi080 ceph-mon[32362]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-26T23:59:22.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:22 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/4278567246' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T23:59:22.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:22 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T23:59:22.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:22.835+0000 7f4cbf200640 1 -- 172.21.15.40:0/2030862569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f4cc0063530 con 0x7f4cc0106620 2024-04-26T23:59:23.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.329+0000 7f4cb6a00640 1 -- 172.21.15.40:0/2030862569 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f4cb809c620 con 0x7f4cc0106620 2024-04-26T23:59:23.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.333+0000 7f4cbf200640 1 -- 172.21.15.40:0/2030862569 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4ca40a3bb0 msgr2=0x7f4ca40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:23.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.333+0000 7f4cbf200640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4ca40a3bb0 0x7f4ca40a6070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4cc019bb40 tx=0x7f4cb0009500 comp rx=0 tx=0).stop 2024-04-26T23:59:23.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.333+0000 7f4cbf200640 1 -- 172.21.15.40:0/2030862569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cc0106620 msgr2=0x7f4cc01a2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:23.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.333+0000 7f4cbf200640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cc0106620 0x7f4cc01a2210 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800e970 tx=0x7f4cb800ee30 comp rx=0 tx=0).stop 2024-04-26T23:59:23.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.334+0000 7f4cbf200640 1 -- 172.21.15.40:0/2030862569 shutdown_connections 2024-04-26T23:59:23.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.334+0000 7f4cbf200640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cc0106620 0x7f4cc01a2210 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:23.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.334+0000 7f4cbf200640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4cc0105c60 0x7f4cc019b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:23.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.334+0000 7f4cbf200640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4ca40a3bb0 0x7f4ca40a6070 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:23.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.334+0000 7f4cbf200640 1 --2- 172.21.15.40:0/2030862569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cc0104a70 0x7f4cc019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:23.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.334+0000 7f4cbf200640 1 -- 172.21.15.40:0/2030862569 >> 172.21.15.40:0/2030862569 conn(0x7f4cc00fffe0 msgr2=0x7f4cc0108fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:23.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.334+0000 7f4cbf200640 1 -- 172.21.15.40:0/2030862569 shutdown_connections 2024-04-26T23:59:23.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.334+0000 7f4cbf200640 1 -- 172.21.15.40:0/2030862569 wait complete. 2024-04-26T23:59:23.365 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-26T23:59:23.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.553+0000 7f7b83200640 1 -- 172.21.15.40:0/502412940 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b841008c0 msgr2=0x7f7b84100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:23.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.553+0000 7f7b83200640 1 --2- 172.21.15.40:0/502412940 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b841008c0 0x7f7b84100d40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7b70009860 tx=0x7f7b7002a340 comp rx=0 tx=0).stop 2024-04-26T23:59:23.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.553+0000 7f7b83200640 1 -- 172.21.15.40:0/502412940 shutdown_connections 2024-04-26T23:59:23.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.553+0000 7f7b83200640 1 --2- 172.21.15.40:0/502412940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b84101280 0x7f7b8410ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:23.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.553+0000 7f7b83200640 1 --2- 172.21.15.40:0/502412940 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b841008c0 0x7f7b84100d40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:23.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.553+0000 7f7b83200640 1 --2- 172.21.15.40:0/502412940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b84103e50 0x7f7b84104270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:23.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.553+0000 7f7b83200640 1 -- 172.21.15.40:0/502412940 >> 172.21.15.40:0/502412940 conn(0x7f7b840fc460 msgr2=0x7f7b840fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:23.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.553+0000 7f7b83200640 1 -- 172.21.15.40:0/502412940 shutdown_connections 2024-04-26T23:59:23.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.553+0000 7f7b83200640 1 -- 172.21.15.40:0/502412940 wait complete. 2024-04-26T23:59:23.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.555+0000 7f7b83200640 1 Processor -- start 2024-04-26T23:59:23.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.555+0000 7f7b83200640 1 -- start start 2024-04-26T23:59:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.555+0000 7f7b83200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b841008c0 0x7f7b8419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b83200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b84101280 0x7f7b8419b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b83200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b84103e50 0x7f7b841a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b83200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8410e390 con 0x7f7b84101280 2024-04-26T23:59:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b83200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8410e210 con 0x7f7b84103e50 2024-04-26T23:59:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b83200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8410e510 con 0x7f7b841008c0 2024-04-26T23:59:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b81400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b84101280 0x7f7b8419b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b81400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b84101280 0x7f7b8419b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49374/0 (socket says 172.21.15.40:49374) 2024-04-26T23:59:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b81400640 1 -- 172.21.15.40:0/1916507302 learned_addr learned my addr 172.21.15.40:0/1916507302 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b81e00640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b841008c0 0x7f7b8419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b82800640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b84103e50 0x7f7b841a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b81400640 1 -- 172.21.15.40:0/1916507302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b841008c0 msgr2=0x7f7b8419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b81400640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b841008c0 0x7f7b8419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b81400640 1 -- 172.21.15.40:0/1916507302 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b84103e50 msgr2=0x7f7b841a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b81400640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b84103e50 0x7f7b841a2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b81400640 1 -- 172.21.15.40:0/1916507302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b841a28e0 con 0x7f7b84101280 2024-04-26T23:59:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.556+0000 7f7b82800640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b84103e50 0x7f7b841a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:59:23.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.557+0000 7f7b81400640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b84101280 0x7f7b8419b1e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7b700057c0 tx=0x7f7b70032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:23.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.557+0000 7f7b7aa00640 1 -- 172.21.15.40:0/1916507302 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b70037040 con 0x7f7b84101280 2024-04-26T23:59:23.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.557+0000 7f7b83200640 1 -- 172.21.15.40:0/1916507302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b841a2b70 con 0x7f7b84101280 2024-04-26T23:59:23.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.557+0000 7f7b7aa00640 1 -- 172.21.15.40:0/1916507302 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b70030bd0 con 0x7f7b84101280 2024-04-26T23:59:23.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.557+0000 7f7b7aa00640 1 -- 172.21.15.40:0/1916507302 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b70005280 con 0x7f7b84101280 2024-04-26T23:59:23.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.557+0000 7f7b83200640 1 -- 172.21.15.40:0/1916507302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b841a3050 con 0x7f7b84101280 2024-04-26T23:59:23.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.558+0000 7f7b83200640 1 -- 172.21.15.40:0/1916507302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b841a32e0 con 0x7f7b84101280 2024-04-26T23:59:23.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.561+0000 7f7b7aa00640 1 -- 172.21.15.40:0/1916507302 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7b70043020 con 0x7f7b84101280 2024-04-26T23:59:23.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.562+0000 7f7b7aa00640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7b680a3c30 0x7f7b680a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:23.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.562+0000 7f7b7aa00640 1 -- 172.21.15.40:0/1916507302 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f7b700efd30 con 0x7f7b84101280 2024-04-26T23:59:23.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.563+0000 7f7b81e00640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7b680a3c30 0x7f7b680a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:23.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.563+0000 7f7b81e00640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7b680a3c30 0x7f7b680a60f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b8419bb70 tx=0x7f7b74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:23.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.567+0000 7f7b7aa00640 1 -- 172.21.15.40:0/1916507302 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7b7004a020 con 0x7f7b84101280 2024-04-26T23:59:23.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:23 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T23:59:23.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:23 smithi103 ceph-mon[32952]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:59:23.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:23 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2030862569' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T23:59:23.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:23 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T23:59:23.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:23 smithi040 ceph-mon[28095]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:59:23.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:23 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2030862569' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T23:59:23.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:23 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T23:59:23.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:23 smithi080 ceph-mon[32362]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:59:23.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:23 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2030862569' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T23:59:23.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:23.816+0000 7f7b83200640 1 -- 172.21.15.40:0/1916507302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f7b841a3960 con 0x7f7b84101280 2024-04-26T23:59:24.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.340+0000 7f7b7aa00640 1 -- 172.21.15.40:0/1916507302 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f7b700bc470 con 0x7f7b84101280 2024-04-26T23:59:24.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.344+0000 7f7b83200640 1 -- 172.21.15.40:0/1916507302 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7b680a3c30 msgr2=0x7f7b680a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:24.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.344+0000 7f7b83200640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7b680a3c30 0x7f7b680a60f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b8419bb70 tx=0x7f7b74009260 comp rx=0 tx=0).stop 2024-04-26T23:59:24.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.344+0000 7f7b83200640 1 -- 172.21.15.40:0/1916507302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b84101280 msgr2=0x7f7b8419b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:24.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.344+0000 7f7b83200640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b84101280 0x7f7b8419b1e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7b700057c0 tx=0x7f7b70032da0 comp rx=0 tx=0).stop 2024-04-26T23:59:24.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.345+0000 7f7b83200640 1 -- 172.21.15.40:0/1916507302 shutdown_connections 2024-04-26T23:59:24.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.345+0000 7f7b83200640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7b84103e50 0x7f7b841a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:24.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.345+0000 7f7b83200640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b84101280 0x7f7b8419b1e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:24.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.345+0000 7f7b83200640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7b680a3c30 0x7f7b680a60f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:24.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.345+0000 7f7b83200640 1 --2- 172.21.15.40:0/1916507302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b841008c0 0x7f7b8419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:24.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.345+0000 7f7b83200640 1 -- 172.21.15.40:0/1916507302 >> 172.21.15.40:0/1916507302 conn(0x7f7b840fc460 msgr2=0x7f7b841019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:24.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.345+0000 7f7b83200640 1 -- 172.21.15.40:0/1916507302 shutdown_connections 2024-04-26T23:59:24.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.345+0000 7f7b83200640 1 -- 172.21.15.40:0/1916507302 wait complete. 2024-04-26T23:59:24.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-26T23:59:24.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.567+0000 7f24d6200640 1 -- 172.21.15.40:0/3519093763 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f24d0104530 msgr2=0x7f24d010adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:24.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.567+0000 7f24d6200640 1 --2- 172.21.15.40:0/3519093763 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f24d0104530 0x7f24d010adc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f24c8009ca0 tx=0x7f24c802a8c0 comp rx=0 tx=0).stop 2024-04-26T23:59:24.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.567+0000 7f24d6200640 1 -- 172.21.15.40:0/3519093763 shutdown_connections 2024-04-26T23:59:24.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.567+0000 7f24d6200640 1 --2- 172.21.15.40:0/3519093763 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f24d0104530 0x7f24d010adc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:24.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.567+0000 7f24d6200640 1 --2- 172.21.15.40:0/3519093763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24d0103b70 0x7f24d0103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:24.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.567+0000 7f24d6200640 1 --2- 172.21.15.40:0/3519093763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24d0102980 0x7f24d0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:24.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.567+0000 7f24d6200640 1 -- 172.21.15.40:0/3519093763 >> 172.21.15.40:0/3519093763 conn(0x7f24d00fdf10 msgr2=0x7f24d0100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:24.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.567+0000 7f24d6200640 1 -- 172.21.15.40:0/3519093763 shutdown_connections 2024-04-26T23:59:24.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.567+0000 7f24d6200640 1 -- 172.21.15.40:0/3519093763 wait complete. 2024-04-26T23:59:24.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.568+0000 7f24d6200640 1 Processor -- start 2024-04-26T23:59:24.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.568+0000 7f24d6200640 1 -- start start 2024-04-26T23:59:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.568+0000 7f24d6200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24d0102980 0x7f24d00766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.568+0000 7f24d6200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f24d0103b70 0x7f24d0074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.568+0000 7f24d6200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24d0104530 0x7f24d0075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.568+0000 7f24d6200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24d010e500 con 0x7f24d0102980 2024-04-26T23:59:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.568+0000 7f24d6200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f24d010e380 con 0x7f24d0103b70 2024-04-26T23:59:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.568+0000 7f24d6200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24d010e680 con 0x7f24d0104530 2024-04-26T23:59:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.568+0000 7f24cea00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f24d0103b70 0x7f24d0074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.568+0000 7f24cf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24d0102980 0x7f24d00766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.569+0000 7f24cea00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f24d0103b70 0x7f24d0074d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:57268/0 (socket says 172.21.15.40:57268) 2024-04-26T23:59:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.569+0000 7f24cea00640 1 -- 172.21.15.40:0/2870991605 learned_addr learned my addr 172.21.15.40:0/2870991605 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.569+0000 7f24cfe00640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24d0104530 0x7f24d0075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:24.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.569+0000 7f24cea00640 1 -- 172.21.15.40:0/2870991605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24d0104530 msgr2=0x7f24d0075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:24.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.569+0000 7f24cea00640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24d0104530 0x7f24d0075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:24.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.569+0000 7f24cea00640 1 -- 172.21.15.40:0/2870991605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24d0102980 msgr2=0x7f24d00766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:24.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.569+0000 7f24cea00640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24d0102980 0x7f24d00766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:24.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.569+0000 7f24cea00640 1 -- 172.21.15.40:0/2870991605 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d00757a0 con 0x7f24d0103b70 2024-04-26T23:59:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.569+0000 7f24cea00640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f24d0103b70 0x7f24d0074d20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00e9f0 tx=0x7f24bc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.570+0000 7f24c7e00640 1 -- 172.21.15.40:0/2870991605 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc00cd80 con 0x7f24d0103b70 2024-04-26T23:59:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.570+0000 7f24c7e00640 1 -- 172.21.15.40:0/2870991605 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24bc0054d0 con 0x7f24d0103b70 2024-04-26T23:59:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.570+0000 7f24d6200640 1 -- 172.21.15.40:0/2870991605 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24d0075a90 con 0x7f24d0103b70 2024-04-26T23:59:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.570+0000 7f24c7e00640 1 -- 172.21.15.40:0/2870991605 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc010690 con 0x7f24d0103b70 2024-04-26T23:59:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.570+0000 7f24d6200640 1 -- 172.21.15.40:0/2870991605 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24d01a7650 con 0x7f24d0103b70 2024-04-26T23:59:24.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.571+0000 7f24d6200640 1 -- 172.21.15.40:0/2870991605 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24d0109ca0 con 0x7f24d0103b70 2024-04-26T23:59:24.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.573+0000 7f24c7e00640 1 -- 172.21.15.40:0/2870991605 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f24bc004e90 con 0x7f24d0103b70 2024-04-26T23:59:24.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.575+0000 7f24c7e00640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f24b40a3c30 0x7f24b40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.575+0000 7f24c7e00640 1 -- 172.21.15.40:0/2870991605 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f24bc0d09b0 con 0x7f24d0103b70 2024-04-26T23:59:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.575+0000 7f24cf400640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f24b40a3c30 0x7f24b40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:24.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.576+0000 7f24cf400640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f24b40a3c30 0x7f24b40a60f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f24c0005fd0 tx=0x7f24c000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:24.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.579+0000 7f24c7e00640 1 -- 172.21.15.40:0/2870991605 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f24bc09d0f0 con 0x7f24d0103b70 2024-04-26T23:59:24.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:24 smithi040 ceph-mon[28095]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-26T23:59:24.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:24 smithi040 ceph-mon[28095]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T23:59:24.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:24 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2030862569' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T23:59:24.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:24 smithi040 ceph-mon[28095]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T23:59:24.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:24 smithi040 ceph-mon[28095]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:59:24.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:24 smithi040 ceph-mon[28095]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-26T23:59:24.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:24 smithi040 ceph-mon[28095]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T23:59:24.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:24 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1916507302' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T23:59:24.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:24 smithi080 ceph-mon[32362]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-26T23:59:24.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:24 smithi080 ceph-mon[32362]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T23:59:24.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:24 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2030862569' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T23:59:24.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:24 smithi080 ceph-mon[32362]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T23:59:24.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:24 smithi080 ceph-mon[32362]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:59:24.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:24 smithi080 ceph-mon[32362]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-26T23:59:24.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:24 smithi080 ceph-mon[32362]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T23:59:24.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:24 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1916507302' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T23:59:24.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:24.827+0000 7f24d6200640 1 -- 172.21.15.40:0/2870991605 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f24d0063560 con 0x7f24d0103b70 2024-04-26T23:59:24.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:24 smithi103 ceph-mon[32952]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-26T23:59:24.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:24 smithi103 ceph-mon[32952]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T23:59:24.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:24 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2030862569' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T23:59:24.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:24 smithi103 ceph-mon[32952]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T23:59:24.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:24 smithi103 ceph-mon[32952]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:59:24.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:24 smithi103 ceph-mon[32952]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-26T23:59:24.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:24 smithi103 ceph-mon[32952]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T23:59:24.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:24 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1916507302' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T23:59:25.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.365+0000 7f24c7e00640 1 -- 172.21.15.40:0/2870991605 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7f24bc09c9f0 con 0x7f24d0103b70 2024-04-26T23:59:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.369+0000 7f24d6200640 1 -- 172.21.15.40:0/2870991605 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f24b40a3c30 msgr2=0x7f24b40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.369+0000 7f24d6200640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f24b40a3c30 0x7f24b40a60f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f24c0005fd0 tx=0x7f24c000b410 comp rx=0 tx=0).stop 2024-04-26T23:59:25.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.369+0000 7f24d6200640 1 -- 172.21.15.40:0/2870991605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f24d0103b70 msgr2=0x7f24d0074d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.369+0000 7f24d6200640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f24d0103b70 0x7f24d0074d20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00e9f0 tx=0x7f24bc00edf0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.369+0000 7f24d6200640 1 -- 172.21.15.40:0/2870991605 shutdown_connections 2024-04-26T23:59:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.369+0000 7f24d6200640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f24b40a3c30 0x7f24b40a60f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.369+0000 7f24d6200640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24d0104530 0x7f24d0075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.369+0000 7f24d6200640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f24d0103b70 0x7f24d0074d20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.369+0000 7f24d6200640 1 --2- 172.21.15.40:0/2870991605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24d0102980 0x7f24d00766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.369+0000 7f24d6200640 1 -- 172.21.15.40:0/2870991605 >> 172.21.15.40:0/2870991605 conn(0x7f24d00fdf10 msgr2=0x7f24d0106e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.370+0000 7f24d6200640 1 -- 172.21.15.40:0/2870991605 shutdown_connections 2024-04-26T23:59:25.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.370+0000 7f24d6200640 1 -- 172.21.15.40:0/2870991605 wait complete. 2024-04-26T23:59:25.393 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-04-26T23:59:25.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-26T23:59:25.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.595+0000 7f16c3e00640 1 -- 172.21.15.40:0/2803429198 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16c410a980 msgr2=0x7f16c41112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:25.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.595+0000 7f16c3e00640 1 --2- 172.21.15.40:0/2803429198 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16c410a980 0x7f16c41112a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009ca0 tx=0x7f16bc02a8c0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.595+0000 7f16c3e00640 1 -- 172.21.15.40:0/2803429198 shutdown_connections 2024-04-26T23:59:25.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.595+0000 7f16c3e00640 1 --2- 172.21.15.40:0/2803429198 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16c410a980 0x7f16c41112a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.595+0000 7f16c3e00640 1 --2- 172.21.15.40:0/2803429198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c4109fc0 0x7f16c410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.595+0000 7f16c3e00640 1 --2- 172.21.15.40:0/2803429198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16c4108e70 0x7f16c4109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.595+0000 7f16c3e00640 1 -- 172.21.15.40:0/2803429198 >> 172.21.15.40:0/2803429198 conn(0x7f16c4077e00 msgr2=0x7f16c407a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:25.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.595+0000 7f16c3e00640 1 -- 172.21.15.40:0/2803429198 shutdown_connections 2024-04-26T23:59:25.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.595+0000 7f16c3e00640 1 -- 172.21.15.40:0/2803429198 wait complete. 2024-04-26T23:59:25.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.596+0000 7f16c3e00640 1 Processor -- start 2024-04-26T23:59:25.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.596+0000 7f16c3e00640 1 -- start start 2024-04-26T23:59:25.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16c3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16c4108e70 0x7f16c419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:25.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16c3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16c4109fc0 0x7f16c419f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:25.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c410a980 0x7f16c41a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:25.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16c3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f16c41149c0 con 0x7f16c410a980 2024-04-26T23:59:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16c3e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f16c4114840 con 0x7f16c4109fc0 2024-04-26T23:59:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16c3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f16c4114b40 con 0x7f16c4108e70 2024-04-26T23:59:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16bbe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16c4109fc0 0x7f16c419f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16bbe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16c4109fc0 0x7f16c419f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:57288/0 (socket says 172.21.15.40:57288) 2024-04-26T23:59:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16bbe00640 1 -- 172.21.15.40:0/298366310 learned_addr learned my addr 172.21.15.40:0/298366310 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16c3400640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c410a980 0x7f16c41a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16c2a00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16c4108e70 0x7f16c419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16bbe00640 1 -- 172.21.15.40:0/298366310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16c4108e70 msgr2=0x7f16c419f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16bbe00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16c4108e70 0x7f16c419f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16bbe00640 1 -- 172.21.15.40:0/298366310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c410a980 msgr2=0x7f16c41a65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16bbe00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c410a980 0x7f16c41a65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.597+0000 7f16bbe00640 1 -- 172.21.15.40:0/298366310 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c41a6c50 con 0x7f16c4109fc0 2024-04-26T23:59:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.598+0000 7f16bbe00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16c4109fc0 0x7f16c419f570 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00e9c0 tx=0x7f16ac00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.598+0000 7f16c8a00640 1 -- 172.21.15.40:0/298366310 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac00cde0 con 0x7f16c4109fc0 2024-04-26T23:59:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.598+0000 7f16c8a00640 1 -- 172.21.15.40:0/298366310 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16ac0108e0 con 0x7f16c4109fc0 2024-04-26T23:59:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.598+0000 7f16c8a00640 1 -- 172.21.15.40:0/298366310 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac018960 con 0x7f16c4109fc0 2024-04-26T23:59:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.598+0000 7f16c3e00640 1 -- 172.21.15.40:0/298366310 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c41a6f40 con 0x7f16c4109fc0 2024-04-26T23:59:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.598+0000 7f16c3e00640 1 -- 172.21.15.40:0/298366310 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c41a7450 con 0x7f16c4109fc0 2024-04-26T23:59:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.599+0000 7f16c3e00640 1 -- 172.21.15.40:0/298366310 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c4110180 con 0x7f16c4109fc0 2024-04-26T23:59:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.601+0000 7f16c8a00640 1 -- 172.21.15.40:0/298366310 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f16ac01f020 con 0x7f16c4109fc0 2024-04-26T23:59:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.602+0000 7f16c8a00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f16a80a3c30 0x7f16a80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:25.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.602+0000 7f16c8a00640 1 -- 172.21.15.40:0/298366310 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f16ac0d0870 con 0x7f16c4109fc0 2024-04-26T23:59:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.603+0000 7f16c2a00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f16a80a3c30 0x7f16a80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:25.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.603+0000 7f16c2a00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f16a80a3c30 0x7f16a80a60f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009bd0 tx=0x7f16b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:25.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.606+0000 7f16c8a00640 1 -- 172.21.15.40:0/298366310 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f16ac054060 con 0x7f16c4109fc0 2024-04-26T23:59:25.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1916507302' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T23:59:25.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] up:active 2024-04-26T23:59:25.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-mon[28095]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T23:59:25.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-mon[28095]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T23:59:25.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-mon[28095]: Cluster is now healthy 2024-04-26T23:59:25.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-mon[28095]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-26T23:59:25.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-mon[28095]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-04-26T23:59:25.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-mon[28095]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T23:59:25.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2870991605' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T23:59:25.718 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T23:59:25.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:25 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1916507302' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T23:59:25.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:25 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] up:active 2024-04-26T23:59:25.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:25 smithi080 ceph-mon[32362]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T23:59:25.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:25 smithi080 ceph-mon[32362]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T23:59:25.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:25 smithi080 ceph-mon[32362]: Cluster is now healthy 2024-04-26T23:59:25.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:25 smithi080 ceph-mon[32362]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-26T23:59:25.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:25 smithi080 ceph-mon[32362]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-04-26T23:59:25.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:25 smithi080 ceph-mon[32362]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T23:59:25.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:25 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2870991605' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T23:59:25.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:25 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T23:59:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.842+0000 7f16c3e00640 1 -- 172.21.15.40:0/298366310 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f16c4062d70 con 0x7f16a80a3c30 2024-04-26T23:59:25.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:25 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1916507302' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T23:59:25.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:25 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] up:active 2024-04-26T23:59:25.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:25 smithi103 ceph-mon[32952]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T23:59:25.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:25 smithi103 ceph-mon[32952]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T23:59:25.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:25 smithi103 ceph-mon[32952]: Cluster is now healthy 2024-04-26T23:59:25.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:25 smithi103 ceph-mon[32952]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-26T23:59:25.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:25 smithi103 ceph-mon[32952]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-04-26T23:59:25.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:25 smithi103 ceph-mon[32952]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-26T23:59:25.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:25 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2870991605' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T23:59:25.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:25 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T23:59:25.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.876+0000 7f16c8a00640 1 -- 172.21.15.40:0/298366310 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f16c4062d70 con 0x7f16a80a3c30 2024-04-26T23:59:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.879+0000 7f16c3e00640 1 -- 172.21.15.40:0/298366310 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f16a80a3c30 msgr2=0x7f16a80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.879+0000 7f16c3e00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f16a80a3c30 0x7f16a80a60f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009bd0 tx=0x7f16b4005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.879+0000 7f16c3e00640 1 -- 172.21.15.40:0/298366310 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16c4109fc0 msgr2=0x7f16c419f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.879+0000 7f16c3e00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16c4109fc0 0x7f16c419f570 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00e9c0 tx=0x7f16ac00ee80 comp rx=0 tx=0).stop 2024-04-26T23:59:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.879+0000 7f16c3e00640 1 -- 172.21.15.40:0/298366310 shutdown_connections 2024-04-26T23:59:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.879+0000 7f16c3e00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c410a980 0x7f16c41a65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.879+0000 7f16c3e00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f16c4109fc0 0x7f16c419f570 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.879+0000 7f16c3e00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f16a80a3c30 0x7f16a80a60f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.879+0000 7f16c3e00640 1 --2- 172.21.15.40:0/298366310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16c4108e70 0x7f16c419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.879+0000 7f16c3e00640 1 -- 172.21.15.40:0/298366310 >> 172.21.15.40:0/298366310 conn(0x7f16c4077e00 msgr2=0x7f16c410d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.879+0000 7f16c3e00640 1 -- 172.21.15.40:0/298366310 shutdown_connections 2024-04-26T23:59:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:25.879+0000 7f16c3e00640 1 -- 172.21.15.40:0/298366310 wait complete. 2024-04-26T23:59:25.901 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-26T23:59:26.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.097+0000 7fa13be00640 1 -- 172.21.15.40:0/1747048391 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa13c076970 msgr2=0x7fa13c074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.097+0000 7fa13be00640 1 --2- 172.21.15.40:0/1747048391 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa13c076970 0x7fa13c074dc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009930 tx=0x7fa12c02a420 comp rx=0 tx=0).stop 2024-04-26T23:59:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.097+0000 7fa13be00640 1 -- 172.21.15.40:0/1747048391 shutdown_connections 2024-04-26T23:59:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.097+0000 7fa13be00640 1 --2- 172.21.15.40:0/1747048391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa13c10ee10 0x7fa13c111200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.097+0000 7fa13be00640 1 --2- 172.21.15.40:0/1747048391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa13c075390 0x7fa13c075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.097+0000 7fa13be00640 1 --2- 172.21.15.40:0/1747048391 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa13c076970 0x7fa13c074dc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.097+0000 7fa13be00640 1 -- 172.21.15.40:0/1747048391 >> 172.21.15.40:0/1747048391 conn(0x7fa13c0fddd0 msgr2=0x7fa13c1001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.098+0000 7fa13be00640 1 -- 172.21.15.40:0/1747048391 shutdown_connections 2024-04-26T23:59:26.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.098+0000 7fa13be00640 1 -- 172.21.15.40:0/1747048391 wait complete. 2024-04-26T23:59:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.099+0000 7fa13be00640 1 Processor -- start 2024-04-26T23:59:26.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.099+0000 7fa13be00640 1 -- start start 2024-04-26T23:59:26.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa13c075390 0x7fa13c19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:26.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa13c076970 0x7fa13c19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:26.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa13c10ee10 0x7fa13c1a6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:26.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa13c114990 con 0x7fa13c076970 2024-04-26T23:59:26.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13be00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa13c114810 con 0x7fa13c075390 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa13c114b10 con 0x7fa13c10ee10 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13aa00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa13c075390 0x7fa13c19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13aa00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa13c075390 0x7fa13c19f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:57322/0 (socket says 172.21.15.40:57322) 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13aa00640 1 -- 172.21.15.40:0/2324934802 learned_addr learned my addr 172.21.15.40:0/2324934802 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa13c076970 0x7fa13c19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13b400640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa13c10ee10 0x7fa13c1a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13aa00640 1 -- 172.21.15.40:0/2324934802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa13c10ee10 msgr2=0x7fa13c1a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13aa00640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa13c10ee10 0x7fa13c1a6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13aa00640 1 -- 172.21.15.40:0/2324934802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa13c076970 msgr2=0x7fa13c19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13aa00640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa13c076970 0x7fa13c19f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.100+0000 7fa13aa00640 1 -- 172.21.15.40:0/2324934802 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13c1a6ce0 con 0x7fa13c075390 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.101+0000 7fa13aa00640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa13c075390 0x7fa13c19f0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa12c031500 tx=0x7fa12c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.101+0000 7fa133400640 1 -- 172.21.15.40:0/2324934802 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c03f070 con 0x7fa13c075390 2024-04-26T23:59:26.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.101+0000 7fa133400640 1 -- 172.21.15.40:0/2324934802 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa12c031c50 con 0x7fa13c075390 2024-04-26T23:59:26.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.101+0000 7fa13be00640 1 -- 172.21.15.40:0/2324934802 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13c1a6f70 con 0x7fa13c075390 2024-04-26T23:59:26.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.101+0000 7fa133400640 1 -- 172.21.15.40:0/2324934802 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c035030 con 0x7fa13c075390 2024-04-26T23:59:26.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.101+0000 7fa13be00640 1 -- 172.21.15.40:0/2324934802 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa13c1a73d0 con 0x7fa13c075390 2024-04-26T23:59:26.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.102+0000 7fa13be00640 1 -- 172.21.15.40:0/2324934802 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa13c104fe0 con 0x7fa13c075390 2024-04-26T23:59:26.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.104+0000 7fa133400640 1 -- 172.21.15.40:0/2324934802 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa12c042070 con 0x7fa13c075390 2024-04-26T23:59:26.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.106+0000 7fa133400640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa1200a3c30 0x7fa1200a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:26.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.106+0000 7fa133400640 1 -- 172.21.15.40:0/2324934802 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa12c0f0cf0 con 0x7fa13c075390 2024-04-26T23:59:26.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.106+0000 7fa13a000640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa1200a3c30 0x7fa1200a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:26.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.107+0000 7fa13a000640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa1200a3c30 0x7fa1200a60f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa13c1a05d0 tx=0x7fa128005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:26.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.110+0000 7fa133400640 1 -- 172.21.15.40:0/2324934802 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa12c0bcde0 con 0x7fa13c075390 2024-04-26T23:59:26.217 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:59:25.851+0000 7f34bc600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T23:59:26.217 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:59:25.851+0000 7f34bc600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T23:59:26.217 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:59:25.851+0000 7f34bc600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T23:59:26.218 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:59:25.851+0000 7f34bc600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T23:59:26.218 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 26 23:59:25 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-26T23:59:25.851+0000 7f34bc600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T23:59:26.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.344+0000 7fa13be00640 1 -- 172.21.15.40:0/2324934802 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa13c062d70 con 0x7fa1200a3c30 2024-04-26T23:59:26.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.383+0000 7fa133400640 1 -- 172.21.15.40:0/2324934802 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa13c062d70 con 0x7fa1200a3c30 2024-04-26T23:59:26.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.386+0000 7fa13be00640 1 -- 172.21.15.40:0/2324934802 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa1200a3c30 msgr2=0x7fa1200a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:26.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.386+0000 7fa13be00640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa1200a3c30 0x7fa1200a60f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa13c1a05d0 tx=0x7fa128005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:26.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.386+0000 7fa13be00640 1 -- 172.21.15.40:0/2324934802 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa13c075390 msgr2=0x7fa13c19f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:26.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.386+0000 7fa13be00640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa13c075390 0x7fa13c19f0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa12c031500 tx=0x7fa12c003c10 comp rx=0 tx=0).stop 2024-04-26T23:59:26.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.386+0000 7fa13be00640 1 -- 172.21.15.40:0/2324934802 shutdown_connections 2024-04-26T23:59:26.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.386+0000 7fa13be00640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa1200a3c30 0x7fa1200a60f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.386+0000 7fa13be00640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa13c10ee10 0x7fa13c1a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.386+0000 7fa13be00640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa13c076970 0x7fa13c19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.386+0000 7fa13be00640 1 --2- 172.21.15.40:0/2324934802 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa13c075390 0x7fa13c19f0c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.386+0000 7fa13be00640 1 -- 172.21.15.40:0/2324934802 >> 172.21.15.40:0/2324934802 conn(0x7fa13c0fddd0 msgr2=0x7fa13c1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:26.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.387+0000 7fa13be00640 1 -- 172.21.15.40:0/2324934802 shutdown_connections 2024-04-26T23:59:26.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.387+0000 7fa13be00640 1 -- 172.21.15.40:0/2324934802 wait complete. 2024-04-26T23:59:26.408 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-26T23:59:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.626+0000 7fc10f200640 1 -- 172.21.15.40:0/511792155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc110108520 msgr2=0x7fc11010a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:26.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.626+0000 7fc10f200640 1 --2- 172.21.15.40:0/511792155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc110108520 0x7fc11010a970 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc108009900 tx=0x7fc10802a410 comp rx=0 tx=0).stop 2024-04-26T23:59:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.626+0000 7fc10f200640 1 -- 172.21.15.40:0/511792155 shutdown_connections 2024-04-26T23:59:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.626+0000 7fc10f200640 1 --2- 172.21.15.40:0/511792155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc110108520 0x7fc11010a970 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.626+0000 7fc10f200640 1 --2- 172.21.15.40:0/511792155 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc110105bf0 0x7fc110107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.626+0000 7fc10f200640 1 --2- 172.21.15.40:0/511792155 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1101032c0 0x7fc1101056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.626+0000 7fc10f200640 1 -- 172.21.15.40:0/511792155 >> 172.21.15.40:0/511792155 conn(0x7fc1100fccd0 msgr2=0x7fc1100ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.626+0000 7fc10f200640 1 -- 172.21.15.40:0/511792155 shutdown_connections 2024-04-26T23:59:26.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.626+0000 7fc10f200640 1 -- 172.21.15.40:0/511792155 wait complete. 2024-04-26T23:59:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.627+0000 7fc10f200640 1 Processor -- start 2024-04-26T23:59:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.627+0000 7fc10f200640 1 -- start start 2024-04-26T23:59:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.628+0000 7fc10f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1101032c0 0x7fc110198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.628+0000 7fc10f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc110105bf0 0x7fc110198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.628+0000 7fc10f200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc110108520 0x7fc11019ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.628+0000 7fc10f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc11010e390 con 0x7fc110105bf0 2024-04-26T23:59:26.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.628+0000 7fc10f200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fc11010e210 con 0x7fc110108520 2024-04-26T23:59:26.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.628+0000 7fc10f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc11010e510 con 0x7fc1101032c0 2024-04-26T23:59:26.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.628+0000 7fc10d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc110105bf0 0x7fc110198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:26.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.628+0000 7fc10de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1101032c0 0x7fc110198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:26.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.628+0000 7fc10e800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc110108520 0x7fc11019ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:26.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.628+0000 7fc10de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1101032c0 0x7fc110198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48050/0 (socket says 172.21.15.40:48050) 2024-04-26T23:59:26.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.628+0000 7fc10de00640 1 -- 172.21.15.40:0/1903589924 learned_addr learned my addr 172.21.15.40:0/1903589924 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:26.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.629+0000 7fc10e800640 1 -- 172.21.15.40:0/1903589924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1101032c0 msgr2=0x7fc110198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:26.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.629+0000 7fc10e800640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1101032c0 0x7fc110198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.629+0000 7fc10e800640 1 -- 172.21.15.40:0/1903589924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc110105bf0 msgr2=0x7fc110198f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:26.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.629+0000 7fc10e800640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc110105bf0 0x7fc110198f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.629+0000 7fc10e800640 1 -- 172.21.15.40:0/1903589924 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1101a0630 con 0x7fc110108520 2024-04-26T23:59:26.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.629+0000 7fc10de00640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1101032c0 0x7fc110198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:26.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.629+0000 7fc10e800640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc110108520 0x7fc11019ffc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc108009230 tx=0x7fc1080042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:26.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.629+0000 7fc106a00640 1 -- 172.21.15.40:0/1903589924 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10803f030 con 0x7fc110108520 2024-04-26T23:59:26.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.629+0000 7fc106a00640 1 -- 172.21.15.40:0/1903589924 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc108031910 con 0x7fc110108520 2024-04-26T23:59:26.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.629+0000 7fc106a00640 1 -- 172.21.15.40:0/1903589924 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc108035070 con 0x7fc110108520 2024-04-26T23:59:26.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.629+0000 7fc10f200640 1 -- 172.21.15.40:0/1903589924 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1101a08c0 con 0x7fc110108520 2024-04-26T23:59:26.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.630+0000 7fc10f200640 1 -- 172.21.15.40:0/1903589924 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1101a0d60 con 0x7fc110108520 2024-04-26T23:59:26.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.631+0000 7fc10f200640 1 -- 172.21.15.40:0/1903589924 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1100fe7c0 con 0x7fc110108520 2024-04-26T23:59:26.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.633+0000 7fc106a00640 1 -- 172.21.15.40:0/1903589924 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc108042030 con 0x7fc110108520 2024-04-26T23:59:26.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.634+0000 7fc106a00640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc0f40a3c30 0x7fc0f40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.634+0000 7fc106a00640 1 -- 172.21.15.40:0/1903589924 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc1080f0a00 con 0x7fc110108520 2024-04-26T23:59:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.634+0000 7fc10de00640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc0f40a3c30 0x7fc0f40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:26.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.635+0000 7fc10de00640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc0f40a3c30 0x7fc0f40a60f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8011550 tx=0x7fc0f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:26.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.638+0000 7fc106a00640 1 -- 172.21.15.40:0/1903589924 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc1080bcaf0 con 0x7fc110108520 2024-04-26T23:59:26.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:26 smithi040 ceph-mon[28095]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T23:59:26.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:26 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T23:59:26.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:26 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] up:active 2024-04-26T23:59:26.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:26 smithi040 ceph-mon[28095]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-04-26T23:59:26.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:26 smithi040 ceph-mon[28095]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T23:59:26.800 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 26 23:59:26 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b[63150]: 2024-04-26T23:59:26.554+0000 7f1faee6c700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T23:59:26.800 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 26 23:59:26 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b[63150]: 2024-04-26T23:59:26.554+0000 7f1faee6c700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:59:26.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:26 smithi080 ceph-mon[32362]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T23:59:26.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:26 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T23:59:26.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:26 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] up:active 2024-04-26T23:59:26.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:26 smithi080 ceph-mon[32362]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-04-26T23:59:26.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:26 smithi080 ceph-mon[32362]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T23:59:26.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:26 smithi103 ceph-mon[32952]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T23:59:26.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:26 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T23:59:26.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:26 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] up:active 2024-04-26T23:59:26.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:26 smithi103 ceph-mon[32952]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-04-26T23:59:26.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:26 smithi103 ceph-mon[32952]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T23:59:26.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.876+0000 7fc10f200640 1 -- 172.21.15.40:0/1903589924 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fc110062e00 con 0x7fc0f40a3c30 2024-04-26T23:59:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.888+0000 7fc106a00640 1 -- 172.21.15.40:0/1903589924 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fc110062e00 con 0x7fc0f40a3c30 2024-04-26T23:59:26.890 INFO:teuthology.orchestra.run.smithi040.stdout:/volumes/qa/sv_0/acce144d-825b-4b70-b840-60eac89d22ee 2024-04-26T23:59:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.890+0000 7fc10f200640 1 -- 172.21.15.40:0/1903589924 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc0f40a3c30 msgr2=0x7fc0f40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.890+0000 7fc10f200640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc0f40a3c30 0x7fc0f40a60f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8011550 tx=0x7fc0f8005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.890+0000 7fc10f200640 1 -- 172.21.15.40:0/1903589924 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc110108520 msgr2=0x7fc11019ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.890+0000 7fc10f200640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc110108520 0x7fc11019ffc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc108009230 tx=0x7fc1080042f0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.891+0000 7fc10f200640 1 -- 172.21.15.40:0/1903589924 shutdown_connections 2024-04-26T23:59:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.891+0000 7fc10f200640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc110108520 0x7fc11019ffc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.891+0000 7fc10f200640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc110105bf0 0x7fc110198f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.891+0000 7fc10f200640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc0f40a3c30 0x7fc0f40a60f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.891+0000 7fc10f200640 1 --2- 172.21.15.40:0/1903589924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1101032c0 0x7fc110198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.891+0000 7fc10f200640 1 -- 172.21.15.40:0/1903589924 >> 172.21.15.40:0/1903589924 conn(0x7fc1100fccd0 msgr2=0x7fc110103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:26.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.891+0000 7fc10f200640 1 -- 172.21.15.40:0/1903589924 shutdown_connections 2024-04-26T23:59:26.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:26.891+0000 7fc10f200640 1 -- 172.21.15.40:0/1903589924 wait complete. 2024-04-26T23:59:26.913 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-26T23:59:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.137+0000 7f15d1600640 1 -- 172.21.15.40:0/4180914555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15cc108e70 msgr2=0x7f15cc109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.137+0000 7f15d1600640 1 --2- 172.21.15.40:0/4180914555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15cc108e70 0x7f15cc109290 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f15bc009860 tx=0x7f15bc02a340 comp rx=0 tx=0).stop 2024-04-26T23:59:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.138+0000 7f15d1600640 1 -- 172.21.15.40:0/4180914555 shutdown_connections 2024-04-26T23:59:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.138+0000 7f15d1600640 1 --2- 172.21.15.40:0/4180914555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15cc10a980 0x7f15cc1112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.138+0000 7f15d1600640 1 --2- 172.21.15.40:0/4180914555 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f15cc109fc0 0x7f15cc10a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.138+0000 7f15d1600640 1 --2- 172.21.15.40:0/4180914555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15cc108e70 0x7f15cc109290 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.138+0000 7f15d1600640 1 -- 172.21.15.40:0/4180914555 >> 172.21.15.40:0/4180914555 conn(0x7f15cc077e00 msgr2=0x7f15cc07a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.138+0000 7f15d1600640 1 -- 172.21.15.40:0/4180914555 shutdown_connections 2024-04-26T23:59:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.138+0000 7f15d1600640 1 -- 172.21.15.40:0/4180914555 wait complete. 2024-04-26T23:59:27.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.139+0000 7f15d1600640 1 Processor -- start 2024-04-26T23:59:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.139+0000 7f15d1600640 1 -- start start 2024-04-26T23:59:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.140+0000 7f15d1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15cc108e70 0x7f15cc19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.140+0000 7f15d1600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15cc109fc0 0x7f15cc19f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.140+0000 7f15d1600640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f15cc10a980 0x7f15cc1a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.140+0000 7f15d1600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc1149a0 con 0x7f15cc108e70 2024-04-26T23:59:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.140+0000 7f15d1600640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc114820 con 0x7f15cc10a980 2024-04-26T23:59:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.140+0000 7f15d1600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc114b20 con 0x7f15cc109fc0 2024-04-26T23:59:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.140+0000 7f15cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15cc108e70 0x7f15cc19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.140+0000 7f15cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15cc108e70 0x7f15cc19f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49494/0 (socket says 172.21.15.40:49494) 2024-04-26T23:59:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.140+0000 7f15cbe00640 1 -- 172.21.15.40:0/2338653562 learned_addr learned my addr 172.21.15.40:0/2338653562 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.140+0000 7f15d0c00640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f15cc10a980 0x7f15cc1a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.140+0000 7f15cb400640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15cc109fc0 0x7f15cc19f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:27.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.140+0000 7f15cbe00640 1 -- 172.21.15.40:0/2338653562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15cc109fc0 msgr2=0x7f15cc19f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.141+0000 7f15cbe00640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15cc109fc0 0x7f15cc19f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.141+0000 7f15cbe00640 1 -- 172.21.15.40:0/2338653562 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f15cc10a980 msgr2=0x7f15cc1a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.141+0000 7f15cbe00640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f15cc10a980 0x7f15cc1a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.141+0000 7f15cbe00640 1 -- 172.21.15.40:0/2338653562 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15cc1a6c80 con 0x7f15cc108e70 2024-04-26T23:59:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.141+0000 7f15d0c00640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f15cc10a980 0x7f15cc1a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.141+0000 7f15cbe00640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15cc108e70 0x7f15cc19f060 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f15bc0057c0 tx=0x7f15bc032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.141+0000 7f15c8c00640 1 -- 172.21.15.40:0/2338653562 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc037040 con 0x7f15cc108e70 2024-04-26T23:59:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.141+0000 7f15c8c00640 1 -- 172.21.15.40:0/2338653562 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15bc030cd0 con 0x7f15cc108e70 2024-04-26T23:59:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.141+0000 7f15d1600640 1 -- 172.21.15.40:0/2338653562 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15cc1a6f10 con 0x7f15cc108e70 2024-04-26T23:59:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.141+0000 7f15c8c00640 1 -- 172.21.15.40:0/2338653562 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc005350 con 0x7f15cc108e70 2024-04-26T23:59:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.142+0000 7f15d1600640 1 -- 172.21.15.40:0/2338653562 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15cc1a72c0 con 0x7f15cc108e70 2024-04-26T23:59:27.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.143+0000 7f15d1600640 1 -- 172.21.15.40:0/2338653562 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15cc110250 con 0x7f15cc108e70 2024-04-26T23:59:27.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.146+0000 7f15c8c00640 1 -- 172.21.15.40:0/2338653562 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f15bc043020 con 0x7f15cc108e70 2024-04-26T23:59:27.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.147+0000 7f15c8c00640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f15b00a3bb0 0x7f15b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:27.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.147+0000 7f15c8c00640 1 -- 172.21.15.40:0/2338653562 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f15bc0efdc0 con 0x7f15cc108e70 2024-04-26T23:59:27.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.147+0000 7f15cb400640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f15b00a3bb0 0x7f15b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:27.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.148+0000 7f15cb400640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f15b00a3bb0 0x7f15b00a6070 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f15cc1a0570 tx=0x7f15b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:27.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.153+0000 7f15c8c00640 1 -- 172.21.15.40:0/2338653562 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f15bc0bc500 con 0x7f15cc108e70 2024-04-26T23:59:27.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.403+0000 7f15c8c00640 1 -- 172.21.15.40:0/2338653562 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f15bc0b8470 con 0x7f15cc108e70 2024-04-26T23:59:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.410+0000 7f15d1600640 1 -- 172.21.15.40:0/2338653562 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f15cc062e00 con 0x7f15b00a3bb0 2024-04-26T23:59:27.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.444+0000 7f15c8c00640 1 -- 172.21.15.40:0/2338653562 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f15cc062e00 con 0x7f15b00a3bb0 2024-04-26T23:59:27.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.447+0000 7f15d1600640 1 -- 172.21.15.40:0/2338653562 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f15b00a3bb0 msgr2=0x7f15b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:27.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.447+0000 7f15d1600640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f15b00a3bb0 0x7f15b00a6070 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f15cc1a0570 tx=0x7f15b8009260 comp rx=0 tx=0).stop 2024-04-26T23:59:27.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.448+0000 7f15d1600640 1 -- 172.21.15.40:0/2338653562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15cc108e70 msgr2=0x7f15cc19f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:27.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.448+0000 7f15d1600640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15cc108e70 0x7f15cc19f060 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f15bc0057c0 tx=0x7f15bc032eb0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.448+0000 7f15d1600640 1 -- 172.21.15.40:0/2338653562 shutdown_connections 2024-04-26T23:59:27.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.448+0000 7f15d1600640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f15cc10a980 0x7f15cc1a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.448+0000 7f15d1600640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f15b00a3bb0 0x7f15b00a6070 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.448+0000 7f15d1600640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15cc109fc0 0x7f15cc19f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.448+0000 7f15d1600640 1 --2- 172.21.15.40:0/2338653562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15cc108e70 0x7f15cc19f060 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.448+0000 7f15d1600640 1 -- 172.21.15.40:0/2338653562 >> 172.21.15.40:0/2338653562 conn(0x7f15cc077e00 msgr2=0x7f15cc10d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:27.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.448+0000 7f15d1600640 1 -- 172.21.15.40:0/2338653562 shutdown_connections 2024-04-26T23:59:27.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.448+0000 7f15d1600640 1 -- 172.21.15.40:0/2338653562 wait complete. 2024-04-26T23:59:27.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-26T23:59:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.675+0000 7fa82b200640 1 -- 172.21.15.40:0/546554576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa82c1010f0 msgr2=0x7fa82c101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.675+0000 7fa82b200640 1 --2- 172.21.15.40:0/546554576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa82c1010f0 0x7fa82c101510 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009900 tx=0x7fa81c02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.676+0000 7fa82b200640 1 -- 172.21.15.40:0/546554576 shutdown_connections 2024-04-26T23:59:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.676+0000 7fa82b200640 1 --2- 172.21.15.40:0/546554576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa82c1023f0 0x7fa82c102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.676+0000 7fa82b200640 1 --2- 172.21.15.40:0/546554576 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa82c101a50 0x7fa82c101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.676+0000 7fa82b200640 1 --2- 172.21.15.40:0/546554576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa82c1010f0 0x7fa82c101510 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.676+0000 7fa82b200640 1 -- 172.21.15.40:0/546554576 >> 172.21.15.40:0/546554576 conn(0x7fa82c0fccd0 msgr2=0x7fa82c0ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.676+0000 7fa82b200640 1 -- 172.21.15.40:0/546554576 shutdown_connections 2024-04-26T23:59:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.676+0000 7fa82b200640 1 -- 172.21.15.40:0/546554576 wait complete. 2024-04-26T23:59:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.677+0000 7fa82b200640 1 Processor -- start 2024-04-26T23:59:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.677+0000 7fa82b200640 1 -- start start 2024-04-26T23:59:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa82b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa82c1010f0 0x7fa82c19aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa82b200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa82c101a50 0x7fa82c19b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa82b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa82c1023f0 0x7fa82c1a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa82b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c1104e0 con 0x7fa82c1023f0 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa82b200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c110360 con 0x7fa82c101a50 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa82b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c110660 con 0x7fa82c1010f0 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa829e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa82c1010f0 0x7fa82c19aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa829e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa82c1010f0 0x7fa82c19aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48080/0 (socket says 172.21.15.40:48080) 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa829e00640 1 -- 172.21.15.40:0/3488641733 learned_addr learned my addr 172.21.15.40:0/3488641733 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa82a800640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa82c1023f0 0x7fa82c1a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa829400640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa82c101a50 0x7fa82c19b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa829e00640 1 -- 172.21.15.40:0/3488641733 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa82c101a50 msgr2=0x7fa82c19b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa829e00640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa82c101a50 0x7fa82c19b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.678+0000 7fa829e00640 1 -- 172.21.15.40:0/3488641733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa82c1023f0 msgr2=0x7fa82c1a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.679+0000 7fa829e00640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa82c1023f0 0x7fa82c1a2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.679+0000 7fa829e00640 1 -- 172.21.15.40:0/3488641733 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82c1a28e0 con 0x7fa82c1010f0 2024-04-26T23:59:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.679+0000 7fa829e00640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa82c1010f0 0x7fa82c19aca0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa81c005940 tx=0x7fa81c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.679+0000 7fa822a00640 1 -- 172.21.15.40:0/3488641733 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c03f030 con 0x7fa82c1010f0 2024-04-26T23:59:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.679+0000 7fa822a00640 1 -- 172.21.15.40:0/3488641733 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa81c003cb0 con 0x7fa82c1010f0 2024-04-26T23:59:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.679+0000 7fa822a00640 1 -- 172.21.15.40:0/3488641733 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c035070 con 0x7fa82c1010f0 2024-04-26T23:59:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.679+0000 7fa82b200640 1 -- 172.21.15.40:0/3488641733 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82c1a2b70 con 0x7fa82c1010f0 2024-04-26T23:59:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.679+0000 7fa82b200640 1 -- 172.21.15.40:0/3488641733 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa82c1a3080 con 0x7fa82c1010f0 2024-04-26T23:59:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.680+0000 7fa82b200640 1 -- 172.21.15.40:0/3488641733 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa82c10be40 con 0x7fa82c1010f0 2024-04-26T23:59:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.683+0000 7fa822a00640 1 -- 172.21.15.40:0/3488641733 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa81c042030 con 0x7fa82c1010f0 2024-04-26T23:59:27.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.684+0000 7fa822a00640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa8100a3cc0 0x7fa8100a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:27.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.684+0000 7fa822a00640 1 -- 172.21.15.40:0/3488641733 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa81c0f0fe0 con 0x7fa82c1010f0 2024-04-26T23:59:27.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.684+0000 7fa829400640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa8100a3cc0 0x7fa8100a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:27.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.685+0000 7fa829400640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa8100a3cc0 0x7fa8100a6180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa82c19c1a0 tx=0x7fa818005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:27.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.687+0000 7fa822a00640 1 -- 172.21.15.40:0/3488641733 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa81c0bd600 con 0x7fa82c1010f0 2024-04-26T23:59:27.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:27 smithi040 ceph-mon[28095]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:27.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:27 smithi040 ceph-mon[28095]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:27.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:27 smithi080 ceph-mon[32362]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:27.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:27 smithi080 ceph-mon[32362]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:27.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:27 smithi103 ceph-mon[32952]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:27.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:27 smithi103 ceph-mon[32952]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:27.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.926+0000 7fa82b200640 1 -- 172.21.15.40:0/3488641733 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fa82c062e00 con 0x7fa8100a3cc0 2024-04-26T23:59:27.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.933+0000 7fa822a00640 1 -- 172.21.15.40:0/3488641733 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fa82c062e00 con 0x7fa8100a3cc0 2024-04-26T23:59:27.935 INFO:teuthology.orchestra.run.smithi040.stdout:/volumes/qa/sv_1/44d4f898-2f89-4f34-8673-5d8bb96f7f18 2024-04-26T23:59:27.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.936+0000 7fa82b200640 1 -- 172.21.15.40:0/3488641733 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa8100a3cc0 msgr2=0x7fa8100a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.936+0000 7fa82b200640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa8100a3cc0 0x7fa8100a6180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa82c19c1a0 tx=0x7fa818005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.936+0000 7fa82b200640 1 -- 172.21.15.40:0/3488641733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa82c1010f0 msgr2=0x7fa82c19aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.936+0000 7fa82b200640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa82c1010f0 0x7fa82c19aca0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa81c005940 tx=0x7fa81c0042f0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.936+0000 7fa82b200640 1 -- 172.21.15.40:0/3488641733 shutdown_connections 2024-04-26T23:59:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.936+0000 7fa82b200640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa82c1023f0 0x7fa82c1a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.936+0000 7fa82b200640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa82c101a50 0x7fa82c19b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.936+0000 7fa82b200640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa8100a3cc0 0x7fa8100a6180 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.936+0000 7fa82b200640 1 --2- 172.21.15.40:0/3488641733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa82c1010f0 0x7fa82c19aca0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.936+0000 7fa82b200640 1 -- 172.21.15.40:0/3488641733 >> 172.21.15.40:0/3488641733 conn(0x7fa82c0fccd0 msgr2=0x7fa82c100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.937+0000 7fa82b200640 1 -- 172.21.15.40:0/3488641733 shutdown_connections 2024-04-26T23:59:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:27.937+0000 7fa82b200640 1 -- 172.21.15.40:0/3488641733 wait complete. 2024-04-26T23:59:27.958 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:59:28.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.173+0000 7f1a9b000640 1 -- 172.21.15.40:0/1728809988 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1a94104360 msgr2=0x7f1a9410abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:28.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.173+0000 7f1a9b000640 1 --2- 172.21.15.40:0/1728809988 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1a94104360 0x7f1a9410abf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009900 tx=0x7f1a8c02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.173+0000 7f1a9b000640 1 -- 172.21.15.40:0/1728809988 shutdown_connections 2024-04-26T23:59:28.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.173+0000 7f1a9b000640 1 --2- 172.21.15.40:0/1728809988 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1a94104360 0x7f1a9410abf0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.173+0000 7f1a9b000640 1 --2- 172.21.15.40:0/1728809988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a941039a0 0x7f1a94103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.173+0000 7f1a9b000640 1 --2- 172.21.15.40:0/1728809988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a941027b0 0x7f1a94102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.173+0000 7f1a9b000640 1 -- 172.21.15.40:0/1728809988 >> 172.21.15.40:0/1728809988 conn(0x7f1a940fdd40 msgr2=0x7f1a94100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:28.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.173+0000 7f1a9b000640 1 -- 172.21.15.40:0/1728809988 shutdown_connections 2024-04-26T23:59:28.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.173+0000 7f1a9b000640 1 -- 172.21.15.40:0/1728809988 wait complete. 2024-04-26T23:59:28.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.174+0000 7f1a9b000640 1 Processor -- start 2024-04-26T23:59:28.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.174+0000 7f1a9b000640 1 -- start start 2024-04-26T23:59:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.175+0000 7f1a9b000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1a941027b0 0x7f1a9419ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.175+0000 7f1a9b000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a941039a0 0x7f1a9419b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.175+0000 7f1a9b000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a94104360 0x7f1a941a2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.175+0000 7f1a9b000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9410e2e0 con 0x7f1a941039a0 2024-04-26T23:59:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.175+0000 7f1a9b000640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9410e160 con 0x7f1a941027b0 2024-04-26T23:59:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.175+0000 7f1a9b000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9410e460 con 0x7f1a94104360 2024-04-26T23:59:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.175+0000 7f1a99200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a941039a0 0x7f1a9419b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.175+0000 7f1a99200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a941039a0 0x7f1a9419b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49520/0 (socket says 172.21.15.40:49520) 2024-04-26T23:59:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.175+0000 7f1a99200640 1 -- 172.21.15.40:0/946349469 learned_addr learned my addr 172.21.15.40:0/946349469 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.175+0000 7f1a9a600640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a94104360 0x7f1a941a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.175+0000 7f1a99c00640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1a941027b0 0x7f1a9419ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.176+0000 7f1a9a600640 1 -- 172.21.15.40:0/946349469 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1a941027b0 msgr2=0x7f1a9419ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:28.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.176+0000 7f1a9a600640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1a941027b0 0x7f1a9419ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.176+0000 7f1a9a600640 1 -- 172.21.15.40:0/946349469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a941039a0 msgr2=0x7f1a9419b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:28.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.176+0000 7f1a9a600640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a941039a0 0x7f1a9419b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.176+0000 7f1a9a600640 1 -- 172.21.15.40:0/946349469 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a941a2770 con 0x7f1a94104360 2024-04-26T23:59:28.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.176+0000 7f1a99200640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a941039a0 0x7f1a9419b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:28.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.176+0000 7f1a9a600640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a94104360 0x7f1a941a2100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009230 tx=0x7f1a8c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:28.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.176+0000 7f1a92000640 1 -- 172.21.15.40:0/946349469 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c03f030 con 0x7f1a94104360 2024-04-26T23:59:28.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.176+0000 7f1a92000640 1 -- 172.21.15.40:0/946349469 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a8c031910 con 0x7f1a94104360 2024-04-26T23:59:28.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.176+0000 7f1a9b000640 1 -- 172.21.15.40:0/946349469 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a941a2a00 con 0x7f1a94104360 2024-04-26T23:59:28.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.176+0000 7f1a92000640 1 -- 172.21.15.40:0/946349469 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c035070 con 0x7f1a94104360 2024-04-26T23:59:28.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.177+0000 7f1a9b000640 1 -- 172.21.15.40:0/946349469 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a941a2ef0 con 0x7f1a94104360 2024-04-26T23:59:28.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.178+0000 7f1a9b000640 1 -- 172.21.15.40:0/946349469 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a941a3120 con 0x7f1a94104360 2024-04-26T23:59:28.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.180+0000 7f1a92000640 1 -- 172.21.15.40:0/946349469 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1a8c042030 con 0x7f1a94104360 2024-04-26T23:59:28.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.181+0000 7f1a92000640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1a780a3c40 0x7f1a780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:28.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.181+0000 7f1a92000640 1 -- 172.21.15.40:0/946349469 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f1a8c0f0be0 con 0x7f1a94104360 2024-04-26T23:59:28.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.181+0000 7f1a99c00640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1a780a3c40 0x7f1a780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:28.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.182+0000 7f1a99c00640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1a780a3c40 0x7f1a780a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1a94103800 tx=0x7f1a84009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:28.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.185+0000 7f1a92000640 1 -- 172.21.15.40:0/946349469 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1a8c0bcb70 con 0x7f1a94104360 2024-04-26T23:59:28.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.435+0000 7f1a9b000640 1 -- 172.21.15.40:0/946349469 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a941a34b0 con 0x7f1a94104360 2024-04-26T23:59:28.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.437+0000 7f1a92000640 1 -- 172.21.15.40:0/946349469 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11872 (secure 0 0 0) 0x7f1a8c0c0e70 con 0x7f1a94104360 2024-04-26T23:59:28.439 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:59:28.439 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/2265538751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":2265538751},{"type":"v1","addr":"172.21.15.40:6841","nonce":2265538751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:58:50.616677+0000","modified":"2024-04-26T23:59:25.374905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24523":{"gid":24523,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.80:6839/3536638444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6838","nonce":3536638444},{"type":"v1","addr":"172.21.15.80:6839","nonce":3536638444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24559,24560,24590]},"id":1}]} 2024-04-26T23:59:28.439 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 23 2024-04-26T23:59:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.439+0000 7f1a9b000640 1 -- 172.21.15.40:0/946349469 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1a780a3c40 msgr2=0x7f1a780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.439+0000 7f1a9b000640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1a780a3c40 0x7f1a780a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1a94103800 tx=0x7f1a84009500 comp rx=0 tx=0).stop 2024-04-26T23:59:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.439+0000 7f1a9b000640 1 -- 172.21.15.40:0/946349469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a94104360 msgr2=0x7f1a941a2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.439+0000 7f1a9b000640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a94104360 0x7f1a941a2100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009230 tx=0x7f1a8c0042f0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.440+0000 7f1a9b000640 1 -- 172.21.15.40:0/946349469 shutdown_connections 2024-04-26T23:59:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.440+0000 7f1a9b000640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1a780a3c40 0x7f1a780a6100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.440+0000 7f1a9b000640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a94104360 0x7f1a941a2100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.440+0000 7f1a9b000640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a941039a0 0x7f1a9419b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.440+0000 7f1a9b000640 1 --2- 172.21.15.40:0/946349469 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1a941027b0 0x7f1a9419ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.440+0000 7f1a9b000640 1 -- 172.21.15.40:0/946349469 >> 172.21.15.40:0/946349469 conn(0x7f1a940fdd40 msgr2=0x7f1a94074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:28.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.440+0000 7f1a9b000640 1 -- 172.21.15.40:0/946349469 shutdown_connections 2024-04-26T23:59:28.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.440+0000 7f1a9b000640 1 -- 172.21.15.40:0/946349469 wait complete. 2024-04-26T23:59:28.462 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:59:28.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.660+0000 7f2cf2200640 1 -- 172.21.15.40:0/3801442146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec105b70 msgr2=0x7f2cec105ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:28.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.660+0000 7f2cf2200640 1 --2- 172.21.15.40:0/3801442146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec105b70 0x7f2cec105ff0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009900 tx=0x7f2cd802a3f0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.660+0000 7f2cf2200640 1 -- 172.21.15.40:0/3801442146 shutdown_connections 2024-04-26T23:59:28.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.660+0000 7f2cf2200640 1 --2- 172.21.15.40:0/3801442146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cec106530 0x7f2cec10cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.660+0000 7f2cf2200640 1 --2- 172.21.15.40:0/3801442146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec105b70 0x7f2cec105ff0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.660+0000 7f2cf2200640 1 --2- 172.21.15.40:0/3801442146 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cec104980 0x7f2cec104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.660+0000 7f2cf2200640 1 -- 172.21.15.40:0/3801442146 >> 172.21.15.40:0/3801442146 conn(0x7f2cec0fff10 msgr2=0x7f2cec102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:28.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.660+0000 7f2cf2200640 1 -- 172.21.15.40:0/3801442146 shutdown_connections 2024-04-26T23:59:28.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.660+0000 7f2cf2200640 1 -- 172.21.15.40:0/3801442146 wait complete. 2024-04-26T23:59:28.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.661+0000 7f2cf2200640 1 Processor -- start 2024-04-26T23:59:28.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.661+0000 7f2cf2200640 1 -- start start 2024-04-26T23:59:28.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.662+0000 7f2cf2200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cec104980 0x7f2cec19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.662+0000 7f2cf2200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cec105b70 0x7f2cec19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.662+0000 7f2cf2200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec106530 0x7f2cec1a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.662+0000 7f2cf2200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec1104e0 con 0x7f2cec105b70 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.662+0000 7f2cf2200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec110360 con 0x7f2cec104980 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.662+0000 7f2cf2200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec110660 con 0x7f2cec106530 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.662+0000 7f2cebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cec105b70 0x7f2cec19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.662+0000 7f2cf0e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cec104980 0x7f2cec19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.662+0000 7f2cebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cec105b70 0x7f2cec19b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49528/0 (socket says 172.21.15.40:49528) 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.662+0000 7f2cebe00640 1 -- 172.21.15.40:0/4108997797 learned_addr learned my addr 172.21.15.40:0/4108997797 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.662+0000 7f2cf1800640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec106530 0x7f2cec1a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.663+0000 7f2cf0e00640 1 -- 172.21.15.40:0/4108997797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec106530 msgr2=0x7f2cec1a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.663+0000 7f2cf0e00640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec106530 0x7f2cec1a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.663+0000 7f2cf0e00640 1 -- 172.21.15.40:0/4108997797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cec105b70 msgr2=0x7f2cec19b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.663+0000 7f2cf0e00640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cec105b70 0x7f2cec19b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.663+0000 7f2cf0e00640 1 -- 172.21.15.40:0/4108997797 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cec1a2780 con 0x7f2cec104980 2024-04-26T23:59:28.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.663+0000 7f2cf0e00640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cec104980 0x7f2cec19ab60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000e9c0 tx=0x7f2ce000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:28.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.663+0000 7f2ce9600640 1 -- 172.21.15.40:0/4108997797 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce000ccc0 con 0x7f2cec104980 2024-04-26T23:59:28.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.663+0000 7f2cf2200640 1 -- 172.21.15.40:0/4108997797 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cec1a2a70 con 0x7f2cec104980 2024-04-26T23:59:28.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.663+0000 7f2ce9600640 1 -- 172.21.15.40:0/4108997797 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ce00035f0 con 0x7f2cec104980 2024-04-26T23:59:28.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.663+0000 7f2ce9600640 1 -- 172.21.15.40:0/4108997797 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce000ce60 con 0x7f2cec104980 2024-04-26T23:59:28.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.664+0000 7f2cf2200640 1 -- 172.21.15.40:0/4108997797 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cec1a3010 con 0x7f2cec104980 2024-04-26T23:59:28.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.665+0000 7f2cf2200640 1 -- 172.21.15.40:0/4108997797 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cec10bca0 con 0x7f2cec104980 2024-04-26T23:59:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.667+0000 7f2ce9600640 1 -- 172.21.15.40:0/4108997797 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2ce0004000 con 0x7f2cec104980 2024-04-26T23:59:28.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.668+0000 7f2ce9600640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2cd40a3c40 0x7f2cd40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:28.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.668+0000 7f2ce9600640 1 -- 172.21.15.40:0/4108997797 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f2ce00d0c40 con 0x7f2cec104980 2024-04-26T23:59:28.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.668+0000 7f2cebe00640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2cd40a3c40 0x7f2cd40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.669+0000 7f2cebe00640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2cd40a3c40 0x7f2cd40a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2cec19c070 tx=0x7f2cd8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:28.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.673+0000 7f2ce9600640 1 -- 172.21.15.40:0/4108997797 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2ce009d220 con 0x7f2cec104980 2024-04-26T23:59:28.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:28 smithi040 ceph-mon[28095]: pgmap v341: 129 pgs: 129 active+clean; 591 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-26T23:59:28.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:28 smithi040 ceph-mon[28095]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:28.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:28 smithi040 ceph-mon[28095]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T23:59:28.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:28 smithi040 ceph-mon[28095]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:28.749 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:28 smithi103 ceph-mon[32952]: pgmap v341: 129 pgs: 129 active+clean; 591 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-26T23:59:28.749 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:28 smithi103 ceph-mon[32952]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:28.749 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:28 smithi103 ceph-mon[32952]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T23:59:28.749 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:28 smithi103 ceph-mon[32952]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:28.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:28 smithi080 ceph-mon[32362]: pgmap v341: 129 pgs: 129 active+clean; 591 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-26T23:59:28.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:28 smithi080 ceph-mon[32362]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:28.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:28 smithi080 ceph-mon[32362]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T23:59:28.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:28 smithi080 ceph-mon[32362]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:28.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.880+0000 7f2cf2200640 1 -- 172.21.15.40:0/4108997797 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2cec02dd30 con 0x7f2cec104980 2024-04-26T23:59:28.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.882+0000 7f2ce9600640 1 -- 172.21.15.40:0/4108997797 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24090 (secure 0 0 0) 0x7f2ce009cb20 con 0x7f2cec104980 2024-04-26T23:59:28.884 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:59:28.884 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:58:55.649955+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:59:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.885+0000 7f2cf2200640 1 -- 172.21.15.40:0/4108997797 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2cd40a3c40 msgr2=0x7f2cd40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.885+0000 7f2cf2200640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2cd40a3c40 0x7f2cd40a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2cec19c070 tx=0x7f2cd8005a20 comp rx=0 tx=0).stop 2024-04-26T23:59:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.885+0000 7f2cf2200640 1 -- 172.21.15.40:0/4108997797 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cec104980 msgr2=0x7f2cec19ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.885+0000 7f2cf2200640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cec104980 0x7f2cec19ab60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000e9c0 tx=0x7f2ce000ee80 comp rx=0 tx=0).stop 2024-04-26T23:59:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.885+0000 7f2cf2200640 1 -- 172.21.15.40:0/4108997797 shutdown_connections 2024-04-26T23:59:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.885+0000 7f2cf2200640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2cd40a3c40 0x7f2cd40a6100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.885+0000 7f2cf2200640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec106530 0x7f2cec1a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.885+0000 7f2cf2200640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cec105b70 0x7f2cec19b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.885+0000 7f2cf2200640 1 --2- 172.21.15.40:0/4108997797 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2cec104980 0x7f2cec19ab60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.885+0000 7f2cf2200640 1 -- 172.21.15.40:0/4108997797 >> 172.21.15.40:0/4108997797 conn(0x7f2cec0fff10 msgr2=0x7f2cec108e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:28.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.885+0000 7f2cf2200640 1 -- 172.21.15.40:0/4108997797 shutdown_connections 2024-04-26T23:59:28.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:28.885+0000 7f2cf2200640 1 -- 172.21.15.40:0/4108997797 wait complete. 2024-04-26T23:59:28.906 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T23:59:28.906 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:59:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.107+0000 7f1da9c00640 1 -- 172.21.15.40:0/846735735 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1da4105b20 msgr2=0x7f1da4107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.107+0000 7f1da9c00640 1 --2- 172.21.15.40:0/846735735 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1da4105b20 0x7f1da4107f10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c009ca0 tx=0x7f1d8c02a8c0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.108+0000 7f1da9c00640 1 -- 172.21.15.40:0/846735735 shutdown_connections 2024-04-26T23:59:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.108+0000 7f1da9c00640 1 --2- 172.21.15.40:0/846735735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da4108450 0x7f1da410a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.108+0000 7f1da9c00640 1 --2- 172.21.15.40:0/846735735 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1da4105b20 0x7f1da4107f10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.108+0000 7f1da9c00640 1 --2- 172.21.15.40:0/846735735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da41031f0 0x7f1da41055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.108+0000 7f1da9c00640 1 -- 172.21.15.40:0/846735735 >> 172.21.15.40:0/846735735 conn(0x7f1da40fcbc0 msgr2=0x7f1da40ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.108+0000 7f1da9c00640 1 -- 172.21.15.40:0/846735735 shutdown_connections 2024-04-26T23:59:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.108+0000 7f1da9c00640 1 -- 172.21.15.40:0/846735735 wait complete. 2024-04-26T23:59:29.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.109+0000 7f1da9c00640 1 Processor -- start 2024-04-26T23:59:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.109+0000 7f1da9c00640 1 -- start start 2024-04-26T23:59:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.110+0000 7f1da9c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da41031f0 0x7f1da419abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.110+0000 7f1da9c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1da4105b20 0x7f1da419b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.110+0000 7f1da9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da4108450 0x7f1da41a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.110+0000 7f1da9c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1da410e2e0 con 0x7f1da4108450 2024-04-26T23:59:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.110+0000 7f1da9c00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1da410e160 con 0x7f1da4105b20 2024-04-26T23:59:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.110+0000 7f1da9c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1da410e460 con 0x7f1da41031f0 2024-04-26T23:59:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.110+0000 7f1da9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da4108450 0x7f1da41a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.110+0000 7f1da3400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1da4105b20 0x7f1da419b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.110+0000 7f1da3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da41031f0 0x7f1da419abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.110+0000 7f1da9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da4108450 0x7f1da41a2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49532/0 (socket says 172.21.15.40:49532) 2024-04-26T23:59:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.110+0000 7f1da9200640 1 -- 172.21.15.40:0/698128345 learned_addr learned my addr 172.21.15.40:0/698128345 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:29.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.111+0000 7f1da9200640 1 -- 172.21.15.40:0/698128345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da41031f0 msgr2=0x7f1da419abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.111+0000 7f1da9200640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da41031f0 0x7f1da419abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.111+0000 7f1da9200640 1 -- 172.21.15.40:0/698128345 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1da4105b20 msgr2=0x7f1da419b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.111+0000 7f1da9200640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1da4105b20 0x7f1da419b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.111+0000 7f1da9200640 1 -- 172.21.15.40:0/698128345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da41a2800 con 0x7f1da4108450 2024-04-26T23:59:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.111+0000 7f1da3e00640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da41031f0 0x7f1da419abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.111+0000 7f1da9200640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da4108450 0x7f1da41a2190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00e970 tx=0x7f1d9c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.111+0000 7f1da0c00640 1 -- 172.21.15.40:0/698128345 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c00ccc0 con 0x7f1da4108450 2024-04-26T23:59:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.111+0000 7f1da0c00640 1 -- 172.21.15.40:0/698128345 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d9c00ce60 con 0x7f1da4108450 2024-04-26T23:59:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.111+0000 7f1da9c00640 1 -- 172.21.15.40:0/698128345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da41a2af0 con 0x7f1da4108450 2024-04-26T23:59:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.111+0000 7f1da0c00640 1 -- 172.21.15.40:0/698128345 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c010430 con 0x7f1da4108450 2024-04-26T23:59:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.112+0000 7f1da9c00640 1 -- 172.21.15.40:0/698128345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da41a2fc0 con 0x7f1da4108450 2024-04-26T23:59:29.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.113+0000 7f1da9c00640 1 -- 172.21.15.40:0/698128345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da4109fa0 con 0x7f1da4108450 2024-04-26T23:59:29.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.115+0000 7f1da0c00640 1 -- 172.21.15.40:0/698128345 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1d9c004ff0 con 0x7f1da4108450 2024-04-26T23:59:29.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.117+0000 7f1da0c00640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1d840a3c40 0x7f1d840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:29.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.117+0000 7f1da0c00640 1 -- 172.21.15.40:0/698128345 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f1d9c0d0bc0 con 0x7f1da4108450 2024-04-26T23:59:29.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.117+0000 7f1da3e00640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1d840a3c40 0x7f1d840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:29.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.118+0000 7f1da3e00640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1d840a3c40 0x7f1d840a6100 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009fc0 tx=0x7f1d94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.122+0000 7f1da0c00640 1 -- 172.21.15.40:0/698128345 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1d9c09d200 con 0x7f1da4108450 2024-04-26T23:59:29.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.341+0000 7f1da9c00640 1 -- 172.21.15.40:0/698128345 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1da4062d70 con 0x7f1d840a3c40 2024-04-26T23:59:29.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.379+0000 7f1da0c00640 1 -- 172.21.15.40:0/698128345 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340361 (secure 0 0 0) 0x7f1da4062d70 con 0x7f1d840a3c40 2024-04-26T23:59:29.386 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:59:29.388 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:59:29.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.386+0000 7f1da9c00640 1 -- 172.21.15.40:0/698128345 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1d840a3c40 msgr2=0x7f1d840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:29.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.387+0000 7f1da9c00640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1d840a3c40 0x7f1d840a6100 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009fc0 tx=0x7f1d94005f90 comp rx=0 tx=0).stop 2024-04-26T23:59:29.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.387+0000 7f1da9c00640 1 -- 172.21.15.40:0/698128345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da4108450 msgr2=0x7f1da41a2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:29.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.387+0000 7f1da9c00640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da4108450 0x7f1da41a2190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00e970 tx=0x7f1d9c00ed70 comp rx=0 tx=0).stop 2024-04-26T23:59:29.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.387+0000 7f1da9c00640 1 -- 172.21.15.40:0/698128345 shutdown_connections 2024-04-26T23:59:29.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.387+0000 7f1da9c00640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1da4108450 0x7f1da41a2190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.387+0000 7f1da9c00640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1da4105b20 0x7f1da419b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.387+0000 7f1da9c00640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1d840a3c40 0x7f1d840a6100 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.387+0000 7f1da9c00640 1 --2- 172.21.15.40:0/698128345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da41031f0 0x7f1da419abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.387+0000 7f1da9c00640 1 -- 172.21.15.40:0/698128345 >> 172.21.15.40:0/698128345 conn(0x7f1da40fcbc0 msgr2=0x7f1da4102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:29.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.387+0000 7f1da9c00640 1 -- 172.21.15.40:0/698128345 shutdown_connections 2024-04-26T23:59:29.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.387+0000 7f1da9c00640 1 -- 172.21.15.40:0/698128345 wait complete. 2024-04-26T23:59:29.413 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-04-26T23:59:28.712858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":604928,"num_objects":63,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":154,"num_read_kb":136,"num_write":316,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":256,"ondisk_log_size":256,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371120,"kb_used_data":13648,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570704,"statfs":{"total":1151940427776,"available":1151560400896,"internally_reserved":0,"allocated":13975552,"data_stored":7610588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9920,"num_objects":39,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":28,"num_read_kb":27,"num_write":86,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006745"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248269+0000","last_change":"2024-04-26T23:58:51.162452+0000","last_active":"2024-04-26T23:58:56.248269+0000","last_peered":"2024-04-26T23:58:56.248269+0000","last_clean":"2024-04-26T23:58:56.248269+0000","last_became_active":"2024-04-26T23:58:51.162065+0000","last_became_peered":"2024-04-26T23:58:51.162065+0000","last_unstale":"2024-04-26T23:58:56.248269+0000","last_undegraded":"2024-04-26T23:58:56.248269+0000","last_fullsized":"2024-04-26T23:58:56.248269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:58:05.949099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248836+0000","last_change":"2024-04-26T23:58:51.161144+0000","last_active":"2024-04-26T23:58:56.248836+0000","last_peered":"2024-04-26T23:58:56.248836+0000","last_clean":"2024-04-26T23:58:56.248836+0000","last_became_active":"2024-04-26T23:58:51.154033+0000","last_became_peered":"2024-04-26T23:58:51.154033+0000","last_unstale":"2024-04-26T23:58:56.248836+0000","last_undegraded":"2024-04-26T23:58:56.248836+0000","last_fullsized":"2024-04-26T23:58:56.248836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:37:39.757654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247575+0000","last_change":"2024-04-26T23:58:51.159715+0000","last_active":"2024-04-26T23:58:56.247575+0000","last_peered":"2024-04-26T23:58:56.247575+0000","last_clean":"2024-04-26T23:58:56.247575+0000","last_became_active":"2024-04-26T23:58:51.152657+0000","last_became_peered":"2024-04-26T23:58:51.152657+0000","last_unstale":"2024-04-26T23:58:56.247575+0000","last_undegraded":"2024-04-26T23:58:56.247575+0000","last_fullsized":"2024-04-26T23:58:56.247575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:29.210295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249104+0000","last_change":"2024-04-26T23:58:51.141814+0000","last_active":"2024-04-26T23:58:56.249104+0000","last_peered":"2024-04-26T23:58:56.249104+0000","last_clean":"2024-04-26T23:58:56.249104+0000","last_became_active":"2024-04-26T23:58:51.141344+0000","last_became_peered":"2024-04-26T23:58:51.141344+0000","last_unstale":"2024-04-26T23:58:56.249104+0000","last_undegraded":"2024-04-26T23:58:56.249104+0000","last_fullsized":"2024-04-26T23:58:56.249104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:05:49.532178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.377427+0000","last_change":"2024-04-26T23:58:51.170537+0000","last_active":"2024-04-26T23:58:56.377427+0000","last_peered":"2024-04-26T23:58:56.377427+0000","last_clean":"2024-04-26T23:58:56.377427+0000","last_became_active":"2024-04-26T23:58:51.170047+0000","last_became_peered":"2024-04-26T23:58:51.170047+0000","last_unstale":"2024-04-26T23:58:56.377427+0000","last_undegraded":"2024-04-26T23:58:56.377427+0000","last_fullsized":"2024-04-26T23:58:56.377427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:46.463214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247811+0000","last_change":"2024-04-26T23:58:51.141041+0000","last_active":"2024-04-26T23:58:56.247811+0000","last_peered":"2024-04-26T23:58:56.247811+0000","last_clean":"2024-04-26T23:58:56.247811+0000","last_became_active":"2024-04-26T23:58:51.140709+0000","last_became_peered":"2024-04-26T23:58:51.140709+0000","last_unstale":"2024-04-26T23:58:56.247811+0000","last_undegraded":"2024-04-26T23:58:56.247811+0000","last_fullsized":"2024-04-26T23:58:56.247811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:12:28.613723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247409+0000","last_change":"2024-04-26T23:58:51.145975+0000","last_active":"2024-04-26T23:58:56.247409+0000","last_peered":"2024-04-26T23:58:56.247409+0000","last_clean":"2024-04-26T23:58:56.247409+0000","last_became_active":"2024-04-26T23:58:51.145594+0000","last_became_peered":"2024-04-26T23:58:51.145594+0000","last_unstale":"2024-04-26T23:58:56.247409+0000","last_undegraded":"2024-04-26T23:58:56.247409+0000","last_fullsized":"2024-04-26T23:58:56.247409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:50:59.548009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211401+0000","last_change":"2024-04-26T23:58:51.160363+0000","last_active":"2024-04-26T23:58:56.211401+0000","last_peered":"2024-04-26T23:58:56.211401+0000","last_clean":"2024-04-26T23:58:56.211401+0000","last_became_active":"2024-04-26T23:58:51.152004+0000","last_became_peered":"2024-04-26T23:58:51.152004+0000","last_unstale":"2024-04-26T23:58:56.211401+0000","last_undegraded":"2024-04-26T23:58:56.211401+0000","last_fullsized":"2024-04-26T23:58:56.211401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:01:03.203404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247147+0000","last_change":"2024-04-26T23:58:51.134247+0000","last_active":"2024-04-26T23:58:56.247147+0000","last_peered":"2024-04-26T23:58:56.247147+0000","last_clean":"2024-04-26T23:58:56.247147+0000","last_became_active":"2024-04-26T23:58:51.133889+0000","last_became_peered":"2024-04-26T23:58:51.133889+0000","last_unstale":"2024-04-26T23:58:56.247147+0000","last_undegraded":"2024-04-26T23:58:56.247147+0000","last_fullsized":"2024-04-26T23:58:56.247147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:51:04.245217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248382+0000","last_change":"2024-04-26T23:58:51.135255+0000","last_active":"2024-04-26T23:58:56.248382+0000","last_peered":"2024-04-26T23:58:56.248382+0000","last_clean":"2024-04-26T23:58:56.248382+0000","last_became_active":"2024-04-26T23:58:51.134905+0000","last_became_peered":"2024-04-26T23:58:51.134905+0000","last_unstale":"2024-04-26T23:58:56.248382+0000","last_undegraded":"2024-04-26T23:58:56.248382+0000","last_fullsized":"2024-04-26T23:58:56.248382+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:51.435008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247933+0000","last_change":"2024-04-26T23:58:51.134962+0000","last_active":"2024-04-26T23:58:56.247933+0000","last_peered":"2024-04-26T23:58:56.247933+0000","last_clean":"2024-04-26T23:58:56.247933+0000","last_became_active":"2024-04-26T23:58:51.134559+0000","last_became_peered":"2024-04-26T23:58:51.134559+0000","last_unstale":"2024-04-26T23:58:56.247933+0000","last_undegraded":"2024-04-26T23:58:56.247933+0000","last_fullsized":"2024-04-26T23:58:56.247933+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:31:15.833585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249016+0000","last_change":"2024-04-26T23:58:51.141872+0000","last_active":"2024-04-26T23:58:56.249016+0000","last_peered":"2024-04-26T23:58:56.249016+0000","last_clean":"2024-04-26T23:58:56.249016+0000","last_became_active":"2024-04-26T23:58:51.141464+0000","last_became_peered":"2024-04-26T23:58:51.141464+0000","last_unstale":"2024-04-26T23:58:56.249016+0000","last_undegraded":"2024-04-26T23:58:56.249016+0000","last_fullsized":"2024-04-26T23:58:56.249016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:37.663379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.251986+0000","last_change":"2024-04-26T23:58:51.157828+0000","last_active":"2024-04-26T23:58:56.251986+0000","last_peered":"2024-04-26T23:58:56.251986+0000","last_clean":"2024-04-26T23:58:56.251986+0000","last_became_active":"2024-04-26T23:58:51.156820+0000","last_became_peered":"2024-04-26T23:58:51.156820+0000","last_unstale":"2024-04-26T23:58:56.251986+0000","last_undegraded":"2024-04-26T23:58:56.251986+0000","last_fullsized":"2024-04-26T23:58:56.251986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:22:43.201795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248034+0000","last_change":"2024-04-26T23:58:51.154108+0000","last_active":"2024-04-26T23:58:56.248034+0000","last_peered":"2024-04-26T23:58:56.248034+0000","last_clean":"2024-04-26T23:58:56.248034+0000","last_became_active":"2024-04-26T23:58:51.149740+0000","last_became_peered":"2024-04-26T23:58:51.149740+0000","last_unstale":"2024-04-26T23:58:56.248034+0000","last_undegraded":"2024-04-26T23:58:56.248034+0000","last_fullsized":"2024-04-26T23:58:56.248034+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:23:58.519297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247376+0000","last_change":"2024-04-26T23:58:51.164578+0000","last_active":"2024-04-26T23:58:56.247376+0000","last_peered":"2024-04-26T23:58:56.247376+0000","last_clean":"2024-04-26T23:58:56.247376+0000","last_became_active":"2024-04-26T23:58:51.152843+0000","last_became_peered":"2024-04-26T23:58:51.152843+0000","last_unstale":"2024-04-26T23:58:56.247376+0000","last_undegraded":"2024-04-26T23:58:56.247376+0000","last_fullsized":"2024-04-26T23:58:56.247376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:02.788917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.245726+0000","last_change":"2024-04-26T23:58:51.162811+0000","last_active":"2024-04-26T23:58:56.245726+0000","last_peered":"2024-04-26T23:58:56.245726+0000","last_clean":"2024-04-26T23:58:56.245726+0000","last_became_active":"2024-04-26T23:58:51.162464+0000","last_became_peered":"2024-04-26T23:58:51.162464+0000","last_unstale":"2024-04-26T23:58:56.245726+0000","last_undegraded":"2024-04-26T23:58:56.245726+0000","last_fullsized":"2024-04-26T23:58:56.245726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:50:26.127894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248127+0000","last_change":"2024-04-26T23:58:51.162363+0000","last_active":"2024-04-26T23:58:56.248127+0000","last_peered":"2024-04-26T23:58:56.248127+0000","last_clean":"2024-04-26T23:58:56.248127+0000","last_became_active":"2024-04-26T23:58:51.162033+0000","last_became_peered":"2024-04-26T23:58:51.162033+0000","last_unstale":"2024-04-26T23:58:56.248127+0000","last_undegraded":"2024-04-26T23:58:56.248127+0000","last_fullsized":"2024-04-26T23:58:56.248127+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:53.755338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250657+0000","last_change":"2024-04-26T23:58:51.147373+0000","last_active":"2024-04-26T23:58:56.250657+0000","last_peered":"2024-04-26T23:58:56.250657+0000","last_clean":"2024-04-26T23:58:56.250657+0000","last_became_active":"2024-04-26T23:58:51.146993+0000","last_became_peered":"2024-04-26T23:58:51.146993+0000","last_unstale":"2024-04-26T23:58:56.250657+0000","last_undegraded":"2024-04-26T23:58:56.250657+0000","last_fullsized":"2024-04-26T23:58:56.250657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:14.887719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247161+0000","last_change":"2024-04-26T23:58:51.159553+0000","last_active":"2024-04-26T23:58:56.247161+0000","last_peered":"2024-04-26T23:58:56.247161+0000","last_clean":"2024-04-26T23:58:56.247161+0000","last_became_active":"2024-04-26T23:58:51.159252+0000","last_became_peered":"2024-04-26T23:58:51.159252+0000","last_unstale":"2024-04-26T23:58:56.247161+0000","last_undegraded":"2024-04-26T23:58:56.247161+0000","last_fullsized":"2024-04-26T23:58:56.247161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:17.408273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246856+0000","last_change":"2024-04-26T23:58:55.203263+0000","last_active":"2024-04-26T23:58:56.246856+0000","last_peered":"2024-04-26T23:58:56.246856+0000","last_clean":"2024-04-26T23:58:56.246856+0000","last_became_active":"2024-04-26T23:58:55.202832+0000","last_became_peered":"2024-04-26T23:58:55.202832+0000","last_unstale":"2024-04-26T23:58:56.246856+0000","last_undegraded":"2024-04-26T23:58:56.246856+0000","last_fullsized":"2024-04-26T23:58:56.246856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:13:38.825916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.371998+0000","last_change":"2024-04-26T23:58:50.124003+0000","last_active":"2024-04-26T23:59:24.371998+0000","last_peered":"2024-04-26T23:59:24.371998+0000","last_clean":"2024-04-26T23:59:24.371998+0000","last_became_active":"2024-04-26T23:58:50.123653+0000","last_became_peered":"2024-04-26T23:58:50.123653+0000","last_unstale":"2024-04-26T23:59:24.371998+0000","last_undegraded":"2024-04-26T23:59:24.371998+0000","last_fullsized":"2024-04-26T23:59:24.371998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:16:57.688594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.212780+0000","last_change":"2024-04-26T23:58:51.175946+0000","last_active":"2024-04-26T23:58:56.212780+0000","last_peered":"2024-04-26T23:58:56.212780+0000","last_clean":"2024-04-26T23:58:56.212780+0000","last_became_active":"2024-04-26T23:58:51.174204+0000","last_became_peered":"2024-04-26T23:58:51.174204+0000","last_unstale":"2024-04-26T23:58:56.212780+0000","last_undegraded":"2024-04-26T23:58:56.212780+0000","last_fullsized":"2024-04-26T23:58:56.212780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:03.975717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248808+0000","last_change":"2024-04-26T23:58:55.204690+0000","last_active":"2024-04-26T23:58:56.248808+0000","last_peered":"2024-04-26T23:58:56.248808+0000","last_clean":"2024-04-26T23:58:56.248808+0000","last_became_active":"2024-04-26T23:58:55.204248+0000","last_became_peered":"2024-04-26T23:58:55.204248+0000","last_unstale":"2024-04-26T23:58:56.248808+0000","last_undegraded":"2024-04-26T23:58:56.248808+0000","last_fullsized":"2024-04-26T23:58:56.248808+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:21.009534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.213176+0000","last_change":"2024-04-26T23:58:50.128773+0000","last_active":"2024-04-26T23:58:56.213176+0000","last_peered":"2024-04-26T23:58:56.213176+0000","last_clean":"2024-04-26T23:58:56.213176+0000","last_became_active":"2024-04-26T23:58:50.128445+0000","last_became_peered":"2024-04-26T23:58:50.128445+0000","last_unstale":"2024-04-26T23:58:56.213176+0000","last_undegraded":"2024-04-26T23:58:56.213176+0000","last_fullsized":"2024-04-26T23:58:56.213176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:27:57.577361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248330+0000","last_change":"2024-04-26T23:58:51.164079+0000","last_active":"2024-04-26T23:58:56.248330+0000","last_peered":"2024-04-26T23:58:56.248330+0000","last_clean":"2024-04-26T23:58:56.248330+0000","last_became_active":"2024-04-26T23:58:51.162514+0000","last_became_peered":"2024-04-26T23:58:51.162514+0000","last_unstale":"2024-04-26T23:58:56.248330+0000","last_undegraded":"2024-04-26T23:58:56.248330+0000","last_fullsized":"2024-04-26T23:58:56.248330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:48:10.904737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249297+0000","last_change":"2024-04-26T23:58:55.211879+0000","last_active":"2024-04-26T23:58:56.249297+0000","last_peered":"2024-04-26T23:58:56.249297+0000","last_clean":"2024-04-26T23:58:56.249297+0000","last_became_active":"2024-04-26T23:58:55.211421+0000","last_became_peered":"2024-04-26T23:58:55.211421+0000","last_unstale":"2024-04-26T23:58:56.249297+0000","last_undegraded":"2024-04-26T23:58:56.249297+0000","last_fullsized":"2024-04-26T23:58:56.249297+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:45:12.754349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.362322+0000","last_change":"2024-04-26T23:58:50.101893+0000","last_active":"2024-04-26T23:59:24.362322+0000","last_peered":"2024-04-26T23:59:24.362322+0000","last_clean":"2024-04-26T23:59:24.362322+0000","last_became_active":"2024-04-26T23:58:50.101625+0000","last_became_peered":"2024-04-26T23:58:50.101625+0000","last_unstale":"2024-04-26T23:59:24.362322+0000","last_undegraded":"2024-04-26T23:59:24.362322+0000","last_fullsized":"2024-04-26T23:59:24.362322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:05:24.365710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246542+0000","last_change":"2024-04-26T23:58:51.146848+0000","last_active":"2024-04-26T23:58:56.246542+0000","last_peered":"2024-04-26T23:58:56.246542+0000","last_clean":"2024-04-26T23:58:56.246542+0000","last_became_active":"2024-04-26T23:58:51.146261+0000","last_became_peered":"2024-04-26T23:58:51.146261+0000","last_unstale":"2024-04-26T23:58:56.246542+0000","last_undegraded":"2024-04-26T23:58:56.246542+0000","last_fullsized":"2024-04-26T23:58:56.246542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:22.200191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248349+0000","last_change":"2024-04-26T23:58:55.196026+0000","last_active":"2024-04-26T23:58:56.248349+0000","last_peered":"2024-04-26T23:58:56.248349+0000","last_clean":"2024-04-26T23:58:56.248349+0000","last_became_active":"2024-04-26T23:58:55.195591+0000","last_became_peered":"2024-04-26T23:58:55.195591+0000","last_unstale":"2024-04-26T23:58:56.248349+0000","last_undegraded":"2024-04-26T23:58:56.248349+0000","last_fullsized":"2024-04-26T23:58:56.248349+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:06.980985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.377471+0000","last_change":"2024-04-26T23:58:50.100655+0000","last_active":"2024-04-26T23:58:56.377471+0000","last_peered":"2024-04-26T23:58:56.377471+0000","last_clean":"2024-04-26T23:58:56.377471+0000","last_became_active":"2024-04-26T23:58:50.100314+0000","last_became_peered":"2024-04-26T23:58:50.100314+0000","last_unstale":"2024-04-26T23:58:56.377471+0000","last_undegraded":"2024-04-26T23:58:56.377471+0000","last_fullsized":"2024-04-26T23:58:56.377471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:10:06.230717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.209327+0000","last_change":"2024-04-26T23:58:51.142669+0000","last_active":"2024-04-26T23:58:56.209327+0000","last_peered":"2024-04-26T23:58:56.209327+0000","last_clean":"2024-04-26T23:58:56.209327+0000","last_became_active":"2024-04-26T23:58:51.142337+0000","last_became_peered":"2024-04-26T23:58:51.142337+0000","last_unstale":"2024-04-26T23:58:56.209327+0000","last_undegraded":"2024-04-26T23:58:56.209327+0000","last_fullsized":"2024-04-26T23:58:56.209327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:33:57.351429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247207+0000","last_change":"2024-04-26T23:58:55.184331+0000","last_active":"2024-04-26T23:58:56.247207+0000","last_peered":"2024-04-26T23:58:56.247207+0000","last_clean":"2024-04-26T23:58:56.247207+0000","last_became_active":"2024-04-26T23:58:55.183928+0000","last_became_peered":"2024-04-26T23:58:55.183928+0000","last_unstale":"2024-04-26T23:58:56.247207+0000","last_undegraded":"2024-04-26T23:58:56.247207+0000","last_fullsized":"2024-04-26T23:58:56.247207+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:47.303104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:25.382919+0000","last_change":"2024-04-26T23:58:50.101084+0000","last_active":"2024-04-26T23:59:25.382919+0000","last_peered":"2024-04-26T23:59:25.382919+0000","last_clean":"2024-04-26T23:59:25.382919+0000","last_became_active":"2024-04-26T23:58:50.100736+0000","last_became_peered":"2024-04-26T23:58:50.100736+0000","last_unstale":"2024-04-26T23:59:25.382919+0000","last_undegraded":"2024-04-26T23:59:25.382919+0000","last_fullsized":"2024-04-26T23:59:25.382919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:43.157122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.377508+0000","last_change":"2024-04-26T23:58:51.143834+0000","last_active":"2024-04-26T23:58:56.377508+0000","last_peered":"2024-04-26T23:58:56.377508+0000","last_clean":"2024-04-26T23:58:56.377508+0000","last_became_active":"2024-04-26T23:58:51.143480+0000","last_became_peered":"2024-04-26T23:58:51.143480+0000","last_unstale":"2024-04-26T23:58:56.377508+0000","last_undegraded":"2024-04-26T23:58:56.377508+0000","last_fullsized":"2024-04-26T23:58:56.377508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:34:25.474562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247193+0000","last_change":"2024-04-26T23:58:55.215809+0000","last_active":"2024-04-26T23:58:56.247193+0000","last_peered":"2024-04-26T23:58:56.247193+0000","last_clean":"2024-04-26T23:58:56.247193+0000","last_became_active":"2024-04-26T23:58:55.215369+0000","last_became_peered":"2024-04-26T23:58:55.215369+0000","last_unstale":"2024-04-26T23:58:56.247193+0000","last_undegraded":"2024-04-26T23:58:56.247193+0000","last_fullsized":"2024-04-26T23:58:56.247193+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:04.053963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.364521+0000","last_change":"2024-04-26T23:58:50.129626+0000","last_active":"2024-04-26T23:59:24.364521+0000","last_peered":"2024-04-26T23:59:24.364521+0000","last_clean":"2024-04-26T23:59:24.364521+0000","last_became_active":"2024-04-26T23:58:50.129386+0000","last_became_peered":"2024-04-26T23:58:50.129386+0000","last_unstale":"2024-04-26T23:59:24.364521+0000","last_undegraded":"2024-04-26T23:59:24.364521+0000","last_fullsized":"2024-04-26T23:59:24.364521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:31:54.432289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.209192+0000","last_change":"2024-04-26T23:58:51.166312+0000","last_active":"2024-04-26T23:58:56.209192+0000","last_peered":"2024-04-26T23:58:56.209192+0000","last_clean":"2024-04-26T23:58:56.209192+0000","last_became_active":"2024-04-26T23:58:51.165986+0000","last_became_peered":"2024-04-26T23:58:51.165986+0000","last_unstale":"2024-04-26T23:58:56.209192+0000","last_undegraded":"2024-04-26T23:58:56.209192+0000","last_fullsized":"2024-04-26T23:58:56.209192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:51:49.549912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248210+0000","last_change":"2024-04-26T23:58:55.204142+0000","last_active":"2024-04-26T23:58:56.248210+0000","last_peered":"2024-04-26T23:58:56.248210+0000","last_clean":"2024-04-26T23:58:56.248210+0000","last_became_active":"2024-04-26T23:58:55.203646+0000","last_became_peered":"2024-04-26T23:58:55.203646+0000","last_unstale":"2024-04-26T23:58:56.248210+0000","last_undegraded":"2024-04-26T23:58:56.248210+0000","last_fullsized":"2024-04-26T23:58:56.248210+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:40.614034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.370945+0000","last_change":"2024-04-26T23:58:50.132080+0000","last_active":"2024-04-26T23:59:24.370945+0000","last_peered":"2024-04-26T23:59:24.370945+0000","last_clean":"2024-04-26T23:59:24.370945+0000","last_became_active":"2024-04-26T23:58:50.131618+0000","last_became_peered":"2024-04-26T23:58:50.131618+0000","last_unstale":"2024-04-26T23:59:24.370945+0000","last_undegraded":"2024-04-26T23:59:24.370945+0000","last_fullsized":"2024-04-26T23:59:24.370945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:42:07.897166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211042+0000","last_change":"2024-04-26T23:58:51.175083+0000","last_active":"2024-04-26T23:58:56.211042+0000","last_peered":"2024-04-26T23:58:56.211042+0000","last_clean":"2024-04-26T23:58:56.211042+0000","last_became_active":"2024-04-26T23:58:51.174159+0000","last_became_peered":"2024-04-26T23:58:51.174159+0000","last_unstale":"2024-04-26T23:58:56.211042+0000","last_undegraded":"2024-04-26T23:58:56.211042+0000","last_fullsized":"2024-04-26T23:58:56.211042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:51:06.583217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248664+0000","last_change":"2024-04-26T23:58:55.200019+0000","last_active":"2024-04-26T23:58:56.248664+0000","last_peered":"2024-04-26T23:58:56.248664+0000","last_clean":"2024-04-26T23:58:56.248664+0000","last_became_active":"2024-04-26T23:58:55.199593+0000","last_became_peered":"2024-04-26T23:58:55.199593+0000","last_unstale":"2024-04-26T23:58:56.248664+0000","last_undegraded":"2024-04-26T23:58:56.248664+0000","last_fullsized":"2024-04-26T23:58:56.248664+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:41.465294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208650+0000","last_change":"2024-04-26T23:58:50.100114+0000","last_active":"2024-04-26T23:58:56.208650+0000","last_peered":"2024-04-26T23:58:56.208650+0000","last_clean":"2024-04-26T23:58:56.208650+0000","last_became_active":"2024-04-26T23:58:50.099762+0000","last_became_peered":"2024-04-26T23:58:50.099762+0000","last_unstale":"2024-04-26T23:58:56.208650+0000","last_undegraded":"2024-04-26T23:58:56.208650+0000","last_fullsized":"2024-04-26T23:58:56.208650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:51:16.855014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249902+0000","last_change":"2024-04-26T23:58:51.171236+0000","last_active":"2024-04-26T23:58:56.249902+0000","last_peered":"2024-04-26T23:58:56.249902+0000","last_clean":"2024-04-26T23:58:56.249902+0000","last_became_active":"2024-04-26T23:58:51.170709+0000","last_became_peered":"2024-04-26T23:58:51.170709+0000","last_unstale":"2024-04-26T23:58:56.249902+0000","last_undegraded":"2024-04-26T23:58:56.249902+0000","last_fullsized":"2024-04-26T23:58:56.249902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:11.404827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.212920+0000","last_change":"2024-04-26T23:58:55.201931+0000","last_active":"2024-04-26T23:58:56.212920+0000","last_peered":"2024-04-26T23:58:56.212920+0000","last_clean":"2024-04-26T23:58:56.212920+0000","last_became_active":"2024-04-26T23:58:55.196810+0000","last_became_peered":"2024-04-26T23:58:55.196810+0000","last_unstale":"2024-04-26T23:58:56.212920+0000","last_undegraded":"2024-04-26T23:58:56.212920+0000","last_fullsized":"2024-04-26T23:58:56.212920+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:04:39.775990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.355047+0000","last_change":"2024-04-26T23:58:50.126477+0000","last_active":"2024-04-26T23:59:23.355047+0000","last_peered":"2024-04-26T23:59:23.355047+0000","last_clean":"2024-04-26T23:59:23.355047+0000","last_became_active":"2024-04-26T23:58:50.126205+0000","last_became_peered":"2024-04-26T23:58:50.126205+0000","last_unstale":"2024-04-26T23:59:23.355047+0000","last_undegraded":"2024-04-26T23:59:23.355047+0000","last_fullsized":"2024-04-26T23:59:23.355047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:03:18.513036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246708+0000","last_change":"2024-04-26T23:58:51.145782+0000","last_active":"2024-04-26T23:58:56.246708+0000","last_peered":"2024-04-26T23:58:56.246708+0000","last_clean":"2024-04-26T23:58:56.246708+0000","last_became_active":"2024-04-26T23:58:51.145344+0000","last_became_peered":"2024-04-26T23:58:51.145344+0000","last_unstale":"2024-04-26T23:58:56.246708+0000","last_undegraded":"2024-04-26T23:58:56.246708+0000","last_fullsized":"2024-04-26T23:58:56.246708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:32.062690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249720+0000","last_change":"2024-04-26T23:58:55.198953+0000","last_active":"2024-04-26T23:58:56.249720+0000","last_peered":"2024-04-26T23:58:56.249720+0000","last_clean":"2024-04-26T23:58:56.249720+0000","last_became_active":"2024-04-26T23:58:55.198560+0000","last_became_peered":"2024-04-26T23:58:55.198560+0000","last_unstale":"2024-04-26T23:58:56.249720+0000","last_undegraded":"2024-04-26T23:58:56.249720+0000","last_fullsized":"2024-04-26T23:58:56.249720+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:57:07.109698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.350161+0000","last_change":"2024-04-26T23:58:50.103489+0000","last_active":"2024-04-26T23:59:23.350161+0000","last_peered":"2024-04-26T23:59:23.350161+0000","last_clean":"2024-04-26T23:59:23.350161+0000","last_became_active":"2024-04-26T23:58:50.103068+0000","last_became_peered":"2024-04-26T23:58:50.103068+0000","last_unstale":"2024-04-26T23:59:23.350161+0000","last_undegraded":"2024-04-26T23:59:23.350161+0000","last_fullsized":"2024-04-26T23:59:23.350161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:22:26.490816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248710+0000","last_change":"2024-04-26T23:58:51.163680+0000","last_active":"2024-04-26T23:58:56.248710+0000","last_peered":"2024-04-26T23:58:56.248710+0000","last_clean":"2024-04-26T23:58:56.248710+0000","last_became_active":"2024-04-26T23:58:51.158135+0000","last_became_peered":"2024-04-26T23:58:51.158135+0000","last_unstale":"2024-04-26T23:58:56.248710+0000","last_undegraded":"2024-04-26T23:58:56.248710+0000","last_fullsized":"2024-04-26T23:58:56.248710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:56.562260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249350+0000","last_change":"2024-04-26T23:58:55.211908+0000","last_active":"2024-04-26T23:58:56.249350+0000","last_peered":"2024-04-26T23:58:56.249350+0000","last_clean":"2024-04-26T23:58:56.249350+0000","last_became_active":"2024-04-26T23:58:55.211430+0000","last_became_peered":"2024-04-26T23:58:55.211430+0000","last_unstale":"2024-04-26T23:58:56.249350+0000","last_undegraded":"2024-04-26T23:58:56.249350+0000","last_fullsized":"2024-04-26T23:58:56.249350+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:51:13.315395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.355592+0000","last_change":"2024-04-26T23:58:50.116030+0000","last_active":"2024-04-26T23:59:23.355592+0000","last_peered":"2024-04-26T23:59:23.355592+0000","last_clean":"2024-04-26T23:59:23.355592+0000","last_became_active":"2024-04-26T23:58:50.115744+0000","last_became_peered":"2024-04-26T23:58:50.115744+0000","last_unstale":"2024-04-26T23:59:23.355592+0000","last_undegraded":"2024-04-26T23:59:23.355592+0000","last_fullsized":"2024-04-26T23:59:23.355592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:05:21.823842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246738+0000","last_change":"2024-04-26T23:58:51.140652+0000","last_active":"2024-04-26T23:58:56.246738+0000","last_peered":"2024-04-26T23:58:56.246738+0000","last_clean":"2024-04-26T23:58:56.246738+0000","last_became_active":"2024-04-26T23:58:51.140330+0000","last_became_peered":"2024-04-26T23:58:51.140330+0000","last_unstale":"2024-04-26T23:58:56.246738+0000","last_undegraded":"2024-04-26T23:58:56.246738+0000","last_fullsized":"2024-04-26T23:58:56.246738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:59:56.037856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250695+0000","last_change":"2024-04-26T23:58:55.200964+0000","last_active":"2024-04-26T23:58:56.250695+0000","last_peered":"2024-04-26T23:58:56.250695+0000","last_clean":"2024-04-26T23:58:56.250695+0000","last_became_active":"2024-04-26T23:58:55.200404+0000","last_became_peered":"2024-04-26T23:58:55.200404+0000","last_unstale":"2024-04-26T23:58:56.250695+0000","last_undegraded":"2024-04-26T23:58:56.250695+0000","last_fullsized":"2024-04-26T23:58:56.250695+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:43:51.159653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'16","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:25.869195+0000","last_change":"2024-04-26T23:58:50.101850+0000","last_active":"2024-04-26T23:59:25.869195+0000","last_peered":"2024-04-26T23:59:25.869195+0000","last_clean":"2024-04-26T23:59:25.869195+0000","last_became_active":"2024-04-26T23:58:50.101521+0000","last_became_peered":"2024-04-26T23:58:50.101521+0000","last_unstale":"2024-04-26T23:59:25.869195+0000","last_undegraded":"2024-04-26T23:59:25.869195+0000","last_fullsized":"2024-04-26T23:59:25.869195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:06.810644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10442,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.377740+0000","last_change":"2024-04-26T23:58:51.143441+0000","last_active":"2024-04-26T23:58:56.377740+0000","last_peered":"2024-04-26T23:58:56.377740+0000","last_clean":"2024-04-26T23:58:56.377740+0000","last_became_active":"2024-04-26T23:58:51.141779+0000","last_became_peered":"2024-04-26T23:58:51.141779+0000","last_unstale":"2024-04-26T23:58:56.377740+0000","last_undegraded":"2024-04-26T23:58:56.377740+0000","last_fullsized":"2024-04-26T23:58:56.377740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:56:28.067685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.209638+0000","last_change":"2024-04-26T23:58:55.207277+0000","last_active":"2024-04-26T23:58:56.209638+0000","last_peered":"2024-04-26T23:58:56.209638+0000","last_clean":"2024-04-26T23:58:56.209638+0000","last_became_active":"2024-04-26T23:58:55.206905+0000","last_became_peered":"2024-04-26T23:58:55.206905+0000","last_unstale":"2024-04-26T23:58:56.209638+0000","last_undegraded":"2024-04-26T23:58:56.209638+0000","last_fullsized":"2024-04-26T23:58:56.209638+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:03:34.254752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.363766+0000","last_change":"2024-04-26T23:58:50.115642+0000","last_active":"2024-04-26T23:59:24.363766+0000","last_peered":"2024-04-26T23:59:24.363766+0000","last_clean":"2024-04-26T23:59:24.363766+0000","last_became_active":"2024-04-26T23:58:50.115310+0000","last_became_peered":"2024-04-26T23:58:50.115310+0000","last_unstale":"2024-04-26T23:59:24.363766+0000","last_undegraded":"2024-04-26T23:59:24.363766+0000","last_fullsized":"2024-04-26T23:59:24.363766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:33.189230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250119+0000","last_change":"2024-04-26T23:58:51.159397+0000","last_active":"2024-04-26T23:58:56.250119+0000","last_peered":"2024-04-26T23:58:56.250119+0000","last_clean":"2024-04-26T23:58:56.250119+0000","last_became_active":"2024-04-26T23:58:51.159054+0000","last_became_peered":"2024-04-26T23:58:51.159054+0000","last_unstale":"2024-04-26T23:58:56.250119+0000","last_undegraded":"2024-04-26T23:58:56.250119+0000","last_fullsized":"2024-04-26T23:58:56.250119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:54:13.306198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247931+0000","last_change":"2024-04-26T23:58:55.215794+0000","last_active":"2024-04-26T23:58:56.247931+0000","last_peered":"2024-04-26T23:58:56.247931+0000","last_clean":"2024-04-26T23:58:56.247931+0000","last_became_active":"2024-04-26T23:58:55.215366+0000","last_became_peered":"2024-04-26T23:58:55.215366+0000","last_unstale":"2024-04-26T23:58:56.247931+0000","last_undegraded":"2024-04-26T23:58:56.247931+0000","last_fullsized":"2024-04-26T23:58:56.247931+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:54:20.347552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.353811+0000","last_change":"2024-04-26T23:58:50.112207+0000","last_active":"2024-04-26T23:59:23.353811+0000","last_peered":"2024-04-26T23:59:23.353811+0000","last_clean":"2024-04-26T23:59:23.353811+0000","last_became_active":"2024-04-26T23:58:50.111975+0000","last_became_peered":"2024-04-26T23:58:50.111975+0000","last_unstale":"2024-04-26T23:59:23.353811+0000","last_undegraded":"2024-04-26T23:59:23.353811+0000","last_fullsized":"2024-04-26T23:59:23.353811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:29.091698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247517+0000","last_change":"2024-04-26T23:58:51.147154+0000","last_active":"2024-04-26T23:58:56.247517+0000","last_peered":"2024-04-26T23:58:56.247517+0000","last_clean":"2024-04-26T23:58:56.247517+0000","last_became_active":"2024-04-26T23:58:51.146809+0000","last_became_peered":"2024-04-26T23:58:51.146809+0000","last_unstale":"2024-04-26T23:58:56.247517+0000","last_undegraded":"2024-04-26T23:58:56.247517+0000","last_fullsized":"2024-04-26T23:58:56.247517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:22.308365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.252855+0000","last_change":"2024-04-26T23:58:55.214461+0000","last_active":"2024-04-26T23:58:56.252855+0000","last_peered":"2024-04-26T23:58:56.252855+0000","last_clean":"2024-04-26T23:58:56.252855+0000","last_became_active":"2024-04-26T23:58:55.213442+0000","last_became_peered":"2024-04-26T23:58:55.213442+0000","last_unstale":"2024-04-26T23:58:56.252855+0000","last_undegraded":"2024-04-26T23:58:56.252855+0000","last_fullsized":"2024-04-26T23:58:56.252855+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:07:03.114274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:26.390040+0000","last_change":"2024-04-26T23:58:50.100554+0000","last_active":"2024-04-26T23:59:26.390040+0000","last_peered":"2024-04-26T23:59:26.390040+0000","last_clean":"2024-04-26T23:59:26.390040+0000","last_became_active":"2024-04-26T23:58:50.100200+0000","last_became_peered":"2024-04-26T23:58:50.100200+0000","last_unstale":"2024-04-26T23:59:26.390040+0000","last_undegraded":"2024-04-26T23:59:26.390040+0000","last_fullsized":"2024-04-26T23:59:26.390040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:59.913424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247246+0000","last_change":"2024-04-26T23:58:51.141030+0000","last_active":"2024-04-26T23:58:56.247246+0000","last_peered":"2024-04-26T23:58:56.247246+0000","last_clean":"2024-04-26T23:58:56.247246+0000","last_became_active":"2024-04-26T23:58:51.140694+0000","last_became_peered":"2024-04-26T23:58:51.140694+0000","last_unstale":"2024-04-26T23:58:56.247246+0000","last_undegraded":"2024-04-26T23:58:56.247246+0000","last_fullsized":"2024-04-26T23:58:56.247246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:35:29.827020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.252130+0000","last_change":"2024-04-26T23:58:55.216556+0000","last_active":"2024-04-26T23:58:56.252130+0000","last_peered":"2024-04-26T23:58:56.252130+0000","last_clean":"2024-04-26T23:58:56.252130+0000","last_became_active":"2024-04-26T23:58:55.215798+0000","last_became_peered":"2024-04-26T23:58:55.215798+0000","last_unstale":"2024-04-26T23:58:56.252130+0000","last_undegraded":"2024-04-26T23:58:56.252130+0000","last_fullsized":"2024-04-26T23:58:56.252130+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:32:08.733655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.367485+0000","last_change":"2024-04-26T23:58:50.119495+0000","last_active":"2024-04-26T23:59:24.367485+0000","last_peered":"2024-04-26T23:59:24.367485+0000","last_clean":"2024-04-26T23:59:24.367485+0000","last_became_active":"2024-04-26T23:58:50.119174+0000","last_became_peered":"2024-04-26T23:58:50.119174+0000","last_unstale":"2024-04-26T23:59:24.367485+0000","last_undegraded":"2024-04-26T23:59:24.367485+0000","last_fullsized":"2024-04-26T23:59:24.367485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:53:23.408578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208753+0000","last_change":"2024-04-26T23:58:51.141943+0000","last_active":"2024-04-26T23:58:56.208753+0000","last_peered":"2024-04-26T23:58:56.208753+0000","last_clean":"2024-04-26T23:58:56.208753+0000","last_became_active":"2024-04-26T23:58:51.141624+0000","last_became_peered":"2024-04-26T23:58:51.141624+0000","last_unstale":"2024-04-26T23:58:56.208753+0000","last_undegraded":"2024-04-26T23:58:56.208753+0000","last_fullsized":"2024-04-26T23:58:56.208753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:46:25.114059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208837+0000","last_change":"2024-04-26T23:58:55.200247+0000","last_active":"2024-04-26T23:58:56.208837+0000","last_peered":"2024-04-26T23:58:56.208837+0000","last_clean":"2024-04-26T23:58:56.208837+0000","last_became_active":"2024-04-26T23:58:55.199854+0000","last_became_peered":"2024-04-26T23:58:55.199854+0000","last_unstale":"2024-04-26T23:58:56.208837+0000","last_undegraded":"2024-04-26T23:58:56.208837+0000","last_fullsized":"2024-04-26T23:58:56.208837+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:22:00.066549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:25.383274+0000","last_change":"2024-04-26T23:58:50.102901+0000","last_active":"2024-04-26T23:59:25.383274+0000","last_peered":"2024-04-26T23:59:25.383274+0000","last_clean":"2024-04-26T23:59:25.383274+0000","last_became_active":"2024-04-26T23:58:50.102537+0000","last_became_peered":"2024-04-26T23:58:50.102537+0000","last_unstale":"2024-04-26T23:59:25.383274+0000","last_undegraded":"2024-04-26T23:59:25.383274+0000","last_fullsized":"2024-04-26T23:59:25.383274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:21:15.043327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248067+0000","last_change":"2024-04-26T23:58:51.135160+0000","last_active":"2024-04-26T23:58:56.248067+0000","last_peered":"2024-04-26T23:58:56.248067+0000","last_clean":"2024-04-26T23:58:56.248067+0000","last_became_active":"2024-04-26T23:58:51.134806+0000","last_became_peered":"2024-04-26T23:58:51.134806+0000","last_unstale":"2024-04-26T23:58:56.248067+0000","last_undegraded":"2024-04-26T23:58:56.248067+0000","last_fullsized":"2024-04-26T23:58:56.248067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:35:10.839180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.377785+0000","last_change":"2024-04-26T23:58:55.203555+0000","last_active":"2024-04-26T23:58:56.377785+0000","last_peered":"2024-04-26T23:58:56.377785+0000","last_clean":"2024-04-26T23:58:56.377785+0000","last_became_active":"2024-04-26T23:58:55.202968+0000","last_became_peered":"2024-04-26T23:58:55.202968+0000","last_unstale":"2024-04-26T23:58:56.377785+0000","last_undegraded":"2024-04-26T23:58:56.377785+0000","last_fullsized":"2024-04-26T23:58:56.377785+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:01.149966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250158+0000","last_change":"2024-04-26T23:58:50.129000+0000","last_active":"2024-04-26T23:58:56.250158+0000","last_peered":"2024-04-26T23:58:56.250158+0000","last_clean":"2024-04-26T23:58:56.250158+0000","last_became_active":"2024-04-26T23:58:50.128646+0000","last_became_peered":"2024-04-26T23:58:50.128646+0000","last_unstale":"2024-04-26T23:58:56.250158+0000","last_undegraded":"2024-04-26T23:58:56.250158+0000","last_fullsized":"2024-04-26T23:58:56.250158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:34.200578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247722+0000","last_change":"2024-04-26T23:58:51.159557+0000","last_active":"2024-04-26T23:58:56.247722+0000","last_peered":"2024-04-26T23:58:56.247722+0000","last_clean":"2024-04-26T23:58:56.247722+0000","last_became_active":"2024-04-26T23:58:51.159211+0000","last_became_peered":"2024-04-26T23:58:51.159211+0000","last_unstale":"2024-04-26T23:58:56.247722+0000","last_undegraded":"2024-04-26T23:58:56.247722+0000","last_fullsized":"2024-04-26T23:58:56.247722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:48.740612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.253576+0000","last_change":"2024-04-26T23:58:55.193585+0000","last_active":"2024-04-26T23:58:56.253576+0000","last_peered":"2024-04-26T23:58:56.253576+0000","last_clean":"2024-04-26T23:58:56.253576+0000","last_became_active":"2024-04-26T23:58:55.193031+0000","last_became_peered":"2024-04-26T23:58:55.193031+0000","last_unstale":"2024-04-26T23:58:56.253576+0000","last_undegraded":"2024-04-26T23:58:56.253576+0000","last_fullsized":"2024-04-26T23:58:56.253576+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:20.689117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247659+0000","last_change":"2024-04-26T23:58:50.124301+0000","last_active":"2024-04-26T23:58:56.247659+0000","last_peered":"2024-04-26T23:58:56.247659+0000","last_clean":"2024-04-26T23:58:56.247659+0000","last_became_active":"2024-04-26T23:58:50.124016+0000","last_became_peered":"2024-04-26T23:58:50.124016+0000","last_unstale":"2024-04-26T23:58:56.247659+0000","last_undegraded":"2024-04-26T23:58:56.247659+0000","last_fullsized":"2024-04-26T23:58:56.247659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:00.835698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247845+0000","last_change":"2024-04-26T23:58:51.164703+0000","last_active":"2024-04-26T23:58:56.247845+0000","last_peered":"2024-04-26T23:58:56.247845+0000","last_clean":"2024-04-26T23:58:56.247845+0000","last_became_active":"2024-04-26T23:58:51.164346+0000","last_became_peered":"2024-04-26T23:58:51.164346+0000","last_unstale":"2024-04-26T23:58:56.247845+0000","last_undegraded":"2024-04-26T23:58:56.247845+0000","last_fullsized":"2024-04-26T23:58:56.247845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:45:39.667034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.252225+0000","last_change":"2024-04-26T23:58:55.193795+0000","last_active":"2024-04-26T23:58:56.252225+0000","last_peered":"2024-04-26T23:58:56.252225+0000","last_clean":"2024-04-26T23:58:56.252225+0000","last_became_active":"2024-04-26T23:58:55.193323+0000","last_became_peered":"2024-04-26T23:58:55.193323+0000","last_unstale":"2024-04-26T23:58:56.252225+0000","last_undegraded":"2024-04-26T23:58:56.252225+0000","last_fullsized":"2024-04-26T23:58:56.252225+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:44.301311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208737+0000","last_change":"2024-04-26T23:58:51.140198+0000","last_active":"2024-04-26T23:58:56.208737+0000","last_peered":"2024-04-26T23:58:56.208737+0000","last_clean":"2024-04-26T23:58:56.208737+0000","last_became_active":"2024-04-26T23:58:51.139851+0000","last_became_peered":"2024-04-26T23:58:51.139851+0000","last_unstale":"2024-04-26T23:58:56.208737+0000","last_undegraded":"2024-04-26T23:58:56.208737+0000","last_fullsized":"2024-04-26T23:58:56.208737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:50.356312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:25.385965+0000","last_change":"2024-04-26T23:58:50.132099+0000","last_active":"2024-04-26T23:59:25.385965+0000","last_peered":"2024-04-26T23:59:25.385965+0000","last_clean":"2024-04-26T23:59:25.385965+0000","last_became_active":"2024-04-26T23:58:50.131712+0000","last_became_peered":"2024-04-26T23:58:50.131712+0000","last_unstale":"2024-04-26T23:59:25.385965+0000","last_undegraded":"2024-04-26T23:59:25.385965+0000","last_fullsized":"2024-04-26T23:59:25.385965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:40.849259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:57.789831+0000","last_change":"2024-04-26T23:58:55.205614+0000","last_active":"2024-04-26T23:58:57.789831+0000","last_peered":"2024-04-26T23:58:57.789831+0000","last_clean":"2024-04-26T23:58:57.789831+0000","last_became_active":"2024-04-26T23:58:55.205206+0000","last_became_peered":"2024-04-26T23:58:55.205206+0000","last_unstale":"2024-04-26T23:58:57.789831+0000","last_undegraded":"2024-04-26T23:58:57.789831+0000","last_fullsized":"2024-04-26T23:58:57.789831+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:42:24.324563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211463+0000","last_change":"2024-04-26T23:58:51.159808+0000","last_active":"2024-04-26T23:58:56.211463+0000","last_peered":"2024-04-26T23:58:56.211463+0000","last_clean":"2024-04-26T23:58:56.211463+0000","last_became_active":"2024-04-26T23:58:51.159444+0000","last_became_peered":"2024-04-26T23:58:51.159444+0000","last_unstale":"2024-04-26T23:58:56.211463+0000","last_undegraded":"2024-04-26T23:58:56.211463+0000","last_fullsized":"2024-04-26T23:58:56.211463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:57.401134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:25.381198+0000","last_change":"2024-04-26T23:58:50.120100+0000","last_active":"2024-04-26T23:59:25.381198+0000","last_peered":"2024-04-26T23:59:25.381198+0000","last_clean":"2024-04-26T23:59:25.381198+0000","last_became_active":"2024-04-26T23:58:50.119613+0000","last_became_peered":"2024-04-26T23:58:50.119613+0000","last_unstale":"2024-04-26T23:59:25.381198+0000","last_undegraded":"2024-04-26T23:59:25.381198+0000","last_fullsized":"2024-04-26T23:59:25.381198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:32:18.116454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":343,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247017+0000","last_change":"2024-04-26T23:57:30.173979+0000","last_active":"2024-04-26T23:58:56.247017+0000","last_peered":"2024-04-26T23:58:56.247017+0000","last_clean":"2024-04-26T23:58:56.247017+0000","last_became_active":"2024-04-26T23:57:30.173549+0000","last_became_peered":"2024-04-26T23:57:30.173549+0000","last_unstale":"2024-04-26T23:58:56.247017+0000","last_undegraded":"2024-04-26T23:58:56.247017+0000","last_fullsized":"2024-04-26T23:58:56.247017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:53:33.634094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:53:33.634094+0000","last_clean_scrub_stamp":"2024-04-26T23:53:33.634094+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:48.026677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.209459+0000","last_change":"2024-04-26T23:58:55.214053+0000","last_active":"2024-04-26T23:58:56.209459+0000","last_peered":"2024-04-26T23:58:56.209459+0000","last_clean":"2024-04-26T23:58:56.209459+0000","last_became_active":"2024-04-26T23:58:55.213586+0000","last_became_peered":"2024-04-26T23:58:55.213586+0000","last_unstale":"2024-04-26T23:58:56.209459+0000","last_undegraded":"2024-04-26T23:58:56.209459+0000","last_fullsized":"2024-04-26T23:58:56.209459+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:57:16.843969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248263+0000","last_change":"2024-04-26T23:58:51.152687+0000","last_active":"2024-04-26T23:58:56.248263+0000","last_peered":"2024-04-26T23:58:56.248263+0000","last_clean":"2024-04-26T23:58:56.248263+0000","last_became_active":"2024-04-26T23:58:51.149082+0000","last_became_peered":"2024-04-26T23:58:51.149082+0000","last_unstale":"2024-04-26T23:58:56.248263+0000","last_undegraded":"2024-04-26T23:58:56.248263+0000","last_fullsized":"2024-04-26T23:58:56.248263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:38:42.630214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.361993+0000","last_change":"2024-04-26T23:58:50.104485+0000","last_active":"2024-04-26T23:59:24.361993+0000","last_peered":"2024-04-26T23:59:24.361993+0000","last_clean":"2024-04-26T23:59:24.361993+0000","last_became_active":"2024-04-26T23:58:50.104260+0000","last_became_peered":"2024-04-26T23:58:50.104260+0000","last_unstale":"2024-04-26T23:59:24.361993+0000","last_undegraded":"2024-04-26T23:59:24.361993+0000","last_fullsized":"2024-04-26T23:59:24.361993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:04:06.268993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247930+0000","last_change":"2024-04-26T23:58:55.190219+0000","last_active":"2024-04-26T23:58:56.247930+0000","last_peered":"2024-04-26T23:58:56.247930+0000","last_clean":"2024-04-26T23:58:56.247930+0000","last_became_active":"2024-04-26T23:58:55.189801+0000","last_became_peered":"2024-04-26T23:58:55.189801+0000","last_unstale":"2024-04-26T23:58:56.247930+0000","last_undegraded":"2024-04-26T23:58:56.247930+0000","last_fullsized":"2024-04-26T23:58:56.247930+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:53:42.170710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.354491+0000","last_change":"2024-04-26T23:58:50.117442+0000","last_active":"2024-04-26T23:59:23.354491+0000","last_peered":"2024-04-26T23:59:23.354491+0000","last_clean":"2024-04-26T23:59:23.354491+0000","last_became_active":"2024-04-26T23:58:50.117112+0000","last_became_peered":"2024-04-26T23:58:50.117112+0000","last_unstale":"2024-04-26T23:59:23.354491+0000","last_undegraded":"2024-04-26T23:59:23.354491+0000","last_fullsized":"2024-04-26T23:59:23.354491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:18.352145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.251795+0000","last_change":"2024-04-26T23:58:51.156670+0000","last_active":"2024-04-26T23:58:56.251795+0000","last_peered":"2024-04-26T23:58:56.251795+0000","last_clean":"2024-04-26T23:58:56.251795+0000","last_became_active":"2024-04-26T23:58:51.155596+0000","last_became_peered":"2024-04-26T23:58:51.155596+0000","last_unstale":"2024-04-26T23:58:56.251795+0000","last_undegraded":"2024-04-26T23:58:56.251795+0000","last_fullsized":"2024-04-26T23:58:56.251795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:38.053814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211366+0000","last_change":"2024-04-26T23:58:55.201644+0000","last_active":"2024-04-26T23:58:56.211366+0000","last_peered":"2024-04-26T23:58:56.211366+0000","last_clean":"2024-04-26T23:58:56.211366+0000","last_became_active":"2024-04-26T23:58:55.196421+0000","last_became_peered":"2024-04-26T23:58:55.196421+0000","last_unstale":"2024-04-26T23:58:56.211366+0000","last_undegraded":"2024-04-26T23:58:56.211366+0000","last_fullsized":"2024-04-26T23:58:56.211366+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:26:30.309773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.372055+0000","last_change":"2024-04-26T23:58:50.120805+0000","last_active":"2024-04-26T23:59:24.372055+0000","last_peered":"2024-04-26T23:59:24.372055+0000","last_clean":"2024-04-26T23:59:24.372055+0000","last_became_active":"2024-04-26T23:58:50.120403+0000","last_became_peered":"2024-04-26T23:58:50.120403+0000","last_unstale":"2024-04-26T23:59:24.372055+0000","last_undegraded":"2024-04-26T23:59:24.372055+0000","last_fullsized":"2024-04-26T23:59:24.372055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:53:28.950313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.212866+0000","last_change":"2024-04-26T23:58:51.167109+0000","last_active":"2024-04-26T23:58:56.212866+0000","last_peered":"2024-04-26T23:58:56.212866+0000","last_clean":"2024-04-26T23:58:56.212866+0000","last_became_active":"2024-04-26T23:58:51.164584+0000","last_became_peered":"2024-04-26T23:58:51.164584+0000","last_unstale":"2024-04-26T23:58:56.212866+0000","last_undegraded":"2024-04-26T23:58:56.212866+0000","last_fullsized":"2024-04-26T23:58:56.212866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:12:12.160553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246637+0000","last_change":"2024-04-26T23:58:55.183138+0000","last_active":"2024-04-26T23:58:56.246637+0000","last_peered":"2024-04-26T23:58:56.246637+0000","last_clean":"2024-04-26T23:58:56.246637+0000","last_became_active":"2024-04-26T23:58:55.182779+0000","last_became_peered":"2024-04-26T23:58:55.182779+0000","last_unstale":"2024-04-26T23:58:56.246637+0000","last_undegraded":"2024-04-26T23:58:56.246637+0000","last_fullsized":"2024-04-26T23:58:56.246637+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:16.866145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.350263+0000","last_change":"2024-04-26T23:58:50.120401+0000","last_active":"2024-04-26T23:59:23.350263+0000","last_peered":"2024-04-26T23:59:23.350263+0000","last_clean":"2024-04-26T23:59:23.350263+0000","last_became_active":"2024-04-26T23:58:50.120036+0000","last_became_peered":"2024-04-26T23:58:50.120036+0000","last_unstale":"2024-04-26T23:59:23.350263+0000","last_undegraded":"2024-04-26T23:59:23.350263+0000","last_fullsized":"2024-04-26T23:59:23.350263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:49.852298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248960+0000","last_change":"2024-04-26T23:58:51.171317+0000","last_active":"2024-04-26T23:58:56.248960+0000","last_peered":"2024-04-26T23:58:56.248960+0000","last_clean":"2024-04-26T23:58:56.248960+0000","last_became_active":"2024-04-26T23:58:51.170978+0000","last_became_peered":"2024-04-26T23:58:51.170978+0000","last_unstale":"2024-04-26T23:58:56.248960+0000","last_undegraded":"2024-04-26T23:58:56.248960+0000","last_fullsized":"2024-04-26T23:58:56.248960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:29.173423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248952+0000","last_change":"2024-04-26T23:58:55.216273+0000","last_active":"2024-04-26T23:58:56.248952+0000","last_peered":"2024-04-26T23:58:56.248952+0000","last_clean":"2024-04-26T23:58:56.248952+0000","last_became_active":"2024-04-26T23:58:55.215630+0000","last_became_peered":"2024-04-26T23:58:55.215630+0000","last_unstale":"2024-04-26T23:58:56.248952+0000","last_undegraded":"2024-04-26T23:58:56.248952+0000","last_fullsized":"2024-04-26T23:58:56.248952+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:15:41.195389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.353662+0000","last_change":"2024-04-26T23:58:50.111216+0000","last_active":"2024-04-26T23:59:23.353662+0000","last_peered":"2024-04-26T23:59:23.353662+0000","last_clean":"2024-04-26T23:59:23.353662+0000","last_became_active":"2024-04-26T23:58:50.110969+0000","last_became_peered":"2024-04-26T23:58:50.110969+0000","last_unstale":"2024-04-26T23:59:23.353662+0000","last_undegraded":"2024-04-26T23:59:23.353662+0000","last_fullsized":"2024-04-26T23:59:23.353662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:01.319448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.377268+0000","last_change":"2024-04-26T23:58:51.132643+0000","last_active":"2024-04-26T23:58:56.377268+0000","last_peered":"2024-04-26T23:58:56.377268+0000","last_clean":"2024-04-26T23:58:56.377268+0000","last_became_active":"2024-04-26T23:58:51.132250+0000","last_became_peered":"2024-04-26T23:58:51.132250+0000","last_unstale":"2024-04-26T23:58:56.377268+0000","last_undegraded":"2024-04-26T23:58:56.377268+0000","last_fullsized":"2024-04-26T23:58:56.377268+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:00:11.180178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.253164+0000","last_change":"2024-04-26T23:58:55.204718+0000","last_active":"2024-04-26T23:58:56.253164+0000","last_peered":"2024-04-26T23:58:56.253164+0000","last_clean":"2024-04-26T23:58:56.253164+0000","last_became_active":"2024-04-26T23:58:55.204159+0000","last_became_peered":"2024-04-26T23:58:55.204159+0000","last_unstale":"2024-04-26T23:58:56.253164+0000","last_undegraded":"2024-04-26T23:58:56.253164+0000","last_fullsized":"2024-04-26T23:58:56.253164+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:42:52.199200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.355283+0000","last_change":"2024-04-26T23:58:50.126926+0000","last_active":"2024-04-26T23:59:23.355283+0000","last_peered":"2024-04-26T23:59:23.355283+0000","last_clean":"2024-04-26T23:59:23.355283+0000","last_became_active":"2024-04-26T23:58:50.126552+0000","last_became_peered":"2024-04-26T23:58:50.126552+0000","last_unstale":"2024-04-26T23:59:23.355283+0000","last_undegraded":"2024-04-26T23:59:23.355283+0000","last_fullsized":"2024-04-26T23:59:23.355283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:18:25.024805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247596+0000","last_change":"2024-04-26T23:58:51.162722+0000","last_active":"2024-04-26T23:58:56.247596+0000","last_peered":"2024-04-26T23:58:56.247596+0000","last_clean":"2024-04-26T23:58:56.247596+0000","last_became_active":"2024-04-26T23:58:51.162387+0000","last_became_peered":"2024-04-26T23:58:51.162387+0000","last_unstale":"2024-04-26T23:58:56.247596+0000","last_undegraded":"2024-04-26T23:58:56.247596+0000","last_fullsized":"2024-04-26T23:58:56.247596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:07.143478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248784+0000","last_change":"2024-04-26T23:58:55.216201+0000","last_active":"2024-04-26T23:58:56.248784+0000","last_peered":"2024-04-26T23:58:56.248784+0000","last_clean":"2024-04-26T23:58:56.248784+0000","last_became_active":"2024-04-26T23:58:55.215730+0000","last_became_peered":"2024-04-26T23:58:55.215730+0000","last_unstale":"2024-04-26T23:58:56.248784+0000","last_undegraded":"2024-04-26T23:58:56.248784+0000","last_fullsized":"2024-04-26T23:58:56.248784+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:21.484879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246797+0000","last_change":"2024-04-26T23:58:50.111313+0000","last_active":"2024-04-26T23:58:56.246797+0000","last_peered":"2024-04-26T23:58:56.246797+0000","last_clean":"2024-04-26T23:58:56.246797+0000","last_became_active":"2024-04-26T23:58:50.110937+0000","last_became_peered":"2024-04-26T23:58:50.110937+0000","last_unstale":"2024-04-26T23:58:56.246797+0000","last_undegraded":"2024-04-26T23:58:56.246797+0000","last_fullsized":"2024-04-26T23:58:56.246797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:52:24.615527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.212337+0000","last_change":"2024-04-26T23:58:51.174152+0000","last_active":"2024-04-26T23:58:56.212337+0000","last_peered":"2024-04-26T23:58:56.212337+0000","last_clean":"2024-04-26T23:58:56.212337+0000","last_became_active":"2024-04-26T23:58:51.170956+0000","last_became_peered":"2024-04-26T23:58:51.170956+0000","last_unstale":"2024-04-26T23:58:56.212337+0000","last_undegraded":"2024-04-26T23:58:56.212337+0000","last_fullsized":"2024-04-26T23:58:56.212337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:09:09.253242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248697+0000","last_change":"2024-04-26T23:58:55.206248+0000","last_active":"2024-04-26T23:58:56.248697+0000","last_peered":"2024-04-26T23:58:56.248697+0000","last_clean":"2024-04-26T23:58:56.248697+0000","last_became_active":"2024-04-26T23:58:55.205820+0000","last_became_peered":"2024-04-26T23:58:55.205820+0000","last_unstale":"2024-04-26T23:58:56.248697+0000","last_undegraded":"2024-04-26T23:58:56.248697+0000","last_fullsized":"2024-04-26T23:58:56.248697+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:52:24.789706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.350261+0000","last_change":"2024-04-26T23:58:50.103874+0000","last_active":"2024-04-26T23:59:23.350261+0000","last_peered":"2024-04-26T23:59:23.350261+0000","last_clean":"2024-04-26T23:59:23.350261+0000","last_became_active":"2024-04-26T23:58:50.103590+0000","last_became_peered":"2024-04-26T23:58:50.103590+0000","last_unstale":"2024-04-26T23:59:23.350261+0000","last_undegraded":"2024-04-26T23:59:23.350261+0000","last_fullsized":"2024-04-26T23:59:23.350261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:34:59.817736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247722+0000","last_change":"2024-04-26T23:58:51.146396+0000","last_active":"2024-04-26T23:58:56.247722+0000","last_peered":"2024-04-26T23:58:56.247722+0000","last_clean":"2024-04-26T23:58:56.247722+0000","last_became_active":"2024-04-26T23:58:51.146046+0000","last_became_peered":"2024-04-26T23:58:51.146046+0000","last_unstale":"2024-04-26T23:58:56.247722+0000","last_undegraded":"2024-04-26T23:58:56.247722+0000","last_fullsized":"2024-04-26T23:58:56.247722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:14.230111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246313+0000","last_change":"2024-04-26T23:58:55.199320+0000","last_active":"2024-04-26T23:58:56.246313+0000","last_peered":"2024-04-26T23:58:56.246313+0000","last_clean":"2024-04-26T23:58:56.246313+0000","last_became_active":"2024-04-26T23:58:55.198843+0000","last_became_peered":"2024-04-26T23:58:55.198843+0000","last_unstale":"2024-04-26T23:58:56.246313+0000","last_undegraded":"2024-04-26T23:58:56.246313+0000","last_fullsized":"2024-04-26T23:58:56.246313+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:56:54.643998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.368654+0000","last_change":"2024-04-26T23:58:50.120892+0000","last_active":"2024-04-26T23:59:24.368654+0000","last_peered":"2024-04-26T23:59:24.368654+0000","last_clean":"2024-04-26T23:59:24.368654+0000","last_became_active":"2024-04-26T23:58:50.120396+0000","last_became_peered":"2024-04-26T23:58:50.120396+0000","last_unstale":"2024-04-26T23:59:24.368654+0000","last_undegraded":"2024-04-26T23:59:24.368654+0000","last_fullsized":"2024-04-26T23:59:24.368654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:16.670482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248035+0000","last_change":"2024-04-26T23:58:51.134917+0000","last_active":"2024-04-26T23:58:56.248035+0000","last_peered":"2024-04-26T23:58:56.248035+0000","last_clean":"2024-04-26T23:58:56.248035+0000","last_became_active":"2024-04-26T23:58:51.134470+0000","last_became_peered":"2024-04-26T23:58:51.134470+0000","last_unstale":"2024-04-26T23:58:56.248035+0000","last_undegraded":"2024-04-26T23:58:56.248035+0000","last_fullsized":"2024-04-26T23:58:56.248035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:03.696427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250223+0000","last_change":"2024-04-26T23:58:55.213966+0000","last_active":"2024-04-26T23:58:56.250223+0000","last_peered":"2024-04-26T23:58:56.250223+0000","last_clean":"2024-04-26T23:58:56.250223+0000","last_became_active":"2024-04-26T23:58:55.213521+0000","last_became_peered":"2024-04-26T23:58:55.213521+0000","last_unstale":"2024-04-26T23:58:56.250223+0000","last_undegraded":"2024-04-26T23:58:56.250223+0000","last_fullsized":"2024-04-26T23:58:56.250223+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:10:02.998037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.209433+0000","last_change":"2024-04-26T23:58:50.118136+0000","last_active":"2024-04-26T23:58:56.209433+0000","last_peered":"2024-04-26T23:58:56.209433+0000","last_clean":"2024-04-26T23:58:56.209433+0000","last_became_active":"2024-04-26T23:58:50.117787+0000","last_became_peered":"2024-04-26T23:58:50.117787+0000","last_unstale":"2024-04-26T23:58:56.209433+0000","last_undegraded":"2024-04-26T23:58:56.209433+0000","last_fullsized":"2024-04-26T23:58:56.209433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:14.145327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247712+0000","last_change":"2024-04-26T23:58:51.166491+0000","last_active":"2024-04-26T23:58:56.247712+0000","last_peered":"2024-04-26T23:58:56.247712+0000","last_clean":"2024-04-26T23:58:56.247712+0000","last_became_active":"2024-04-26T23:58:51.166106+0000","last_became_peered":"2024-04-26T23:58:51.166106+0000","last_unstale":"2024-04-26T23:58:56.247712+0000","last_undegraded":"2024-04-26T23:58:56.247712+0000","last_fullsized":"2024-04-26T23:58:56.247712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:18:58.690731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208576+0000","last_change":"2024-04-26T23:58:55.184623+0000","last_active":"2024-04-26T23:58:56.208576+0000","last_peered":"2024-04-26T23:58:56.208576+0000","last_clean":"2024-04-26T23:58:56.208576+0000","last_became_active":"2024-04-26T23:58:55.184029+0000","last_became_peered":"2024-04-26T23:58:55.184029+0000","last_unstale":"2024-04-26T23:58:56.208576+0000","last_undegraded":"2024-04-26T23:58:56.208576+0000","last_fullsized":"2024-04-26T23:58:56.208576+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:39:36.267352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.251842+0000","last_change":"2024-04-26T23:58:50.127630+0000","last_active":"2024-04-26T23:58:56.251842+0000","last_peered":"2024-04-26T23:58:56.251842+0000","last_clean":"2024-04-26T23:58:56.251842+0000","last_became_active":"2024-04-26T23:58:50.127134+0000","last_became_peered":"2024-04-26T23:58:50.127134+0000","last_unstale":"2024-04-26T23:58:56.251842+0000","last_undegraded":"2024-04-26T23:58:56.251842+0000","last_fullsized":"2024-04-26T23:58:56.251842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:15.061919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248456+0000","last_change":"2024-04-26T23:58:51.134241+0000","last_active":"2024-04-26T23:58:56.248456+0000","last_peered":"2024-04-26T23:58:56.248456+0000","last_clean":"2024-04-26T23:58:56.248456+0000","last_became_active":"2024-04-26T23:58:51.133867+0000","last_became_peered":"2024-04-26T23:58:51.133867+0000","last_unstale":"2024-04-26T23:58:56.248456+0000","last_undegraded":"2024-04-26T23:58:56.248456+0000","last_fullsized":"2024-04-26T23:58:56.248456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:12:20.171560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247719+0000","last_change":"2024-04-26T23:58:51.132681+0000","last_active":"2024-04-26T23:58:56.247719+0000","last_peered":"2024-04-26T23:58:56.247719+0000","last_clean":"2024-04-26T23:58:56.247719+0000","last_became_active":"2024-04-26T23:58:51.132213+0000","last_became_peered":"2024-04-26T23:58:51.132213+0000","last_unstale":"2024-04-26T23:58:56.247719+0000","last_undegraded":"2024-04-26T23:58:56.247719+0000","last_fullsized":"2024-04-26T23:58:56.247719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:26:52.080670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246905+0000","last_change":"2024-04-26T23:58:51.135731+0000","last_active":"2024-04-26T23:58:56.246905+0000","last_peered":"2024-04-26T23:58:56.246905+0000","last_clean":"2024-04-26T23:58:56.246905+0000","last_became_active":"2024-04-26T23:58:51.135352+0000","last_became_peered":"2024-04-26T23:58:51.135352+0000","last_unstale":"2024-04-26T23:58:56.246905+0000","last_undegraded":"2024-04-26T23:58:56.246905+0000","last_fullsized":"2024-04-26T23:58:56.246905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:45:37.903396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246573+0000","last_change":"2024-04-26T23:58:51.160014+0000","last_active":"2024-04-26T23:58:56.246573+0000","last_peered":"2024-04-26T23:58:56.246573+0000","last_clean":"2024-04-26T23:58:56.246573+0000","last_became_active":"2024-04-26T23:58:51.159459+0000","last_became_peered":"2024-04-26T23:58:51.159459+0000","last_unstale":"2024-04-26T23:58:56.246573+0000","last_undegraded":"2024-04-26T23:58:56.246573+0000","last_fullsized":"2024-04-26T23:58:56.246573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:12.757558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248891+0000","last_change":"2024-04-26T23:58:51.166942+0000","last_active":"2024-04-26T23:58:56.248891+0000","last_peered":"2024-04-26T23:58:56.248891+0000","last_clean":"2024-04-26T23:58:56.248891+0000","last_became_active":"2024-04-26T23:58:51.162605+0000","last_became_peered":"2024-04-26T23:58:51.162605+0000","last_unstale":"2024-04-26T23:58:56.248891+0000","last_undegraded":"2024-04-26T23:58:56.248891+0000","last_fullsized":"2024-04-26T23:58:56.248891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:38:46.346098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.252080+0000","last_change":"2024-04-26T23:58:51.162638+0000","last_active":"2024-04-26T23:58:56.252080+0000","last_peered":"2024-04-26T23:58:56.252080+0000","last_clean":"2024-04-26T23:58:56.252080+0000","last_became_active":"2024-04-26T23:58:51.162165+0000","last_became_peered":"2024-04-26T23:58:51.162165+0000","last_unstale":"2024-04-26T23:58:56.252080+0000","last_undegraded":"2024-04-26T23:58:56.252080+0000","last_fullsized":"2024-04-26T23:58:56.252080+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:26:32.277183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247961+0000","last_change":"2024-04-26T23:58:51.132667+0000","last_active":"2024-04-26T23:58:56.247961+0000","last_peered":"2024-04-26T23:58:56.247961+0000","last_clean":"2024-04-26T23:58:56.247961+0000","last_became_active":"2024-04-26T23:58:51.132316+0000","last_became_peered":"2024-04-26T23:58:51.132316+0000","last_unstale":"2024-04-26T23:58:56.247961+0000","last_undegraded":"2024-04-26T23:58:56.247961+0000","last_fullsized":"2024-04-26T23:58:56.247961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:41.427650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247285+0000","last_change":"2024-04-26T23:58:51.132614+0000","last_active":"2024-04-26T23:58:56.247285+0000","last_peered":"2024-04-26T23:58:56.247285+0000","last_clean":"2024-04-26T23:58:56.247285+0000","last_became_active":"2024-04-26T23:58:51.132274+0000","last_became_peered":"2024-04-26T23:58:51.132274+0000","last_unstale":"2024-04-26T23:58:56.247285+0000","last_undegraded":"2024-04-26T23:58:56.247285+0000","last_fullsized":"2024-04-26T23:58:56.247285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:52.810401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.213133+0000","last_change":"2024-04-26T23:58:51.165470+0000","last_active":"2024-04-26T23:58:56.213133+0000","last_peered":"2024-04-26T23:58:56.213133+0000","last_clean":"2024-04-26T23:58:56.213133+0000","last_became_active":"2024-04-26T23:58:51.164998+0000","last_became_peered":"2024-04-26T23:58:51.164998+0000","last_unstale":"2024-04-26T23:58:56.213133+0000","last_undegraded":"2024-04-26T23:58:56.213133+0000","last_fullsized":"2024-04-26T23:58:56.213133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:02.703215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208735+0000","last_change":"2024-04-26T23:58:51.166390+0000","last_active":"2024-04-26T23:58:56.208735+0000","last_peered":"2024-04-26T23:58:56.208735+0000","last_clean":"2024-04-26T23:58:56.208735+0000","last_became_active":"2024-04-26T23:58:51.166113+0000","last_became_peered":"2024-04-26T23:58:51.166113+0000","last_unstale":"2024-04-26T23:58:56.208735+0000","last_undegraded":"2024-04-26T23:58:56.208735+0000","last_fullsized":"2024-04-26T23:58:56.208735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:24:49.676207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211288+0000","last_change":"2024-04-26T23:58:51.165272+0000","last_active":"2024-04-26T23:58:56.211288+0000","last_peered":"2024-04-26T23:58:56.211288+0000","last_clean":"2024-04-26T23:58:56.211288+0000","last_became_active":"2024-04-26T23:58:51.160355+0000","last_became_peered":"2024-04-26T23:58:51.160355+0000","last_unstale":"2024-04-26T23:58:56.211288+0000","last_undegraded":"2024-04-26T23:58:56.211288+0000","last_fullsized":"2024-04-26T23:58:56.211288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:32.910129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246420+0000","last_change":"2024-04-26T23:58:51.159564+0000","last_active":"2024-04-26T23:58:56.246420+0000","last_peered":"2024-04-26T23:58:56.246420+0000","last_clean":"2024-04-26T23:58:56.246420+0000","last_became_active":"2024-04-26T23:58:51.159229+0000","last_became_peered":"2024-04-26T23:58:51.159229+0000","last_unstale":"2024-04-26T23:58:56.246420+0000","last_undegraded":"2024-04-26T23:58:56.246420+0000","last_fullsized":"2024-04-26T23:58:56.246420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:39.032595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.251450+0000","last_change":"2024-04-26T23:58:51.163868+0000","last_active":"2024-04-26T23:58:56.251450+0000","last_peered":"2024-04-26T23:58:56.251450+0000","last_clean":"2024-04-26T23:58:56.251450+0000","last_became_active":"2024-04-26T23:58:51.163450+0000","last_became_peered":"2024-04-26T23:58:51.163450+0000","last_unstale":"2024-04-26T23:58:56.251450+0000","last_undegraded":"2024-04-26T23:58:56.251450+0000","last_fullsized":"2024-04-26T23:58:56.251450+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:42:06.945605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247845+0000","last_change":"2024-04-26T23:58:51.136047+0000","last_active":"2024-04-26T23:58:56.247845+0000","last_peered":"2024-04-26T23:58:56.247845+0000","last_clean":"2024-04-26T23:58:56.247845+0000","last_became_active":"2024-04-26T23:58:51.135690+0000","last_became_peered":"2024-04-26T23:58:51.135690+0000","last_unstale":"2024-04-26T23:58:56.247845+0000","last_undegraded":"2024-04-26T23:58:56.247845+0000","last_fullsized":"2024-04-26T23:58:56.247845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:55.721858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14443,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":28,"num_read_kb":27,"num_write":134,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":75575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350997,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":484944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612633,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1630208,"data_stored":1076054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776159,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":481084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037799,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":485100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299435,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1605632,"data_stored":1077304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462962,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":480356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":518073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":481744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":492449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":480460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888126,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":480910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724599,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1072110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":41906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":12183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:59:29.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:59:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.668+0000 7f1e3e800640 1 -- 172.21.15.40:0/4248247293 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e38105b20 msgr2=0x7f1e38107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.668+0000 7f1e3e800640 1 --2- 172.21.15.40:0/4248247293 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e38105b20 0x7f1e38107f10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009900 tx=0x7f1e2402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.668+0000 7f1e3e800640 1 -- 172.21.15.40:0/4248247293 shutdown_connections 2024-04-26T23:59:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.668+0000 7f1e3e800640 1 --2- 172.21.15.40:0/4248247293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e38108450 0x7f1e3810a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.668+0000 7f1e3e800640 1 --2- 172.21.15.40:0/4248247293 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e38105b20 0x7f1e38107f10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.668+0000 7f1e3e800640 1 --2- 172.21.15.40:0/4248247293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e381031f0 0x7f1e381055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.668+0000 7f1e3e800640 1 -- 172.21.15.40:0/4248247293 >> 172.21.15.40:0/4248247293 conn(0x7f1e380fcbc0 msgr2=0x7f1e380ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.668+0000 7f1e3e800640 1 -- 172.21.15.40:0/4248247293 shutdown_connections 2024-04-26T23:59:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.668+0000 7f1e3e800640 1 -- 172.21.15.40:0/4248247293 wait complete. 2024-04-26T23:59:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.669+0000 7f1e3e800640 1 Processor -- start 2024-04-26T23:59:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.669+0000 7f1e3e800640 1 -- start start 2024-04-26T23:59:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.670+0000 7f1e3e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e381031f0 0x7f1e3819abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.670+0000 7f1e3e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e38105b20 0x7f1e3819b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.670+0000 7f1e3e800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e38108450 0x7f1e381a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.670+0000 7f1e3e800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3810e300 con 0x7f1e381031f0 2024-04-26T23:59:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.670+0000 7f1e3e800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3810e180 con 0x7f1e38108450 2024-04-26T23:59:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.670+0000 7f1e3e800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3810e480 con 0x7f1e38105b20 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.670+0000 7f1e3d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e381031f0 0x7f1e3819abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.670+0000 7f1e3d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e381031f0 0x7f1e3819abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49540/0 (socket says 172.21.15.40:49540) 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.670+0000 7f1e3d400640 1 -- 172.21.15.40:0/1173814881 learned_addr learned my addr 172.21.15.40:0/1173814881 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.670+0000 7f1e3de00640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e38108450 0x7f1e381a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.670+0000 7f1e3ca00640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e38105b20 0x7f1e3819b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.671+0000 7f1e3de00640 1 -- 172.21.15.40:0/1173814881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e38105b20 msgr2=0x7f1e3819b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.671+0000 7f1e3de00640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e38105b20 0x7f1e3819b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.671+0000 7f1e3de00640 1 -- 172.21.15.40:0/1173814881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e381031f0 msgr2=0x7f1e3819abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.671+0000 7f1e3de00640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e381031f0 0x7f1e3819abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.671+0000 7f1e3de00640 1 -- 172.21.15.40:0/1173814881 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e381a2800 con 0x7f1e38108450 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.671+0000 7f1e3d400640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e381031f0 0x7f1e3819abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.671+0000 7f1e3de00640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e38108450 0x7f1e381a2190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000e970 tx=0x7f1e3000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.671+0000 7f1e36000640 1 -- 172.21.15.40:0/1173814881 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3000ccc0 con 0x7f1e38108450 2024-04-26T23:59:29.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.671+0000 7f1e36000640 1 -- 172.21.15.40:0/1173814881 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e30005020 con 0x7f1e38108450 2024-04-26T23:59:29.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.671+0000 7f1e3e800640 1 -- 172.21.15.40:0/1173814881 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e381a2af0 con 0x7f1e38108450 2024-04-26T23:59:29.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.671+0000 7f1e36000640 1 -- 172.21.15.40:0/1173814881 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e30004a00 con 0x7f1e38108450 2024-04-26T23:59:29.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.671+0000 7f1e3e800640 1 -- 172.21.15.40:0/1173814881 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e381a2fc0 con 0x7f1e38108450 2024-04-26T23:59:29.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.673+0000 7f1e3e800640 1 -- 172.21.15.40:0/1173814881 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e38109fa0 con 0x7f1e38108450 2024-04-26T23:59:29.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.674+0000 7f1e36000640 1 -- 172.21.15.40:0/1173814881 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1e30004ba0 con 0x7f1e38108450 2024-04-26T23:59:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.676+0000 7f1e36000640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1e1c0a3c40 0x7f1e1c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.676+0000 7f1e36000640 1 -- 172.21.15.40:0/1173814881 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f1e300d0770 con 0x7f1e38108450 2024-04-26T23:59:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.676+0000 7f1e3d400640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1e1c0a3c40 0x7f1e1c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.677+0000 7f1e3d400640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1e1c0a3c40 0x7f1e1c0a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1e280049d0 tx=0x7f1e28009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:29.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.680+0000 7f1e36000640 1 -- 172.21.15.40:0/1173814881 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1e3009cd50 con 0x7f1e38108450 2024-04-26T23:59:29.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:29 smithi040 ceph-mon[28095]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:29.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:29 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/946349469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:59:29.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:29 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/4108997797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:29.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:29 smithi080 ceph-mon[32362]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:29.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:29 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/946349469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:59:29.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:29 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/4108997797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:29.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:29 smithi103 ceph-mon[32952]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:59:29.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:29 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/946349469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:59:29.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:29 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/4108997797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:29.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.885+0000 7f1e3e800640 1 -- 172.21.15.40:0/1173814881 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1e38062d70 con 0x7f1e1c0a3c40 2024-04-26T23:59:29.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.915+0000 7f1e36000640 1 -- 172.21.15.40:0/1173814881 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340361 (secure 0 0 0) 0x7f1e38062d70 con 0x7f1e1c0a3c40 2024-04-26T23:59:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:59:29.923 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-26T23:59:29.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.921+0000 7f1e3e800640 1 -- 172.21.15.40:0/1173814881 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1e1c0a3c40 msgr2=0x7f1e1c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:29.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.921+0000 7f1e3e800640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1e1c0a3c40 0x7f1e1c0a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1e280049d0 tx=0x7f1e28009500 comp rx=0 tx=0).stop 2024-04-26T23:59:29.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.921+0000 7f1e3e800640 1 -- 172.21.15.40:0/1173814881 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e38108450 msgr2=0x7f1e381a2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:29.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.921+0000 7f1e3e800640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e38108450 0x7f1e381a2190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000e970 tx=0x7f1e3000ee30 comp rx=0 tx=0).stop 2024-04-26T23:59:29.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.921+0000 7f1e3e800640 1 -- 172.21.15.40:0/1173814881 shutdown_connections 2024-04-26T23:59:29.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.921+0000 7f1e3e800640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1e38108450 0x7f1e381a2190 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.921+0000 7f1e3e800640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1e1c0a3c40 0x7f1e1c0a6100 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.921+0000 7f1e3e800640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e38105b20 0x7f1e3819b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.921+0000 7f1e3e800640 1 --2- 172.21.15.40:0/1173814881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e381031f0 0x7f1e3819abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:29.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.921+0000 7f1e3e800640 1 -- 172.21.15.40:0/1173814881 >> 172.21.15.40:0/1173814881 conn(0x7f1e380fcbc0 msgr2=0x7f1e38102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:29.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.921+0000 7f1e3e800640 1 -- 172.21.15.40:0/1173814881 shutdown_connections 2024-04-26T23:59:29.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:29.921+0000 7f1e3e800640 1 -- 172.21.15.40:0/1173814881 wait complete. 2024-04-26T23:59:29.944 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-04-26T23:59:28.712858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":604928,"num_objects":63,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":154,"num_read_kb":136,"num_write":316,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":256,"ondisk_log_size":256,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371120,"kb_used_data":13648,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570704,"statfs":{"total":1151940427776,"available":1151560400896,"internally_reserved":0,"allocated":13975552,"data_stored":7610588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9920,"num_objects":39,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":28,"num_read_kb":27,"num_write":86,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006745"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248269+0000","last_change":"2024-04-26T23:58:51.162452+0000","last_active":"2024-04-26T23:58:56.248269+0000","last_peered":"2024-04-26T23:58:56.248269+0000","last_clean":"2024-04-26T23:58:56.248269+0000","last_became_active":"2024-04-26T23:58:51.162065+0000","last_became_peered":"2024-04-26T23:58:51.162065+0000","last_unstale":"2024-04-26T23:58:56.248269+0000","last_undegraded":"2024-04-26T23:58:56.248269+0000","last_fullsized":"2024-04-26T23:58:56.248269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:58:05.949099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248836+0000","last_change":"2024-04-26T23:58:51.161144+0000","last_active":"2024-04-26T23:58:56.248836+0000","last_peered":"2024-04-26T23:58:56.248836+0000","last_clean":"2024-04-26T23:58:56.248836+0000","last_became_active":"2024-04-26T23:58:51.154033+0000","last_became_peered":"2024-04-26T23:58:51.154033+0000","last_unstale":"2024-04-26T23:58:56.248836+0000","last_undegraded":"2024-04-26T23:58:56.248836+0000","last_fullsized":"2024-04-26T23:58:56.248836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:37:39.757654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247575+0000","last_change":"2024-04-26T23:58:51.159715+0000","last_active":"2024-04-26T23:58:56.247575+0000","last_peered":"2024-04-26T23:58:56.247575+0000","last_clean":"2024-04-26T23:58:56.247575+0000","last_became_active":"2024-04-26T23:58:51.152657+0000","last_became_peered":"2024-04-26T23:58:51.152657+0000","last_unstale":"2024-04-26T23:58:56.247575+0000","last_undegraded":"2024-04-26T23:58:56.247575+0000","last_fullsized":"2024-04-26T23:58:56.247575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:29.210295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249104+0000","last_change":"2024-04-26T23:58:51.141814+0000","last_active":"2024-04-26T23:58:56.249104+0000","last_peered":"2024-04-26T23:58:56.249104+0000","last_clean":"2024-04-26T23:58:56.249104+0000","last_became_active":"2024-04-26T23:58:51.141344+0000","last_became_peered":"2024-04-26T23:58:51.141344+0000","last_unstale":"2024-04-26T23:58:56.249104+0000","last_undegraded":"2024-04-26T23:58:56.249104+0000","last_fullsized":"2024-04-26T23:58:56.249104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:05:49.532178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.377427+0000","last_change":"2024-04-26T23:58:51.170537+0000","last_active":"2024-04-26T23:58:56.377427+0000","last_peered":"2024-04-26T23:58:56.377427+0000","last_clean":"2024-04-26T23:58:56.377427+0000","last_became_active":"2024-04-26T23:58:51.170047+0000","last_became_peered":"2024-04-26T23:58:51.170047+0000","last_unstale":"2024-04-26T23:58:56.377427+0000","last_undegraded":"2024-04-26T23:58:56.377427+0000","last_fullsized":"2024-04-26T23:58:56.377427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:46.463214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247811+0000","last_change":"2024-04-26T23:58:51.141041+0000","last_active":"2024-04-26T23:58:56.247811+0000","last_peered":"2024-04-26T23:58:56.247811+0000","last_clean":"2024-04-26T23:58:56.247811+0000","last_became_active":"2024-04-26T23:58:51.140709+0000","last_became_peered":"2024-04-26T23:58:51.140709+0000","last_unstale":"2024-04-26T23:58:56.247811+0000","last_undegraded":"2024-04-26T23:58:56.247811+0000","last_fullsized":"2024-04-26T23:58:56.247811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:12:28.613723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247409+0000","last_change":"2024-04-26T23:58:51.145975+0000","last_active":"2024-04-26T23:58:56.247409+0000","last_peered":"2024-04-26T23:58:56.247409+0000","last_clean":"2024-04-26T23:58:56.247409+0000","last_became_active":"2024-04-26T23:58:51.145594+0000","last_became_peered":"2024-04-26T23:58:51.145594+0000","last_unstale":"2024-04-26T23:58:56.247409+0000","last_undegraded":"2024-04-26T23:58:56.247409+0000","last_fullsized":"2024-04-26T23:58:56.247409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:50:59.548009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211401+0000","last_change":"2024-04-26T23:58:51.160363+0000","last_active":"2024-04-26T23:58:56.211401+0000","last_peered":"2024-04-26T23:58:56.211401+0000","last_clean":"2024-04-26T23:58:56.211401+0000","last_became_active":"2024-04-26T23:58:51.152004+0000","last_became_peered":"2024-04-26T23:58:51.152004+0000","last_unstale":"2024-04-26T23:58:56.211401+0000","last_undegraded":"2024-04-26T23:58:56.211401+0000","last_fullsized":"2024-04-26T23:58:56.211401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:01:03.203404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247147+0000","last_change":"2024-04-26T23:58:51.134247+0000","last_active":"2024-04-26T23:58:56.247147+0000","last_peered":"2024-04-26T23:58:56.247147+0000","last_clean":"2024-04-26T23:58:56.247147+0000","last_became_active":"2024-04-26T23:58:51.133889+0000","last_became_peered":"2024-04-26T23:58:51.133889+0000","last_unstale":"2024-04-26T23:58:56.247147+0000","last_undegraded":"2024-04-26T23:58:56.247147+0000","last_fullsized":"2024-04-26T23:58:56.247147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:51:04.245217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248382+0000","last_change":"2024-04-26T23:58:51.135255+0000","last_active":"2024-04-26T23:58:56.248382+0000","last_peered":"2024-04-26T23:58:56.248382+0000","last_clean":"2024-04-26T23:58:56.248382+0000","last_became_active":"2024-04-26T23:58:51.134905+0000","last_became_peered":"2024-04-26T23:58:51.134905+0000","last_unstale":"2024-04-26T23:58:56.248382+0000","last_undegraded":"2024-04-26T23:58:56.248382+0000","last_fullsized":"2024-04-26T23:58:56.248382+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:51.435008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247933+0000","last_change":"2024-04-26T23:58:51.134962+0000","last_active":"2024-04-26T23:58:56.247933+0000","last_peered":"2024-04-26T23:58:56.247933+0000","last_clean":"2024-04-26T23:58:56.247933+0000","last_became_active":"2024-04-26T23:58:51.134559+0000","last_became_peered":"2024-04-26T23:58:51.134559+0000","last_unstale":"2024-04-26T23:58:56.247933+0000","last_undegraded":"2024-04-26T23:58:56.247933+0000","last_fullsized":"2024-04-26T23:58:56.247933+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:31:15.833585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249016+0000","last_change":"2024-04-26T23:58:51.141872+0000","last_active":"2024-04-26T23:58:56.249016+0000","last_peered":"2024-04-26T23:58:56.249016+0000","last_clean":"2024-04-26T23:58:56.249016+0000","last_became_active":"2024-04-26T23:58:51.141464+0000","last_became_peered":"2024-04-26T23:58:51.141464+0000","last_unstale":"2024-04-26T23:58:56.249016+0000","last_undegraded":"2024-04-26T23:58:56.249016+0000","last_fullsized":"2024-04-26T23:58:56.249016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:37.663379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.251986+0000","last_change":"2024-04-26T23:58:51.157828+0000","last_active":"2024-04-26T23:58:56.251986+0000","last_peered":"2024-04-26T23:58:56.251986+0000","last_clean":"2024-04-26T23:58:56.251986+0000","last_became_active":"2024-04-26T23:58:51.156820+0000","last_became_peered":"2024-04-26T23:58:51.156820+0000","last_unstale":"2024-04-26T23:58:56.251986+0000","last_undegraded":"2024-04-26T23:58:56.251986+0000","last_fullsized":"2024-04-26T23:58:56.251986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:22:43.201795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248034+0000","last_change":"2024-04-26T23:58:51.154108+0000","last_active":"2024-04-26T23:58:56.248034+0000","last_peered":"2024-04-26T23:58:56.248034+0000","last_clean":"2024-04-26T23:58:56.248034+0000","last_became_active":"2024-04-26T23:58:51.149740+0000","last_became_peered":"2024-04-26T23:58:51.149740+0000","last_unstale":"2024-04-26T23:58:56.248034+0000","last_undegraded":"2024-04-26T23:58:56.248034+0000","last_fullsized":"2024-04-26T23:58:56.248034+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:23:58.519297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247376+0000","last_change":"2024-04-26T23:58:51.164578+0000","last_active":"2024-04-26T23:58:56.247376+0000","last_peered":"2024-04-26T23:58:56.247376+0000","last_clean":"2024-04-26T23:58:56.247376+0000","last_became_active":"2024-04-26T23:58:51.152843+0000","last_became_peered":"2024-04-26T23:58:51.152843+0000","last_unstale":"2024-04-26T23:58:56.247376+0000","last_undegraded":"2024-04-26T23:58:56.247376+0000","last_fullsized":"2024-04-26T23:58:56.247376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:02.788917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.245726+0000","last_change":"2024-04-26T23:58:51.162811+0000","last_active":"2024-04-26T23:58:56.245726+0000","last_peered":"2024-04-26T23:58:56.245726+0000","last_clean":"2024-04-26T23:58:56.245726+0000","last_became_active":"2024-04-26T23:58:51.162464+0000","last_became_peered":"2024-04-26T23:58:51.162464+0000","last_unstale":"2024-04-26T23:58:56.245726+0000","last_undegraded":"2024-04-26T23:58:56.245726+0000","last_fullsized":"2024-04-26T23:58:56.245726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:50:26.127894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248127+0000","last_change":"2024-04-26T23:58:51.162363+0000","last_active":"2024-04-26T23:58:56.248127+0000","last_peered":"2024-04-26T23:58:56.248127+0000","last_clean":"2024-04-26T23:58:56.248127+0000","last_became_active":"2024-04-26T23:58:51.162033+0000","last_became_peered":"2024-04-26T23:58:51.162033+0000","last_unstale":"2024-04-26T23:58:56.248127+0000","last_undegraded":"2024-04-26T23:58:56.248127+0000","last_fullsized":"2024-04-26T23:58:56.248127+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:53.755338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250657+0000","last_change":"2024-04-26T23:58:51.147373+0000","last_active":"2024-04-26T23:58:56.250657+0000","last_peered":"2024-04-26T23:58:56.250657+0000","last_clean":"2024-04-26T23:58:56.250657+0000","last_became_active":"2024-04-26T23:58:51.146993+0000","last_became_peered":"2024-04-26T23:58:51.146993+0000","last_unstale":"2024-04-26T23:58:56.250657+0000","last_undegraded":"2024-04-26T23:58:56.250657+0000","last_fullsized":"2024-04-26T23:58:56.250657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:14.887719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247161+0000","last_change":"2024-04-26T23:58:51.159553+0000","last_active":"2024-04-26T23:58:56.247161+0000","last_peered":"2024-04-26T23:58:56.247161+0000","last_clean":"2024-04-26T23:58:56.247161+0000","last_became_active":"2024-04-26T23:58:51.159252+0000","last_became_peered":"2024-04-26T23:58:51.159252+0000","last_unstale":"2024-04-26T23:58:56.247161+0000","last_undegraded":"2024-04-26T23:58:56.247161+0000","last_fullsized":"2024-04-26T23:58:56.247161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:17.408273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246856+0000","last_change":"2024-04-26T23:58:55.203263+0000","last_active":"2024-04-26T23:58:56.246856+0000","last_peered":"2024-04-26T23:58:56.246856+0000","last_clean":"2024-04-26T23:58:56.246856+0000","last_became_active":"2024-04-26T23:58:55.202832+0000","last_became_peered":"2024-04-26T23:58:55.202832+0000","last_unstale":"2024-04-26T23:58:56.246856+0000","last_undegraded":"2024-04-26T23:58:56.246856+0000","last_fullsized":"2024-04-26T23:58:56.246856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:13:38.825916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.371998+0000","last_change":"2024-04-26T23:58:50.124003+0000","last_active":"2024-04-26T23:59:24.371998+0000","last_peered":"2024-04-26T23:59:24.371998+0000","last_clean":"2024-04-26T23:59:24.371998+0000","last_became_active":"2024-04-26T23:58:50.123653+0000","last_became_peered":"2024-04-26T23:58:50.123653+0000","last_unstale":"2024-04-26T23:59:24.371998+0000","last_undegraded":"2024-04-26T23:59:24.371998+0000","last_fullsized":"2024-04-26T23:59:24.371998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:16:57.688594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.212780+0000","last_change":"2024-04-26T23:58:51.175946+0000","last_active":"2024-04-26T23:58:56.212780+0000","last_peered":"2024-04-26T23:58:56.212780+0000","last_clean":"2024-04-26T23:58:56.212780+0000","last_became_active":"2024-04-26T23:58:51.174204+0000","last_became_peered":"2024-04-26T23:58:51.174204+0000","last_unstale":"2024-04-26T23:58:56.212780+0000","last_undegraded":"2024-04-26T23:58:56.212780+0000","last_fullsized":"2024-04-26T23:58:56.212780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:03.975717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248808+0000","last_change":"2024-04-26T23:58:55.204690+0000","last_active":"2024-04-26T23:58:56.248808+0000","last_peered":"2024-04-26T23:58:56.248808+0000","last_clean":"2024-04-26T23:58:56.248808+0000","last_became_active":"2024-04-26T23:58:55.204248+0000","last_became_peered":"2024-04-26T23:58:55.204248+0000","last_unstale":"2024-04-26T23:58:56.248808+0000","last_undegraded":"2024-04-26T23:58:56.248808+0000","last_fullsized":"2024-04-26T23:58:56.248808+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:21.009534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.213176+0000","last_change":"2024-04-26T23:58:50.128773+0000","last_active":"2024-04-26T23:58:56.213176+0000","last_peered":"2024-04-26T23:58:56.213176+0000","last_clean":"2024-04-26T23:58:56.213176+0000","last_became_active":"2024-04-26T23:58:50.128445+0000","last_became_peered":"2024-04-26T23:58:50.128445+0000","last_unstale":"2024-04-26T23:58:56.213176+0000","last_undegraded":"2024-04-26T23:58:56.213176+0000","last_fullsized":"2024-04-26T23:58:56.213176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:27:57.577361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248330+0000","last_change":"2024-04-26T23:58:51.164079+0000","last_active":"2024-04-26T23:58:56.248330+0000","last_peered":"2024-04-26T23:58:56.248330+0000","last_clean":"2024-04-26T23:58:56.248330+0000","last_became_active":"2024-04-26T23:58:51.162514+0000","last_became_peered":"2024-04-26T23:58:51.162514+0000","last_unstale":"2024-04-26T23:58:56.248330+0000","last_undegraded":"2024-04-26T23:58:56.248330+0000","last_fullsized":"2024-04-26T23:58:56.248330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:48:10.904737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249297+0000","last_change":"2024-04-26T23:58:55.211879+0000","last_active":"2024-04-26T23:58:56.249297+0000","last_peered":"2024-04-26T23:58:56.249297+0000","last_clean":"2024-04-26T23:58:56.249297+0000","last_became_active":"2024-04-26T23:58:55.211421+0000","last_became_peered":"2024-04-26T23:58:55.211421+0000","last_unstale":"2024-04-26T23:58:56.249297+0000","last_undegraded":"2024-04-26T23:58:56.249297+0000","last_fullsized":"2024-04-26T23:58:56.249297+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:45:12.754349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.362322+0000","last_change":"2024-04-26T23:58:50.101893+0000","last_active":"2024-04-26T23:59:24.362322+0000","last_peered":"2024-04-26T23:59:24.362322+0000","last_clean":"2024-04-26T23:59:24.362322+0000","last_became_active":"2024-04-26T23:58:50.101625+0000","last_became_peered":"2024-04-26T23:58:50.101625+0000","last_unstale":"2024-04-26T23:59:24.362322+0000","last_undegraded":"2024-04-26T23:59:24.362322+0000","last_fullsized":"2024-04-26T23:59:24.362322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:05:24.365710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246542+0000","last_change":"2024-04-26T23:58:51.146848+0000","last_active":"2024-04-26T23:58:56.246542+0000","last_peered":"2024-04-26T23:58:56.246542+0000","last_clean":"2024-04-26T23:58:56.246542+0000","last_became_active":"2024-04-26T23:58:51.146261+0000","last_became_peered":"2024-04-26T23:58:51.146261+0000","last_unstale":"2024-04-26T23:58:56.246542+0000","last_undegraded":"2024-04-26T23:58:56.246542+0000","last_fullsized":"2024-04-26T23:58:56.246542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:22.200191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248349+0000","last_change":"2024-04-26T23:58:55.196026+0000","last_active":"2024-04-26T23:58:56.248349+0000","last_peered":"2024-04-26T23:58:56.248349+0000","last_clean":"2024-04-26T23:58:56.248349+0000","last_became_active":"2024-04-26T23:58:55.195591+0000","last_became_peered":"2024-04-26T23:58:55.195591+0000","last_unstale":"2024-04-26T23:58:56.248349+0000","last_undegraded":"2024-04-26T23:58:56.248349+0000","last_fullsized":"2024-04-26T23:58:56.248349+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:06.980985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.377471+0000","last_change":"2024-04-26T23:58:50.100655+0000","last_active":"2024-04-26T23:58:56.377471+0000","last_peered":"2024-04-26T23:58:56.377471+0000","last_clean":"2024-04-26T23:58:56.377471+0000","last_became_active":"2024-04-26T23:58:50.100314+0000","last_became_peered":"2024-04-26T23:58:50.100314+0000","last_unstale":"2024-04-26T23:58:56.377471+0000","last_undegraded":"2024-04-26T23:58:56.377471+0000","last_fullsized":"2024-04-26T23:58:56.377471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:10:06.230717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.209327+0000","last_change":"2024-04-26T23:58:51.142669+0000","last_active":"2024-04-26T23:58:56.209327+0000","last_peered":"2024-04-26T23:58:56.209327+0000","last_clean":"2024-04-26T23:58:56.209327+0000","last_became_active":"2024-04-26T23:58:51.142337+0000","last_became_peered":"2024-04-26T23:58:51.142337+0000","last_unstale":"2024-04-26T23:58:56.209327+0000","last_undegraded":"2024-04-26T23:58:56.209327+0000","last_fullsized":"2024-04-26T23:58:56.209327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:33:57.351429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247207+0000","last_change":"2024-04-26T23:58:55.184331+0000","last_active":"2024-04-26T23:58:56.247207+0000","last_peered":"2024-04-26T23:58:56.247207+0000","last_clean":"2024-04-26T23:58:56.247207+0000","last_became_active":"2024-04-26T23:58:55.183928+0000","last_became_peered":"2024-04-26T23:58:55.183928+0000","last_unstale":"2024-04-26T23:58:56.247207+0000","last_undegraded":"2024-04-26T23:58:56.247207+0000","last_fullsized":"2024-04-26T23:58:56.247207+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:47.303104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:25.382919+0000","last_change":"2024-04-26T23:58:50.101084+0000","last_active":"2024-04-26T23:59:25.382919+0000","last_peered":"2024-04-26T23:59:25.382919+0000","last_clean":"2024-04-26T23:59:25.382919+0000","last_became_active":"2024-04-26T23:58:50.100736+0000","last_became_peered":"2024-04-26T23:58:50.100736+0000","last_unstale":"2024-04-26T23:59:25.382919+0000","last_undegraded":"2024-04-26T23:59:25.382919+0000","last_fullsized":"2024-04-26T23:59:25.382919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:43.157122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.377508+0000","last_change":"2024-04-26T23:58:51.143834+0000","last_active":"2024-04-26T23:58:56.377508+0000","last_peered":"2024-04-26T23:58:56.377508+0000","last_clean":"2024-04-26T23:58:56.377508+0000","last_became_active":"2024-04-26T23:58:51.143480+0000","last_became_peered":"2024-04-26T23:58:51.143480+0000","last_unstale":"2024-04-26T23:58:56.377508+0000","last_undegraded":"2024-04-26T23:58:56.377508+0000","last_fullsized":"2024-04-26T23:58:56.377508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:34:25.474562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247193+0000","last_change":"2024-04-26T23:58:55.215809+0000","last_active":"2024-04-26T23:58:56.247193+0000","last_peered":"2024-04-26T23:58:56.247193+0000","last_clean":"2024-04-26T23:58:56.247193+0000","last_became_active":"2024-04-26T23:58:55.215369+0000","last_became_peered":"2024-04-26T23:58:55.215369+0000","last_unstale":"2024-04-26T23:58:56.247193+0000","last_undegraded":"2024-04-26T23:58:56.247193+0000","last_fullsized":"2024-04-26T23:58:56.247193+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:04.053963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.364521+0000","last_change":"2024-04-26T23:58:50.129626+0000","last_active":"2024-04-26T23:59:24.364521+0000","last_peered":"2024-04-26T23:59:24.364521+0000","last_clean":"2024-04-26T23:59:24.364521+0000","last_became_active":"2024-04-26T23:58:50.129386+0000","last_became_peered":"2024-04-26T23:58:50.129386+0000","last_unstale":"2024-04-26T23:59:24.364521+0000","last_undegraded":"2024-04-26T23:59:24.364521+0000","last_fullsized":"2024-04-26T23:59:24.364521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:31:54.432289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.209192+0000","last_change":"2024-04-26T23:58:51.166312+0000","last_active":"2024-04-26T23:58:56.209192+0000","last_peered":"2024-04-26T23:58:56.209192+0000","last_clean":"2024-04-26T23:58:56.209192+0000","last_became_active":"2024-04-26T23:58:51.165986+0000","last_became_peered":"2024-04-26T23:58:51.165986+0000","last_unstale":"2024-04-26T23:58:56.209192+0000","last_undegraded":"2024-04-26T23:58:56.209192+0000","last_fullsized":"2024-04-26T23:58:56.209192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:51:49.549912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248210+0000","last_change":"2024-04-26T23:58:55.204142+0000","last_active":"2024-04-26T23:58:56.248210+0000","last_peered":"2024-04-26T23:58:56.248210+0000","last_clean":"2024-04-26T23:58:56.248210+0000","last_became_active":"2024-04-26T23:58:55.203646+0000","last_became_peered":"2024-04-26T23:58:55.203646+0000","last_unstale":"2024-04-26T23:58:56.248210+0000","last_undegraded":"2024-04-26T23:58:56.248210+0000","last_fullsized":"2024-04-26T23:58:56.248210+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:40.614034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.370945+0000","last_change":"2024-04-26T23:58:50.132080+0000","last_active":"2024-04-26T23:59:24.370945+0000","last_peered":"2024-04-26T23:59:24.370945+0000","last_clean":"2024-04-26T23:59:24.370945+0000","last_became_active":"2024-04-26T23:58:50.131618+0000","last_became_peered":"2024-04-26T23:58:50.131618+0000","last_unstale":"2024-04-26T23:59:24.370945+0000","last_undegraded":"2024-04-26T23:59:24.370945+0000","last_fullsized":"2024-04-26T23:59:24.370945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:42:07.897166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211042+0000","last_change":"2024-04-26T23:58:51.175083+0000","last_active":"2024-04-26T23:58:56.211042+0000","last_peered":"2024-04-26T23:58:56.211042+0000","last_clean":"2024-04-26T23:58:56.211042+0000","last_became_active":"2024-04-26T23:58:51.174159+0000","last_became_peered":"2024-04-26T23:58:51.174159+0000","last_unstale":"2024-04-26T23:58:56.211042+0000","last_undegraded":"2024-04-26T23:58:56.211042+0000","last_fullsized":"2024-04-26T23:58:56.211042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:51:06.583217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248664+0000","last_change":"2024-04-26T23:58:55.200019+0000","last_active":"2024-04-26T23:58:56.248664+0000","last_peered":"2024-04-26T23:58:56.248664+0000","last_clean":"2024-04-26T23:58:56.248664+0000","last_became_active":"2024-04-26T23:58:55.199593+0000","last_became_peered":"2024-04-26T23:58:55.199593+0000","last_unstale":"2024-04-26T23:58:56.248664+0000","last_undegraded":"2024-04-26T23:58:56.248664+0000","last_fullsized":"2024-04-26T23:58:56.248664+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:41.465294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208650+0000","last_change":"2024-04-26T23:58:50.100114+0000","last_active":"2024-04-26T23:58:56.208650+0000","last_peered":"2024-04-26T23:58:56.208650+0000","last_clean":"2024-04-26T23:58:56.208650+0000","last_became_active":"2024-04-26T23:58:50.099762+0000","last_became_peered":"2024-04-26T23:58:50.099762+0000","last_unstale":"2024-04-26T23:58:56.208650+0000","last_undegraded":"2024-04-26T23:58:56.208650+0000","last_fullsized":"2024-04-26T23:58:56.208650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:51:16.855014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249902+0000","last_change":"2024-04-26T23:58:51.171236+0000","last_active":"2024-04-26T23:58:56.249902+0000","last_peered":"2024-04-26T23:58:56.249902+0000","last_clean":"2024-04-26T23:58:56.249902+0000","last_became_active":"2024-04-26T23:58:51.170709+0000","last_became_peered":"2024-04-26T23:58:51.170709+0000","last_unstale":"2024-04-26T23:58:56.249902+0000","last_undegraded":"2024-04-26T23:58:56.249902+0000","last_fullsized":"2024-04-26T23:58:56.249902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:11.404827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.212920+0000","last_change":"2024-04-26T23:58:55.201931+0000","last_active":"2024-04-26T23:58:56.212920+0000","last_peered":"2024-04-26T23:58:56.212920+0000","last_clean":"2024-04-26T23:58:56.212920+0000","last_became_active":"2024-04-26T23:58:55.196810+0000","last_became_peered":"2024-04-26T23:58:55.196810+0000","last_unstale":"2024-04-26T23:58:56.212920+0000","last_undegraded":"2024-04-26T23:58:56.212920+0000","last_fullsized":"2024-04-26T23:58:56.212920+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:04:39.775990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.355047+0000","last_change":"2024-04-26T23:58:50.126477+0000","last_active":"2024-04-26T23:59:23.355047+0000","last_peered":"2024-04-26T23:59:23.355047+0000","last_clean":"2024-04-26T23:59:23.355047+0000","last_became_active":"2024-04-26T23:58:50.126205+0000","last_became_peered":"2024-04-26T23:58:50.126205+0000","last_unstale":"2024-04-26T23:59:23.355047+0000","last_undegraded":"2024-04-26T23:59:23.355047+0000","last_fullsized":"2024-04-26T23:59:23.355047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:03:18.513036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246708+0000","last_change":"2024-04-26T23:58:51.145782+0000","last_active":"2024-04-26T23:58:56.246708+0000","last_peered":"2024-04-26T23:58:56.246708+0000","last_clean":"2024-04-26T23:58:56.246708+0000","last_became_active":"2024-04-26T23:58:51.145344+0000","last_became_peered":"2024-04-26T23:58:51.145344+0000","last_unstale":"2024-04-26T23:58:56.246708+0000","last_undegraded":"2024-04-26T23:58:56.246708+0000","last_fullsized":"2024-04-26T23:58:56.246708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:32.062690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249720+0000","last_change":"2024-04-26T23:58:55.198953+0000","last_active":"2024-04-26T23:58:56.249720+0000","last_peered":"2024-04-26T23:58:56.249720+0000","last_clean":"2024-04-26T23:58:56.249720+0000","last_became_active":"2024-04-26T23:58:55.198560+0000","last_became_peered":"2024-04-26T23:58:55.198560+0000","last_unstale":"2024-04-26T23:58:56.249720+0000","last_undegraded":"2024-04-26T23:58:56.249720+0000","last_fullsized":"2024-04-26T23:58:56.249720+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:57:07.109698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.350161+0000","last_change":"2024-04-26T23:58:50.103489+0000","last_active":"2024-04-26T23:59:23.350161+0000","last_peered":"2024-04-26T23:59:23.350161+0000","last_clean":"2024-04-26T23:59:23.350161+0000","last_became_active":"2024-04-26T23:58:50.103068+0000","last_became_peered":"2024-04-26T23:58:50.103068+0000","last_unstale":"2024-04-26T23:59:23.350161+0000","last_undegraded":"2024-04-26T23:59:23.350161+0000","last_fullsized":"2024-04-26T23:59:23.350161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:22:26.490816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248710+0000","last_change":"2024-04-26T23:58:51.163680+0000","last_active":"2024-04-26T23:58:56.248710+0000","last_peered":"2024-04-26T23:58:56.248710+0000","last_clean":"2024-04-26T23:58:56.248710+0000","last_became_active":"2024-04-26T23:58:51.158135+0000","last_became_peered":"2024-04-26T23:58:51.158135+0000","last_unstale":"2024-04-26T23:58:56.248710+0000","last_undegraded":"2024-04-26T23:58:56.248710+0000","last_fullsized":"2024-04-26T23:58:56.248710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:56.562260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249350+0000","last_change":"2024-04-26T23:58:55.211908+0000","last_active":"2024-04-26T23:58:56.249350+0000","last_peered":"2024-04-26T23:58:56.249350+0000","last_clean":"2024-04-26T23:58:56.249350+0000","last_became_active":"2024-04-26T23:58:55.211430+0000","last_became_peered":"2024-04-26T23:58:55.211430+0000","last_unstale":"2024-04-26T23:58:56.249350+0000","last_undegraded":"2024-04-26T23:58:56.249350+0000","last_fullsized":"2024-04-26T23:58:56.249350+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:51:13.315395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.355592+0000","last_change":"2024-04-26T23:58:50.116030+0000","last_active":"2024-04-26T23:59:23.355592+0000","last_peered":"2024-04-26T23:59:23.355592+0000","last_clean":"2024-04-26T23:59:23.355592+0000","last_became_active":"2024-04-26T23:58:50.115744+0000","last_became_peered":"2024-04-26T23:58:50.115744+0000","last_unstale":"2024-04-26T23:59:23.355592+0000","last_undegraded":"2024-04-26T23:59:23.355592+0000","last_fullsized":"2024-04-26T23:59:23.355592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:05:21.823842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246738+0000","last_change":"2024-04-26T23:58:51.140652+0000","last_active":"2024-04-26T23:58:56.246738+0000","last_peered":"2024-04-26T23:58:56.246738+0000","last_clean":"2024-04-26T23:58:56.246738+0000","last_became_active":"2024-04-26T23:58:51.140330+0000","last_became_peered":"2024-04-26T23:58:51.140330+0000","last_unstale":"2024-04-26T23:58:56.246738+0000","last_undegraded":"2024-04-26T23:58:56.246738+0000","last_fullsized":"2024-04-26T23:58:56.246738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:59:56.037856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250695+0000","last_change":"2024-04-26T23:58:55.200964+0000","last_active":"2024-04-26T23:58:56.250695+0000","last_peered":"2024-04-26T23:58:56.250695+0000","last_clean":"2024-04-26T23:58:56.250695+0000","last_became_active":"2024-04-26T23:58:55.200404+0000","last_became_peered":"2024-04-26T23:58:55.200404+0000","last_unstale":"2024-04-26T23:58:56.250695+0000","last_undegraded":"2024-04-26T23:58:56.250695+0000","last_fullsized":"2024-04-26T23:58:56.250695+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:43:51.159653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'16","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:25.869195+0000","last_change":"2024-04-26T23:58:50.101850+0000","last_active":"2024-04-26T23:59:25.869195+0000","last_peered":"2024-04-26T23:59:25.869195+0000","last_clean":"2024-04-26T23:59:25.869195+0000","last_became_active":"2024-04-26T23:58:50.101521+0000","last_became_peered":"2024-04-26T23:58:50.101521+0000","last_unstale":"2024-04-26T23:59:25.869195+0000","last_undegraded":"2024-04-26T23:59:25.869195+0000","last_fullsized":"2024-04-26T23:59:25.869195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:06.810644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10442,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.377740+0000","last_change":"2024-04-26T23:58:51.143441+0000","last_active":"2024-04-26T23:58:56.377740+0000","last_peered":"2024-04-26T23:58:56.377740+0000","last_clean":"2024-04-26T23:58:56.377740+0000","last_became_active":"2024-04-26T23:58:51.141779+0000","last_became_peered":"2024-04-26T23:58:51.141779+0000","last_unstale":"2024-04-26T23:58:56.377740+0000","last_undegraded":"2024-04-26T23:58:56.377740+0000","last_fullsized":"2024-04-26T23:58:56.377740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:56:28.067685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.209638+0000","last_change":"2024-04-26T23:58:55.207277+0000","last_active":"2024-04-26T23:58:56.209638+0000","last_peered":"2024-04-26T23:58:56.209638+0000","last_clean":"2024-04-26T23:58:56.209638+0000","last_became_active":"2024-04-26T23:58:55.206905+0000","last_became_peered":"2024-04-26T23:58:55.206905+0000","last_unstale":"2024-04-26T23:58:56.209638+0000","last_undegraded":"2024-04-26T23:58:56.209638+0000","last_fullsized":"2024-04-26T23:58:56.209638+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:03:34.254752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.363766+0000","last_change":"2024-04-26T23:58:50.115642+0000","last_active":"2024-04-26T23:59:24.363766+0000","last_peered":"2024-04-26T23:59:24.363766+0000","last_clean":"2024-04-26T23:59:24.363766+0000","last_became_active":"2024-04-26T23:58:50.115310+0000","last_became_peered":"2024-04-26T23:58:50.115310+0000","last_unstale":"2024-04-26T23:59:24.363766+0000","last_undegraded":"2024-04-26T23:59:24.363766+0000","last_fullsized":"2024-04-26T23:59:24.363766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:33.189230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250119+0000","last_change":"2024-04-26T23:58:51.159397+0000","last_active":"2024-04-26T23:58:56.250119+0000","last_peered":"2024-04-26T23:58:56.250119+0000","last_clean":"2024-04-26T23:58:56.250119+0000","last_became_active":"2024-04-26T23:58:51.159054+0000","last_became_peered":"2024-04-26T23:58:51.159054+0000","last_unstale":"2024-04-26T23:58:56.250119+0000","last_undegraded":"2024-04-26T23:58:56.250119+0000","last_fullsized":"2024-04-26T23:58:56.250119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:54:13.306198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247931+0000","last_change":"2024-04-26T23:58:55.215794+0000","last_active":"2024-04-26T23:58:56.247931+0000","last_peered":"2024-04-26T23:58:56.247931+0000","last_clean":"2024-04-26T23:58:56.247931+0000","last_became_active":"2024-04-26T23:58:55.215366+0000","last_became_peered":"2024-04-26T23:58:55.215366+0000","last_unstale":"2024-04-26T23:58:56.247931+0000","last_undegraded":"2024-04-26T23:58:56.247931+0000","last_fullsized":"2024-04-26T23:58:56.247931+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:54:20.347552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.353811+0000","last_change":"2024-04-26T23:58:50.112207+0000","last_active":"2024-04-26T23:59:23.353811+0000","last_peered":"2024-04-26T23:59:23.353811+0000","last_clean":"2024-04-26T23:59:23.353811+0000","last_became_active":"2024-04-26T23:58:50.111975+0000","last_became_peered":"2024-04-26T23:58:50.111975+0000","last_unstale":"2024-04-26T23:59:23.353811+0000","last_undegraded":"2024-04-26T23:59:23.353811+0000","last_fullsized":"2024-04-26T23:59:23.353811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:29.091698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247517+0000","last_change":"2024-04-26T23:58:51.147154+0000","last_active":"2024-04-26T23:58:56.247517+0000","last_peered":"2024-04-26T23:58:56.247517+0000","last_clean":"2024-04-26T23:58:56.247517+0000","last_became_active":"2024-04-26T23:58:51.146809+0000","last_became_peered":"2024-04-26T23:58:51.146809+0000","last_unstale":"2024-04-26T23:58:56.247517+0000","last_undegraded":"2024-04-26T23:58:56.247517+0000","last_fullsized":"2024-04-26T23:58:56.247517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:22.308365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.252855+0000","last_change":"2024-04-26T23:58:55.214461+0000","last_active":"2024-04-26T23:58:56.252855+0000","last_peered":"2024-04-26T23:58:56.252855+0000","last_clean":"2024-04-26T23:58:56.252855+0000","last_became_active":"2024-04-26T23:58:55.213442+0000","last_became_peered":"2024-04-26T23:58:55.213442+0000","last_unstale":"2024-04-26T23:58:56.252855+0000","last_undegraded":"2024-04-26T23:58:56.252855+0000","last_fullsized":"2024-04-26T23:58:56.252855+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:07:03.114274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:26.390040+0000","last_change":"2024-04-26T23:58:50.100554+0000","last_active":"2024-04-26T23:59:26.390040+0000","last_peered":"2024-04-26T23:59:26.390040+0000","last_clean":"2024-04-26T23:59:26.390040+0000","last_became_active":"2024-04-26T23:58:50.100200+0000","last_became_peered":"2024-04-26T23:58:50.100200+0000","last_unstale":"2024-04-26T23:59:26.390040+0000","last_undegraded":"2024-04-26T23:59:26.390040+0000","last_fullsized":"2024-04-26T23:59:26.390040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:59.913424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247246+0000","last_change":"2024-04-26T23:58:51.141030+0000","last_active":"2024-04-26T23:58:56.247246+0000","last_peered":"2024-04-26T23:58:56.247246+0000","last_clean":"2024-04-26T23:58:56.247246+0000","last_became_active":"2024-04-26T23:58:51.140694+0000","last_became_peered":"2024-04-26T23:58:51.140694+0000","last_unstale":"2024-04-26T23:58:56.247246+0000","last_undegraded":"2024-04-26T23:58:56.247246+0000","last_fullsized":"2024-04-26T23:58:56.247246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:35:29.827020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.252130+0000","last_change":"2024-04-26T23:58:55.216556+0000","last_active":"2024-04-26T23:58:56.252130+0000","last_peered":"2024-04-26T23:58:56.252130+0000","last_clean":"2024-04-26T23:58:56.252130+0000","last_became_active":"2024-04-26T23:58:55.215798+0000","last_became_peered":"2024-04-26T23:58:55.215798+0000","last_unstale":"2024-04-26T23:58:56.252130+0000","last_undegraded":"2024-04-26T23:58:56.252130+0000","last_fullsized":"2024-04-26T23:58:56.252130+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:32:08.733655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.367485+0000","last_change":"2024-04-26T23:58:50.119495+0000","last_active":"2024-04-26T23:59:24.367485+0000","last_peered":"2024-04-26T23:59:24.367485+0000","last_clean":"2024-04-26T23:59:24.367485+0000","last_became_active":"2024-04-26T23:58:50.119174+0000","last_became_peered":"2024-04-26T23:58:50.119174+0000","last_unstale":"2024-04-26T23:59:24.367485+0000","last_undegraded":"2024-04-26T23:59:24.367485+0000","last_fullsized":"2024-04-26T23:59:24.367485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:53:23.408578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208753+0000","last_change":"2024-04-26T23:58:51.141943+0000","last_active":"2024-04-26T23:58:56.208753+0000","last_peered":"2024-04-26T23:58:56.208753+0000","last_clean":"2024-04-26T23:58:56.208753+0000","last_became_active":"2024-04-26T23:58:51.141624+0000","last_became_peered":"2024-04-26T23:58:51.141624+0000","last_unstale":"2024-04-26T23:58:56.208753+0000","last_undegraded":"2024-04-26T23:58:56.208753+0000","last_fullsized":"2024-04-26T23:58:56.208753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:46:25.114059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208837+0000","last_change":"2024-04-26T23:58:55.200247+0000","last_active":"2024-04-26T23:58:56.208837+0000","last_peered":"2024-04-26T23:58:56.208837+0000","last_clean":"2024-04-26T23:58:56.208837+0000","last_became_active":"2024-04-26T23:58:55.199854+0000","last_became_peered":"2024-04-26T23:58:55.199854+0000","last_unstale":"2024-04-26T23:58:56.208837+0000","last_undegraded":"2024-04-26T23:58:56.208837+0000","last_fullsized":"2024-04-26T23:58:56.208837+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:22:00.066549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:25.383274+0000","last_change":"2024-04-26T23:58:50.102901+0000","last_active":"2024-04-26T23:59:25.383274+0000","last_peered":"2024-04-26T23:59:25.383274+0000","last_clean":"2024-04-26T23:59:25.383274+0000","last_became_active":"2024-04-26T23:58:50.102537+0000","last_became_peered":"2024-04-26T23:58:50.102537+0000","last_unstale":"2024-04-26T23:59:25.383274+0000","last_undegraded":"2024-04-26T23:59:25.383274+0000","last_fullsized":"2024-04-26T23:59:25.383274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:21:15.043327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248067+0000","last_change":"2024-04-26T23:58:51.135160+0000","last_active":"2024-04-26T23:58:56.248067+0000","last_peered":"2024-04-26T23:58:56.248067+0000","last_clean":"2024-04-26T23:58:56.248067+0000","last_became_active":"2024-04-26T23:58:51.134806+0000","last_became_peered":"2024-04-26T23:58:51.134806+0000","last_unstale":"2024-04-26T23:58:56.248067+0000","last_undegraded":"2024-04-26T23:58:56.248067+0000","last_fullsized":"2024-04-26T23:58:56.248067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:35:10.839180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.377785+0000","last_change":"2024-04-26T23:58:55.203555+0000","last_active":"2024-04-26T23:58:56.377785+0000","last_peered":"2024-04-26T23:58:56.377785+0000","last_clean":"2024-04-26T23:58:56.377785+0000","last_became_active":"2024-04-26T23:58:55.202968+0000","last_became_peered":"2024-04-26T23:58:55.202968+0000","last_unstale":"2024-04-26T23:58:56.377785+0000","last_undegraded":"2024-04-26T23:58:56.377785+0000","last_fullsized":"2024-04-26T23:58:56.377785+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:01.149966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250158+0000","last_change":"2024-04-26T23:58:50.129000+0000","last_active":"2024-04-26T23:58:56.250158+0000","last_peered":"2024-04-26T23:58:56.250158+0000","last_clean":"2024-04-26T23:58:56.250158+0000","last_became_active":"2024-04-26T23:58:50.128646+0000","last_became_peered":"2024-04-26T23:58:50.128646+0000","last_unstale":"2024-04-26T23:58:56.250158+0000","last_undegraded":"2024-04-26T23:58:56.250158+0000","last_fullsized":"2024-04-26T23:58:56.250158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:34.200578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247722+0000","last_change":"2024-04-26T23:58:51.159557+0000","last_active":"2024-04-26T23:58:56.247722+0000","last_peered":"2024-04-26T23:58:56.247722+0000","last_clean":"2024-04-26T23:58:56.247722+0000","last_became_active":"2024-04-26T23:58:51.159211+0000","last_became_peered":"2024-04-26T23:58:51.159211+0000","last_unstale":"2024-04-26T23:58:56.247722+0000","last_undegraded":"2024-04-26T23:58:56.247722+0000","last_fullsized":"2024-04-26T23:58:56.247722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:48.740612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.253576+0000","last_change":"2024-04-26T23:58:55.193585+0000","last_active":"2024-04-26T23:58:56.253576+0000","last_peered":"2024-04-26T23:58:56.253576+0000","last_clean":"2024-04-26T23:58:56.253576+0000","last_became_active":"2024-04-26T23:58:55.193031+0000","last_became_peered":"2024-04-26T23:58:55.193031+0000","last_unstale":"2024-04-26T23:58:56.253576+0000","last_undegraded":"2024-04-26T23:58:56.253576+0000","last_fullsized":"2024-04-26T23:58:56.253576+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:20.689117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247659+0000","last_change":"2024-04-26T23:58:50.124301+0000","last_active":"2024-04-26T23:58:56.247659+0000","last_peered":"2024-04-26T23:58:56.247659+0000","last_clean":"2024-04-26T23:58:56.247659+0000","last_became_active":"2024-04-26T23:58:50.124016+0000","last_became_peered":"2024-04-26T23:58:50.124016+0000","last_unstale":"2024-04-26T23:58:56.247659+0000","last_undegraded":"2024-04-26T23:58:56.247659+0000","last_fullsized":"2024-04-26T23:58:56.247659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:00.835698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247845+0000","last_change":"2024-04-26T23:58:51.164703+0000","last_active":"2024-04-26T23:58:56.247845+0000","last_peered":"2024-04-26T23:58:56.247845+0000","last_clean":"2024-04-26T23:58:56.247845+0000","last_became_active":"2024-04-26T23:58:51.164346+0000","last_became_peered":"2024-04-26T23:58:51.164346+0000","last_unstale":"2024-04-26T23:58:56.247845+0000","last_undegraded":"2024-04-26T23:58:56.247845+0000","last_fullsized":"2024-04-26T23:58:56.247845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:45:39.667034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.252225+0000","last_change":"2024-04-26T23:58:55.193795+0000","last_active":"2024-04-26T23:58:56.252225+0000","last_peered":"2024-04-26T23:58:56.252225+0000","last_clean":"2024-04-26T23:58:56.252225+0000","last_became_active":"2024-04-26T23:58:55.193323+0000","last_became_peered":"2024-04-26T23:58:55.193323+0000","last_unstale":"2024-04-26T23:58:56.252225+0000","last_undegraded":"2024-04-26T23:58:56.252225+0000","last_fullsized":"2024-04-26T23:58:56.252225+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:44.301311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208737+0000","last_change":"2024-04-26T23:58:51.140198+0000","last_active":"2024-04-26T23:58:56.208737+0000","last_peered":"2024-04-26T23:58:56.208737+0000","last_clean":"2024-04-26T23:58:56.208737+0000","last_became_active":"2024-04-26T23:58:51.139851+0000","last_became_peered":"2024-04-26T23:58:51.139851+0000","last_unstale":"2024-04-26T23:58:56.208737+0000","last_undegraded":"2024-04-26T23:58:56.208737+0000","last_fullsized":"2024-04-26T23:58:56.208737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:50.356312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:25.385965+0000","last_change":"2024-04-26T23:58:50.132099+0000","last_active":"2024-04-26T23:59:25.385965+0000","last_peered":"2024-04-26T23:59:25.385965+0000","last_clean":"2024-04-26T23:59:25.385965+0000","last_became_active":"2024-04-26T23:58:50.131712+0000","last_became_peered":"2024-04-26T23:58:50.131712+0000","last_unstale":"2024-04-26T23:59:25.385965+0000","last_undegraded":"2024-04-26T23:59:25.385965+0000","last_fullsized":"2024-04-26T23:59:25.385965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:40.849259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:57.789831+0000","last_change":"2024-04-26T23:58:55.205614+0000","last_active":"2024-04-26T23:58:57.789831+0000","last_peered":"2024-04-26T23:58:57.789831+0000","last_clean":"2024-04-26T23:58:57.789831+0000","last_became_active":"2024-04-26T23:58:55.205206+0000","last_became_peered":"2024-04-26T23:58:55.205206+0000","last_unstale":"2024-04-26T23:58:57.789831+0000","last_undegraded":"2024-04-26T23:58:57.789831+0000","last_fullsized":"2024-04-26T23:58:57.789831+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:42:24.324563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211463+0000","last_change":"2024-04-26T23:58:51.159808+0000","last_active":"2024-04-26T23:58:56.211463+0000","last_peered":"2024-04-26T23:58:56.211463+0000","last_clean":"2024-04-26T23:58:56.211463+0000","last_became_active":"2024-04-26T23:58:51.159444+0000","last_became_peered":"2024-04-26T23:58:51.159444+0000","last_unstale":"2024-04-26T23:58:56.211463+0000","last_undegraded":"2024-04-26T23:58:56.211463+0000","last_fullsized":"2024-04-26T23:58:56.211463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:57.401134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:25.381198+0000","last_change":"2024-04-26T23:58:50.120100+0000","last_active":"2024-04-26T23:59:25.381198+0000","last_peered":"2024-04-26T23:59:25.381198+0000","last_clean":"2024-04-26T23:59:25.381198+0000","last_became_active":"2024-04-26T23:58:50.119613+0000","last_became_peered":"2024-04-26T23:58:50.119613+0000","last_unstale":"2024-04-26T23:59:25.381198+0000","last_undegraded":"2024-04-26T23:59:25.381198+0000","last_fullsized":"2024-04-26T23:59:25.381198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:32:18.116454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":343,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247017+0000","last_change":"2024-04-26T23:57:30.173979+0000","last_active":"2024-04-26T23:58:56.247017+0000","last_peered":"2024-04-26T23:58:56.247017+0000","last_clean":"2024-04-26T23:58:56.247017+0000","last_became_active":"2024-04-26T23:57:30.173549+0000","last_became_peered":"2024-04-26T23:57:30.173549+0000","last_unstale":"2024-04-26T23:58:56.247017+0000","last_undegraded":"2024-04-26T23:58:56.247017+0000","last_fullsized":"2024-04-26T23:58:56.247017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:53:33.634094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:53:33.634094+0000","last_clean_scrub_stamp":"2024-04-26T23:53:33.634094+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:48.026677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.209459+0000","last_change":"2024-04-26T23:58:55.214053+0000","last_active":"2024-04-26T23:58:56.209459+0000","last_peered":"2024-04-26T23:58:56.209459+0000","last_clean":"2024-04-26T23:58:56.209459+0000","last_became_active":"2024-04-26T23:58:55.213586+0000","last_became_peered":"2024-04-26T23:58:55.213586+0000","last_unstale":"2024-04-26T23:58:56.209459+0000","last_undegraded":"2024-04-26T23:58:56.209459+0000","last_fullsized":"2024-04-26T23:58:56.209459+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:57:16.843969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248263+0000","last_change":"2024-04-26T23:58:51.152687+0000","last_active":"2024-04-26T23:58:56.248263+0000","last_peered":"2024-04-26T23:58:56.248263+0000","last_clean":"2024-04-26T23:58:56.248263+0000","last_became_active":"2024-04-26T23:58:51.149082+0000","last_became_peered":"2024-04-26T23:58:51.149082+0000","last_unstale":"2024-04-26T23:58:56.248263+0000","last_undegraded":"2024-04-26T23:58:56.248263+0000","last_fullsized":"2024-04-26T23:58:56.248263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:38:42.630214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.361993+0000","last_change":"2024-04-26T23:58:50.104485+0000","last_active":"2024-04-26T23:59:24.361993+0000","last_peered":"2024-04-26T23:59:24.361993+0000","last_clean":"2024-04-26T23:59:24.361993+0000","last_became_active":"2024-04-26T23:58:50.104260+0000","last_became_peered":"2024-04-26T23:58:50.104260+0000","last_unstale":"2024-04-26T23:59:24.361993+0000","last_undegraded":"2024-04-26T23:59:24.361993+0000","last_fullsized":"2024-04-26T23:59:24.361993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:04:06.268993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247930+0000","last_change":"2024-04-26T23:58:55.190219+0000","last_active":"2024-04-26T23:58:56.247930+0000","last_peered":"2024-04-26T23:58:56.247930+0000","last_clean":"2024-04-26T23:58:56.247930+0000","last_became_active":"2024-04-26T23:58:55.189801+0000","last_became_peered":"2024-04-26T23:58:55.189801+0000","last_unstale":"2024-04-26T23:58:56.247930+0000","last_undegraded":"2024-04-26T23:58:56.247930+0000","last_fullsized":"2024-04-26T23:58:56.247930+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:53:42.170710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.354491+0000","last_change":"2024-04-26T23:58:50.117442+0000","last_active":"2024-04-26T23:59:23.354491+0000","last_peered":"2024-04-26T23:59:23.354491+0000","last_clean":"2024-04-26T23:59:23.354491+0000","last_became_active":"2024-04-26T23:58:50.117112+0000","last_became_peered":"2024-04-26T23:58:50.117112+0000","last_unstale":"2024-04-26T23:59:23.354491+0000","last_undegraded":"2024-04-26T23:59:23.354491+0000","last_fullsized":"2024-04-26T23:59:23.354491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:18.352145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.251795+0000","last_change":"2024-04-26T23:58:51.156670+0000","last_active":"2024-04-26T23:58:56.251795+0000","last_peered":"2024-04-26T23:58:56.251795+0000","last_clean":"2024-04-26T23:58:56.251795+0000","last_became_active":"2024-04-26T23:58:51.155596+0000","last_became_peered":"2024-04-26T23:58:51.155596+0000","last_unstale":"2024-04-26T23:58:56.251795+0000","last_undegraded":"2024-04-26T23:58:56.251795+0000","last_fullsized":"2024-04-26T23:58:56.251795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:38.053814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211366+0000","last_change":"2024-04-26T23:58:55.201644+0000","last_active":"2024-04-26T23:58:56.211366+0000","last_peered":"2024-04-26T23:58:56.211366+0000","last_clean":"2024-04-26T23:58:56.211366+0000","last_became_active":"2024-04-26T23:58:55.196421+0000","last_became_peered":"2024-04-26T23:58:55.196421+0000","last_unstale":"2024-04-26T23:58:56.211366+0000","last_undegraded":"2024-04-26T23:58:56.211366+0000","last_fullsized":"2024-04-26T23:58:56.211366+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:26:30.309773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.372055+0000","last_change":"2024-04-26T23:58:50.120805+0000","last_active":"2024-04-26T23:59:24.372055+0000","last_peered":"2024-04-26T23:59:24.372055+0000","last_clean":"2024-04-26T23:59:24.372055+0000","last_became_active":"2024-04-26T23:58:50.120403+0000","last_became_peered":"2024-04-26T23:58:50.120403+0000","last_unstale":"2024-04-26T23:59:24.372055+0000","last_undegraded":"2024-04-26T23:59:24.372055+0000","last_fullsized":"2024-04-26T23:59:24.372055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:53:28.950313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.212866+0000","last_change":"2024-04-26T23:58:51.167109+0000","last_active":"2024-04-26T23:58:56.212866+0000","last_peered":"2024-04-26T23:58:56.212866+0000","last_clean":"2024-04-26T23:58:56.212866+0000","last_became_active":"2024-04-26T23:58:51.164584+0000","last_became_peered":"2024-04-26T23:58:51.164584+0000","last_unstale":"2024-04-26T23:58:56.212866+0000","last_undegraded":"2024-04-26T23:58:56.212866+0000","last_fullsized":"2024-04-26T23:58:56.212866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:12:12.160553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246637+0000","last_change":"2024-04-26T23:58:55.183138+0000","last_active":"2024-04-26T23:58:56.246637+0000","last_peered":"2024-04-26T23:58:56.246637+0000","last_clean":"2024-04-26T23:58:56.246637+0000","last_became_active":"2024-04-26T23:58:55.182779+0000","last_became_peered":"2024-04-26T23:58:55.182779+0000","last_unstale":"2024-04-26T23:58:56.246637+0000","last_undegraded":"2024-04-26T23:58:56.246637+0000","last_fullsized":"2024-04-26T23:58:56.246637+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:16.866145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.350263+0000","last_change":"2024-04-26T23:58:50.120401+0000","last_active":"2024-04-26T23:59:23.350263+0000","last_peered":"2024-04-26T23:59:23.350263+0000","last_clean":"2024-04-26T23:59:23.350263+0000","last_became_active":"2024-04-26T23:58:50.120036+0000","last_became_peered":"2024-04-26T23:58:50.120036+0000","last_unstale":"2024-04-26T23:59:23.350263+0000","last_undegraded":"2024-04-26T23:59:23.350263+0000","last_fullsized":"2024-04-26T23:59:23.350263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:49.852298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248960+0000","last_change":"2024-04-26T23:58:51.171317+0000","last_active":"2024-04-26T23:58:56.248960+0000","last_peered":"2024-04-26T23:58:56.248960+0000","last_clean":"2024-04-26T23:58:56.248960+0000","last_became_active":"2024-04-26T23:58:51.170978+0000","last_became_peered":"2024-04-26T23:58:51.170978+0000","last_unstale":"2024-04-26T23:58:56.248960+0000","last_undegraded":"2024-04-26T23:58:56.248960+0000","last_fullsized":"2024-04-26T23:58:56.248960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:09:29.173423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248952+0000","last_change":"2024-04-26T23:58:55.216273+0000","last_active":"2024-04-26T23:58:56.248952+0000","last_peered":"2024-04-26T23:58:56.248952+0000","last_clean":"2024-04-26T23:58:56.248952+0000","last_became_active":"2024-04-26T23:58:55.215630+0000","last_became_peered":"2024-04-26T23:58:55.215630+0000","last_unstale":"2024-04-26T23:58:56.248952+0000","last_undegraded":"2024-04-26T23:58:56.248952+0000","last_fullsized":"2024-04-26T23:58:56.248952+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:15:41.195389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.353662+0000","last_change":"2024-04-26T23:58:50.111216+0000","last_active":"2024-04-26T23:59:23.353662+0000","last_peered":"2024-04-26T23:59:23.353662+0000","last_clean":"2024-04-26T23:59:23.353662+0000","last_became_active":"2024-04-26T23:58:50.110969+0000","last_became_peered":"2024-04-26T23:58:50.110969+0000","last_unstale":"2024-04-26T23:59:23.353662+0000","last_undegraded":"2024-04-26T23:59:23.353662+0000","last_fullsized":"2024-04-26T23:59:23.353662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:01.319448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.377268+0000","last_change":"2024-04-26T23:58:51.132643+0000","last_active":"2024-04-26T23:58:56.377268+0000","last_peered":"2024-04-26T23:58:56.377268+0000","last_clean":"2024-04-26T23:58:56.377268+0000","last_became_active":"2024-04-26T23:58:51.132250+0000","last_became_peered":"2024-04-26T23:58:51.132250+0000","last_unstale":"2024-04-26T23:58:56.377268+0000","last_undegraded":"2024-04-26T23:58:56.377268+0000","last_fullsized":"2024-04-26T23:58:56.377268+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:00:11.180178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.253164+0000","last_change":"2024-04-26T23:58:55.204718+0000","last_active":"2024-04-26T23:58:56.253164+0000","last_peered":"2024-04-26T23:58:56.253164+0000","last_clean":"2024-04-26T23:58:56.253164+0000","last_became_active":"2024-04-26T23:58:55.204159+0000","last_became_peered":"2024-04-26T23:58:55.204159+0000","last_unstale":"2024-04-26T23:58:56.253164+0000","last_undegraded":"2024-04-26T23:58:56.253164+0000","last_fullsized":"2024-04-26T23:58:56.253164+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:42:52.199200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.355283+0000","last_change":"2024-04-26T23:58:50.126926+0000","last_active":"2024-04-26T23:59:23.355283+0000","last_peered":"2024-04-26T23:59:23.355283+0000","last_clean":"2024-04-26T23:59:23.355283+0000","last_became_active":"2024-04-26T23:58:50.126552+0000","last_became_peered":"2024-04-26T23:58:50.126552+0000","last_unstale":"2024-04-26T23:59:23.355283+0000","last_undegraded":"2024-04-26T23:59:23.355283+0000","last_fullsized":"2024-04-26T23:59:23.355283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:18:25.024805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247596+0000","last_change":"2024-04-26T23:58:51.162722+0000","last_active":"2024-04-26T23:58:56.247596+0000","last_peered":"2024-04-26T23:58:56.247596+0000","last_clean":"2024-04-26T23:58:56.247596+0000","last_became_active":"2024-04-26T23:58:51.162387+0000","last_became_peered":"2024-04-26T23:58:51.162387+0000","last_unstale":"2024-04-26T23:58:56.247596+0000","last_undegraded":"2024-04-26T23:58:56.247596+0000","last_fullsized":"2024-04-26T23:58:56.247596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:07.143478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248784+0000","last_change":"2024-04-26T23:58:55.216201+0000","last_active":"2024-04-26T23:58:56.248784+0000","last_peered":"2024-04-26T23:58:56.248784+0000","last_clean":"2024-04-26T23:58:56.248784+0000","last_became_active":"2024-04-26T23:58:55.215730+0000","last_became_peered":"2024-04-26T23:58:55.215730+0000","last_unstale":"2024-04-26T23:58:56.248784+0000","last_undegraded":"2024-04-26T23:58:56.248784+0000","last_fullsized":"2024-04-26T23:58:56.248784+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:21.484879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246797+0000","last_change":"2024-04-26T23:58:50.111313+0000","last_active":"2024-04-26T23:58:56.246797+0000","last_peered":"2024-04-26T23:58:56.246797+0000","last_clean":"2024-04-26T23:58:56.246797+0000","last_became_active":"2024-04-26T23:58:50.110937+0000","last_became_peered":"2024-04-26T23:58:50.110937+0000","last_unstale":"2024-04-26T23:58:56.246797+0000","last_undegraded":"2024-04-26T23:58:56.246797+0000","last_fullsized":"2024-04-26T23:58:56.246797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:52:24.615527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.212337+0000","last_change":"2024-04-26T23:58:51.174152+0000","last_active":"2024-04-26T23:58:56.212337+0000","last_peered":"2024-04-26T23:58:56.212337+0000","last_clean":"2024-04-26T23:58:56.212337+0000","last_became_active":"2024-04-26T23:58:51.170956+0000","last_became_peered":"2024-04-26T23:58:51.170956+0000","last_unstale":"2024-04-26T23:58:56.212337+0000","last_undegraded":"2024-04-26T23:58:56.212337+0000","last_fullsized":"2024-04-26T23:58:56.212337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:09:09.253242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248697+0000","last_change":"2024-04-26T23:58:55.206248+0000","last_active":"2024-04-26T23:58:56.248697+0000","last_peered":"2024-04-26T23:58:56.248697+0000","last_clean":"2024-04-26T23:58:56.248697+0000","last_became_active":"2024-04-26T23:58:55.205820+0000","last_became_peered":"2024-04-26T23:58:55.205820+0000","last_unstale":"2024-04-26T23:58:56.248697+0000","last_undegraded":"2024-04-26T23:58:56.248697+0000","last_fullsized":"2024-04-26T23:58:56.248697+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:52:24.789706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.350261+0000","last_change":"2024-04-26T23:58:50.103874+0000","last_active":"2024-04-26T23:59:23.350261+0000","last_peered":"2024-04-26T23:59:23.350261+0000","last_clean":"2024-04-26T23:59:23.350261+0000","last_became_active":"2024-04-26T23:58:50.103590+0000","last_became_peered":"2024-04-26T23:58:50.103590+0000","last_unstale":"2024-04-26T23:59:23.350261+0000","last_undegraded":"2024-04-26T23:59:23.350261+0000","last_fullsized":"2024-04-26T23:59:23.350261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:34:59.817736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247722+0000","last_change":"2024-04-26T23:58:51.146396+0000","last_active":"2024-04-26T23:58:56.247722+0000","last_peered":"2024-04-26T23:58:56.247722+0000","last_clean":"2024-04-26T23:58:56.247722+0000","last_became_active":"2024-04-26T23:58:51.146046+0000","last_became_peered":"2024-04-26T23:58:51.146046+0000","last_unstale":"2024-04-26T23:58:56.247722+0000","last_undegraded":"2024-04-26T23:58:56.247722+0000","last_fullsized":"2024-04-26T23:58:56.247722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:14.230111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246313+0000","last_change":"2024-04-26T23:58:55.199320+0000","last_active":"2024-04-26T23:58:56.246313+0000","last_peered":"2024-04-26T23:58:56.246313+0000","last_clean":"2024-04-26T23:58:56.246313+0000","last_became_active":"2024-04-26T23:58:55.198843+0000","last_became_peered":"2024-04-26T23:58:55.198843+0000","last_unstale":"2024-04-26T23:58:56.246313+0000","last_undegraded":"2024-04-26T23:58:56.246313+0000","last_fullsized":"2024-04-26T23:58:56.246313+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:56:54.643998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.368654+0000","last_change":"2024-04-26T23:58:50.120892+0000","last_active":"2024-04-26T23:59:24.368654+0000","last_peered":"2024-04-26T23:59:24.368654+0000","last_clean":"2024-04-26T23:59:24.368654+0000","last_became_active":"2024-04-26T23:58:50.120396+0000","last_became_peered":"2024-04-26T23:58:50.120396+0000","last_unstale":"2024-04-26T23:59:24.368654+0000","last_undegraded":"2024-04-26T23:59:24.368654+0000","last_fullsized":"2024-04-26T23:59:24.368654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:16.670482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248035+0000","last_change":"2024-04-26T23:58:51.134917+0000","last_active":"2024-04-26T23:58:56.248035+0000","last_peered":"2024-04-26T23:58:56.248035+0000","last_clean":"2024-04-26T23:58:56.248035+0000","last_became_active":"2024-04-26T23:58:51.134470+0000","last_became_peered":"2024-04-26T23:58:51.134470+0000","last_unstale":"2024-04-26T23:58:56.248035+0000","last_undegraded":"2024-04-26T23:58:56.248035+0000","last_fullsized":"2024-04-26T23:58:56.248035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:03.696427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250223+0000","last_change":"2024-04-26T23:58:55.213966+0000","last_active":"2024-04-26T23:58:56.250223+0000","last_peered":"2024-04-26T23:58:56.250223+0000","last_clean":"2024-04-26T23:58:56.250223+0000","last_became_active":"2024-04-26T23:58:55.213521+0000","last_became_peered":"2024-04-26T23:58:55.213521+0000","last_unstale":"2024-04-26T23:58:56.250223+0000","last_undegraded":"2024-04-26T23:58:56.250223+0000","last_fullsized":"2024-04-26T23:58:56.250223+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:10:02.998037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.209433+0000","last_change":"2024-04-26T23:58:50.118136+0000","last_active":"2024-04-26T23:58:56.209433+0000","last_peered":"2024-04-26T23:58:56.209433+0000","last_clean":"2024-04-26T23:58:56.209433+0000","last_became_active":"2024-04-26T23:58:50.117787+0000","last_became_peered":"2024-04-26T23:58:50.117787+0000","last_unstale":"2024-04-26T23:58:56.209433+0000","last_undegraded":"2024-04-26T23:58:56.209433+0000","last_fullsized":"2024-04-26T23:58:56.209433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:14.145327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247712+0000","last_change":"2024-04-26T23:58:51.166491+0000","last_active":"2024-04-26T23:58:56.247712+0000","last_peered":"2024-04-26T23:58:56.247712+0000","last_clean":"2024-04-26T23:58:56.247712+0000","last_became_active":"2024-04-26T23:58:51.166106+0000","last_became_peered":"2024-04-26T23:58:51.166106+0000","last_unstale":"2024-04-26T23:58:56.247712+0000","last_undegraded":"2024-04-26T23:58:56.247712+0000","last_fullsized":"2024-04-26T23:58:56.247712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:18:58.690731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208576+0000","last_change":"2024-04-26T23:58:55.184623+0000","last_active":"2024-04-26T23:58:56.208576+0000","last_peered":"2024-04-26T23:58:56.208576+0000","last_clean":"2024-04-26T23:58:56.208576+0000","last_became_active":"2024-04-26T23:58:55.184029+0000","last_became_peered":"2024-04-26T23:58:55.184029+0000","last_unstale":"2024-04-26T23:58:56.208576+0000","last_undegraded":"2024-04-26T23:58:56.208576+0000","last_fullsized":"2024-04-26T23:58:56.208576+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:39:36.267352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.251842+0000","last_change":"2024-04-26T23:58:50.127630+0000","last_active":"2024-04-26T23:58:56.251842+0000","last_peered":"2024-04-26T23:58:56.251842+0000","last_clean":"2024-04-26T23:58:56.251842+0000","last_became_active":"2024-04-26T23:58:50.127134+0000","last_became_peered":"2024-04-26T23:58:50.127134+0000","last_unstale":"2024-04-26T23:58:56.251842+0000","last_undegraded":"2024-04-26T23:58:56.251842+0000","last_fullsized":"2024-04-26T23:58:56.251842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:15.061919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248456+0000","last_change":"2024-04-26T23:58:51.134241+0000","last_active":"2024-04-26T23:58:56.248456+0000","last_peered":"2024-04-26T23:58:56.248456+0000","last_clean":"2024-04-26T23:58:56.248456+0000","last_became_active":"2024-04-26T23:58:51.133867+0000","last_became_peered":"2024-04-26T23:58:51.133867+0000","last_unstale":"2024-04-26T23:58:56.248456+0000","last_undegraded":"2024-04-26T23:58:56.248456+0000","last_fullsized":"2024-04-26T23:58:56.248456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:12:20.171560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247719+0000","last_change":"2024-04-26T23:58:51.132681+0000","last_active":"2024-04-26T23:58:56.247719+0000","last_peered":"2024-04-26T23:58:56.247719+0000","last_clean":"2024-04-26T23:58:56.247719+0000","last_became_active":"2024-04-26T23:58:51.132213+0000","last_became_peered":"2024-04-26T23:58:51.132213+0000","last_unstale":"2024-04-26T23:58:56.247719+0000","last_undegraded":"2024-04-26T23:58:56.247719+0000","last_fullsized":"2024-04-26T23:58:56.247719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:26:52.080670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246905+0000","last_change":"2024-04-26T23:58:51.135731+0000","last_active":"2024-04-26T23:58:56.246905+0000","last_peered":"2024-04-26T23:58:56.246905+0000","last_clean":"2024-04-26T23:58:56.246905+0000","last_became_active":"2024-04-26T23:58:51.135352+0000","last_became_peered":"2024-04-26T23:58:51.135352+0000","last_unstale":"2024-04-26T23:58:56.246905+0000","last_undegraded":"2024-04-26T23:58:56.246905+0000","last_fullsized":"2024-04-26T23:58:56.246905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:45:37.903396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246573+0000","last_change":"2024-04-26T23:58:51.160014+0000","last_active":"2024-04-26T23:58:56.246573+0000","last_peered":"2024-04-26T23:58:56.246573+0000","last_clean":"2024-04-26T23:58:56.246573+0000","last_became_active":"2024-04-26T23:58:51.159459+0000","last_became_peered":"2024-04-26T23:58:51.159459+0000","last_unstale":"2024-04-26T23:58:56.246573+0000","last_undegraded":"2024-04-26T23:58:56.246573+0000","last_fullsized":"2024-04-26T23:58:56.246573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:12.757558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.248891+0000","last_change":"2024-04-26T23:58:51.166942+0000","last_active":"2024-04-26T23:58:56.248891+0000","last_peered":"2024-04-26T23:58:56.248891+0000","last_clean":"2024-04-26T23:58:56.248891+0000","last_became_active":"2024-04-26T23:58:51.162605+0000","last_became_peered":"2024-04-26T23:58:51.162605+0000","last_unstale":"2024-04-26T23:58:56.248891+0000","last_undegraded":"2024-04-26T23:58:56.248891+0000","last_fullsized":"2024-04-26T23:58:56.248891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:38:46.346098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.252080+0000","last_change":"2024-04-26T23:58:51.162638+0000","last_active":"2024-04-26T23:58:56.252080+0000","last_peered":"2024-04-26T23:58:56.252080+0000","last_clean":"2024-04-26T23:58:56.252080+0000","last_became_active":"2024-04-26T23:58:51.162165+0000","last_became_peered":"2024-04-26T23:58:51.162165+0000","last_unstale":"2024-04-26T23:58:56.252080+0000","last_undegraded":"2024-04-26T23:58:56.252080+0000","last_fullsized":"2024-04-26T23:58:56.252080+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:26:32.277183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247961+0000","last_change":"2024-04-26T23:58:51.132667+0000","last_active":"2024-04-26T23:58:56.247961+0000","last_peered":"2024-04-26T23:58:56.247961+0000","last_clean":"2024-04-26T23:58:56.247961+0000","last_became_active":"2024-04-26T23:58:51.132316+0000","last_became_peered":"2024-04-26T23:58:51.132316+0000","last_unstale":"2024-04-26T23:58:56.247961+0000","last_undegraded":"2024-04-26T23:58:56.247961+0000","last_fullsized":"2024-04-26T23:58:56.247961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:41.427650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247285+0000","last_change":"2024-04-26T23:58:51.132614+0000","last_active":"2024-04-26T23:58:56.247285+0000","last_peered":"2024-04-26T23:58:56.247285+0000","last_clean":"2024-04-26T23:58:56.247285+0000","last_became_active":"2024-04-26T23:58:51.132274+0000","last_became_peered":"2024-04-26T23:58:51.132274+0000","last_unstale":"2024-04-26T23:58:56.247285+0000","last_undegraded":"2024-04-26T23:58:56.247285+0000","last_fullsized":"2024-04-26T23:58:56.247285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:52.810401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.213133+0000","last_change":"2024-04-26T23:58:51.165470+0000","last_active":"2024-04-26T23:58:56.213133+0000","last_peered":"2024-04-26T23:58:56.213133+0000","last_clean":"2024-04-26T23:58:56.213133+0000","last_became_active":"2024-04-26T23:58:51.164998+0000","last_became_peered":"2024-04-26T23:58:51.164998+0000","last_unstale":"2024-04-26T23:58:56.213133+0000","last_undegraded":"2024-04-26T23:58:56.213133+0000","last_fullsized":"2024-04-26T23:58:56.213133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:02.703215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208735+0000","last_change":"2024-04-26T23:58:51.166390+0000","last_active":"2024-04-26T23:58:56.208735+0000","last_peered":"2024-04-26T23:58:56.208735+0000","last_clean":"2024-04-26T23:58:56.208735+0000","last_became_active":"2024-04-26T23:58:51.166113+0000","last_became_peered":"2024-04-26T23:58:51.166113+0000","last_unstale":"2024-04-26T23:58:56.208735+0000","last_undegraded":"2024-04-26T23:58:56.208735+0000","last_fullsized":"2024-04-26T23:58:56.208735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:24:49.676207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211288+0000","last_change":"2024-04-26T23:58:51.165272+0000","last_active":"2024-04-26T23:58:56.211288+0000","last_peered":"2024-04-26T23:58:56.211288+0000","last_clean":"2024-04-26T23:58:56.211288+0000","last_became_active":"2024-04-26T23:58:51.160355+0000","last_became_peered":"2024-04-26T23:58:51.160355+0000","last_unstale":"2024-04-26T23:58:56.211288+0000","last_undegraded":"2024-04-26T23:58:56.211288+0000","last_fullsized":"2024-04-26T23:58:56.211288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:32.910129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.246420+0000","last_change":"2024-04-26T23:58:51.159564+0000","last_active":"2024-04-26T23:58:56.246420+0000","last_peered":"2024-04-26T23:58:56.246420+0000","last_clean":"2024-04-26T23:58:56.246420+0000","last_became_active":"2024-04-26T23:58:51.159229+0000","last_became_peered":"2024-04-26T23:58:51.159229+0000","last_unstale":"2024-04-26T23:58:56.246420+0000","last_undegraded":"2024-04-26T23:58:56.246420+0000","last_fullsized":"2024-04-26T23:58:56.246420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:39.032595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.251450+0000","last_change":"2024-04-26T23:58:51.163868+0000","last_active":"2024-04-26T23:58:56.251450+0000","last_peered":"2024-04-26T23:58:56.251450+0000","last_clean":"2024-04-26T23:58:56.251450+0000","last_became_active":"2024-04-26T23:58:51.163450+0000","last_became_peered":"2024-04-26T23:58:51.163450+0000","last_unstale":"2024-04-26T23:58:56.251450+0000","last_undegraded":"2024-04-26T23:58:56.251450+0000","last_fullsized":"2024-04-26T23:58:56.251450+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:42:06.945605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.247845+0000","last_change":"2024-04-26T23:58:51.136047+0000","last_active":"2024-04-26T23:58:56.247845+0000","last_peered":"2024-04-26T23:58:56.247845+0000","last_clean":"2024-04-26T23:58:56.247845+0000","last_became_active":"2024-04-26T23:58:51.135690+0000","last_became_peered":"2024-04-26T23:58:51.135690+0000","last_unstale":"2024-04-26T23:58:56.247845+0000","last_undegraded":"2024-04-26T23:58:56.247845+0000","last_fullsized":"2024-04-26T23:58:56.247845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:55.721858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14443,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":28,"num_read_kb":27,"num_write":134,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":75575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350997,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":484944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612633,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1630208,"data_stored":1076054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776159,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":481084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037799,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":485100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299435,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1605632,"data_stored":1077304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462962,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":480356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":518073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":481744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":492449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":480460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888126,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":480910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724599,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1072110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":41906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":12183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:59:29.949 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T23:59:29.950 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T23:59:29.950 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T23:59:30.461 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:30.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:30 smithi040 ceph-mon[28095]: pgmap v342: 129 pgs: 129 active+clean; 591 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.9 KiB/s wr, 9 op/s 2024-04-26T23:59:30.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:30 smithi040 ceph-mon[28095]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:59:30.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:30 smithi080 ceph-mon[32362]: pgmap v342: 129 pgs: 129 active+clean; 591 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.9 KiB/s wr, 9 op/s 2024-04-26T23:59:30.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:30 smithi080 ceph-mon[32362]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:59:30.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:30 smithi103 ceph-mon[32952]: pgmap v342: 129 pgs: 129 active+clean; 591 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.9 KiB/s wr, 9 op/s 2024-04-26T23:59:30.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:30 smithi103 ceph-mon[32952]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:59:31.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:31 smithi040 ceph-mon[28095]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:59:31.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:31 smithi080 ceph-mon[32362]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:59:31.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:31 smithi103 ceph-mon[32952]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:59:32.686 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:32 smithi040 ceph-mon[28095]: pgmap v343: 129 pgs: 129 active+clean; 591 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-26T23:59:32.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:32 smithi080 ceph-mon[32362]: pgmap v343: 129 pgs: 129 active+clean; 591 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-26T23:59:32.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:32 smithi103 ceph-mon[32952]: pgmap v343: 129 pgs: 129 active+clean; 591 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-04-26T23:59:32.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.870+0000 7f6ae2c00700 1 -- 172.21.15.40:0/2084822916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc10b380 msgr2=0x7f6adc10d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:32.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.870+0000 7f6ae2c00700 1 --2- 172.21.15.40:0/2084822916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc10b380 0x7f6adc10d810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400b0a0 tx=0x7f6ad400b3b0 comp rx=0 tx=0).stop 2024-04-26T23:59:32.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.870+0000 7f6ae2c00700 1 -- 172.21.15.40:0/2084822916 shutdown_connections 2024-04-26T23:59:32.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.870+0000 7f6ae2c00700 1 --2- 172.21.15.40:0/2084822916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc10b380 0x7f6adc10d810 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:32.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.870+0000 7f6ae2c00700 1 --2- 172.21.15.40:0/2084822916 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6adc077450 0x7f6adc0758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:32.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.870+0000 7f6ae2c00700 1 --2- 172.21.15.40:0/2084822916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6adc075e80 0x7f6adc076300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:32.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.870+0000 7f6ae2c00700 1 -- 172.21.15.40:0/2084822916 >> 172.21.15.40:0/2084822916 conn(0x7f6adc0fea50 msgr2=0x7f6adc100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:32.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.870+0000 7f6ae2c00700 1 -- 172.21.15.40:0/2084822916 shutdown_connections 2024-04-26T23:59:32.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.871+0000 7f6ae2c00700 1 -- 172.21.15.40:0/2084822916 wait complete. 2024-04-26T23:59:32.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.872+0000 7f6ae2c00700 1 Processor -- start 2024-04-26T23:59:32.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.872+0000 7f6ae2c00700 1 -- start start 2024-04-26T23:59:32.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.872+0000 7f6ae2c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6adc075e80 0x7f6adc19b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:32.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.872+0000 7f6ae2c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc077450 0x7f6adc19bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:32.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.872+0000 7f6ae2c00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6adc10b380 0x7f6adc1a2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:32.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.872+0000 7f6ae2c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc111190 con 0x7f6adc077450 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.872+0000 7f6ae2c00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc111010 con 0x7f6adc10b380 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.872+0000 7f6ae2c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc111310 con 0x7f6adc075e80 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.872+0000 7f6ad3400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc077450 0x7f6adc19bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.873+0000 7f6ad3400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc077450 0x7f6adc19bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55780/0 (socket says 172.21.15.40:55780) 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.873+0000 7f6ad3400700 1 -- 172.21.15.40:0/1268010332 learned_addr learned my addr 172.21.15.40:0/1268010332 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.873+0000 7f6ae0c00700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6adc10b380 0x7f6adc1a2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.873+0000 7f6adbe00700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6adc075e80 0x7f6adc19b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.873+0000 7f6ad3400700 1 -- 172.21.15.40:0/1268010332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6adc075e80 msgr2=0x7f6adc19b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.873+0000 7f6ad3400700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6adc075e80 0x7f6adc19b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.873+0000 7f6ad3400700 1 -- 172.21.15.40:0/1268010332 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6adc10b380 msgr2=0x7f6adc1a2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.873+0000 7f6ad3400700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6adc10b380 0x7f6adc1a2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.873+0000 7f6ad3400700 1 -- 172.21.15.40:0/1268010332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6adc1a3340 con 0x7f6adc077450 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.873+0000 7f6ae0c00700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6adc10b380 0x7f6adc1a2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.873+0000 7f6ad3400700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc077450 0x7f6adc19bc20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400e9a0 tx=0x7f6ac400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:32.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.874+0000 7f6ad9600700 1 -- 172.21.15.40:0/1268010332 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4004220 con 0x7f6adc077450 2024-04-26T23:59:32.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.874+0000 7f6ad9600700 1 -- 172.21.15.40:0/1268010332 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ac40043c0 con 0x7f6adc077450 2024-04-26T23:59:32.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.874+0000 7f6ae2c00700 1 -- 172.21.15.40:0/1268010332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6adc1a3630 con 0x7f6adc077450 2024-04-26T23:59:32.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.874+0000 7f6ad9600700 1 -- 172.21.15.40:0/1268010332 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4003cc0 con 0x7f6adc077450 2024-04-26T23:59:32.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.874+0000 7f6ae2c00700 1 -- 172.21.15.40:0/1268010332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6adc1a3b70 con 0x7f6adc077450 2024-04-26T23:59:32.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.875+0000 7f6ae2c00700 1 -- 172.21.15.40:0/1268010332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6adc051960 con 0x7f6adc077450 2024-04-26T23:59:32.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.877+0000 7f6ad9600700 1 -- 172.21.15.40:0/1268010332 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6ac400c810 con 0x7f6adc077450 2024-04-26T23:59:32.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.879+0000 7f6ad9600700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6ac00a3d80 0x7f6ac00a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:32.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.879+0000 7f6ad9600700 1 -- 172.21.15.40:0/1268010332 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f6ac40d0a00 con 0x7f6adc077450 2024-04-26T23:59:32.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.879+0000 7f6adbe00700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6ac00a3d80 0x7f6ac00a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:32.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.880+0000 7f6adbe00700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6ac00a3d80 0x7f6ac00a6240 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6adc077830 tx=0x7f6acc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:32.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:32.883+0000 7f6ad9600700 1 -- 172.21.15.40:0/1268010332 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6ac409cfe0 con 0x7f6adc077450 2024-04-26T23:59:33.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.209+0000 7f6ae2c00700 1 -- 172.21.15.40:0/1268010332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f6adc02e4c0 con 0x7f6adc077450 2024-04-26T23:59:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.214+0000 7f6ad9600700 1 -- 172.21.15.40:0/1268010332 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f6ac409c8e0 con 0x7f6adc077450 2024-04-26T23:59:33.216 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-04-26T23:59:33.216 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDlPyxmpxGjDBAABt4IxcjpGTOjOk7Wfi5zNg== 2024-04-26T23:59:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.217+0000 7f6ae2c00700 1 -- 172.21.15.40:0/1268010332 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6ac00a3d80 msgr2=0x7f6ac00a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.217+0000 7f6ae2c00700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6ac00a3d80 0x7f6ac00a6240 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6adc077830 tx=0x7f6acc009260 comp rx=0 tx=0).stop 2024-04-26T23:59:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.217+0000 7f6ae2c00700 1 -- 172.21.15.40:0/1268010332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc077450 msgr2=0x7f6adc19bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.217+0000 7f6ae2c00700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc077450 0x7f6adc19bc20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400e9a0 tx=0x7f6ac400ecb0 comp rx=0 tx=0).stop 2024-04-26T23:59:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.218+0000 7f6ae2c00700 1 -- 172.21.15.40:0/1268010332 shutdown_connections 2024-04-26T23:59:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.218+0000 7f6ae2c00700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6ac00a3d80 0x7f6ac00a6240 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.218+0000 7f6ae2c00700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6adc10b380 0x7f6adc1a2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.218+0000 7f6ae2c00700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6adc075e80 0x7f6adc19b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.218+0000 7f6ae2c00700 1 --2- 172.21.15.40:0/1268010332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6adc077450 0x7f6adc19bc20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.218+0000 7f6ae2c00700 1 -- 172.21.15.40:0/1268010332 >> 172.21.15.40:0/1268010332 conn(0x7f6adc0fea50 msgr2=0x7f6adc107280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.218+0000 7f6ae2c00700 1 -- 172.21.15.40:0/1268010332 shutdown_connections 2024-04-26T23:59:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:33.218+0000 7f6ae2c00700 1 -- 172.21.15.40:0/1268010332 wait complete. 2024-04-26T23:59:33.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:33 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1268010332' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T23:59:33.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:33 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1268010332' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T23:59:33.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:33 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1268010332' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T23:59:33.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:33 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1268010332' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T23:59:33.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:33 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1268010332' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T23:59:33.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:33 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1268010332' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T23:59:33.999 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-26T23:59:34.000 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T23:59:34.000 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T23:59:34.152 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T23:59:34.152 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T23:59:34.153 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T23:59:34.666 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:34.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:34 smithi040 ceph-mon[28095]: pgmap v344: 129 pgs: 129 active+clean; 626 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2024-04-26T23:59:34.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:34 smithi080 ceph-mon[32362]: pgmap v344: 129 pgs: 129 active+clean; 626 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2024-04-26T23:59:34.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:34 smithi103 ceph-mon[32952]: pgmap v344: 129 pgs: 129 active+clean; 626 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2024-04-26T23:59:36.680 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:36 smithi040 ceph-mon[28095]: pgmap v345: 129 pgs: 129 active+clean; 626 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2024-04-26T23:59:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:36 smithi080 ceph-mon[32362]: pgmap v345: 129 pgs: 129 active+clean; 626 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2024-04-26T23:59:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:36 smithi103 ceph-mon[32952]: pgmap v345: 129 pgs: 129 active+clean; 626 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2024-04-26T23:59:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.110+0000 7f51c6000700 1 -- 172.21.15.40:0/3250143374 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51c0103470 msgr2=0x7f51c01038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.110+0000 7f51c6000700 1 --2- 172.21.15.40:0/3250143374 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51c0103470 0x7f51c01038f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f51a8009990 tx=0x7f51a8009ca0 comp rx=0 tx=0).stop 2024-04-26T23:59:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.110+0000 7f51c6000700 1 -- 172.21.15.40:0/3250143374 shutdown_connections 2024-04-26T23:59:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.110+0000 7f51c6000700 1 --2- 172.21.15.40:0/3250143374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51c0103e30 0x7f51c0110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.110+0000 7f51c6000700 1 --2- 172.21.15.40:0/3250143374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51c01094c0 0x7f51c01098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.110+0000 7f51c6000700 1 --2- 172.21.15.40:0/3250143374 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51c0103470 0x7f51c01038f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.110+0000 7f51c6000700 1 -- 172.21.15.40:0/3250143374 >> 172.21.15.40:0/3250143374 conn(0x7f51c007a580 msgr2=0x7f51c007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:37.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.110+0000 7f51c6000700 1 -- 172.21.15.40:0/3250143374 shutdown_connections 2024-04-26T23:59:37.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.110+0000 7f51c6000700 1 -- 172.21.15.40:0/3250143374 wait complete. 2024-04-26T23:59:37.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.111+0000 7f51c6000700 1 Processor -- start 2024-04-26T23:59:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.111+0000 7f51c6000700 1 -- start start 2024-04-26T23:59:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51c6000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51c0103470 0x7f51c0075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51c6000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51c0103e30 0x7f51c0075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51c6000700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51c01094c0 0x7f51c00761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51c6000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f51c01135c0 con 0x7f51c0103470 2024-04-26T23:59:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51c6000700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f51c0113440 con 0x7f51c01094c0 2024-04-26T23:59:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51c6000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51c0113740 con 0x7f51c0103e30 2024-04-26T23:59:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51bf400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51c0103470 0x7f51c0075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51bf400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51c0103470 0x7f51c0075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55792/0 (socket says 172.21.15.40:55792) 2024-04-26T23:59:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51bf400700 1 -- 172.21.15.40:0/1578978501 learned_addr learned my addr 172.21.15.40:0/1578978501 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:37.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51b6a00700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51c0103e30 0x7f51c0075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:37.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51bfe00700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51c01094c0 0x7f51c00761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:37.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51bf400700 1 -- 172.21.15.40:0/1578978501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51c0103e30 msgr2=0x7f51c0075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:37.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51bf400700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51c0103e30 0x7f51c0075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:37.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51bf400700 1 -- 172.21.15.40:0/1578978501 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51c01094c0 msgr2=0x7f51c00761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51bf400700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51c01094c0 0x7f51c00761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.112+0000 7f51bf400700 1 -- 172.21.15.40:0/1578978501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c01a8060 con 0x7f51c0103470 2024-04-26T23:59:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.113+0000 7f51bf400700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51c0103470 0x7f51c0075720 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f51b000ef10 tx=0x7f51b000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.113+0000 7f51bcc00700 1 -- 172.21.15.40:0/1578978501 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b000c9c0 con 0x7f51c0103470 2024-04-26T23:59:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.113+0000 7f51bcc00700 1 -- 172.21.15.40:0/1578978501 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51b0019070 con 0x7f51c0103470 2024-04-26T23:59:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.113+0000 7f51c6000700 1 -- 172.21.15.40:0/1578978501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51c01a82f0 con 0x7f51c0103470 2024-04-26T23:59:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.113+0000 7f51bcc00700 1 -- 172.21.15.40:0/1578978501 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0014480 con 0x7f51c0103470 2024-04-26T23:59:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.113+0000 7f51c6000700 1 -- 172.21.15.40:0/1578978501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51c01a8800 con 0x7f51c0103470 2024-04-26T23:59:37.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.115+0000 7f51c6000700 1 -- 172.21.15.40:0/1578978501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51c006a110 con 0x7f51c0103470 2024-04-26T23:59:37.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.119+0000 7f51bcc00700 1 -- 172.21.15.40:0/1578978501 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f51b0012050 con 0x7f51c0103470 2024-04-26T23:59:37.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.120+0000 7f51bcc00700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f51a40a3cd0 0x7f51a40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:37.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.120+0000 7f51bcc00700 1 -- 172.21.15.40:0/1578978501 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f51b00d0dd0 con 0x7f51c0103470 2024-04-26T23:59:37.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.120+0000 7f51b6a00700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f51a40a3cd0 0x7f51a40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:37.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.121+0000 7f51b6a00700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f51a40a3cd0 0x7f51a40a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f51a80098e0 tx=0x7f51a8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:37.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.123+0000 7f51bcc00700 1 -- 172.21.15.40:0/1578978501 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f51b0010040 con 0x7f51c0103470 2024-04-26T23:59:37.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.417+0000 7f51c6000700 1 -- 172.21.15.40:0/1578978501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f51c002e490 con 0x7f51c0103470 2024-04-26T23:59:37.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.430+0000 7f51bcc00700 1 -- 172.21.15.40:0/1578978501 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264998 (secure 0 0 0) 0x7f51b009d3b0 con 0x7f51c0103470 2024-04-26T23:59:37.437 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:59:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.435+0000 7f51c6000700 1 -- 172.21.15.40:0/1578978501 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f51a40a3cd0 msgr2=0x7f51a40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.435+0000 7f51c6000700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f51a40a3cd0 0x7f51a40a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f51a80098e0 tx=0x7f51a8005960 comp rx=0 tx=0).stop 2024-04-26T23:59:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.435+0000 7f51c6000700 1 -- 172.21.15.40:0/1578978501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51c0103470 msgr2=0x7f51c0075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.435+0000 7f51c6000700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51c0103470 0x7f51c0075720 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f51b000ef10 tx=0x7f51b000c3a0 comp rx=0 tx=0).stop 2024-04-26T23:59:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.436+0000 7f51c6000700 1 -- 172.21.15.40:0/1578978501 shutdown_connections 2024-04-26T23:59:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.436+0000 7f51c6000700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f51a40a3cd0 0x7f51a40a6190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.436+0000 7f51c6000700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51c01094c0 0x7f51c00761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.436+0000 7f51c6000700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51c0103470 0x7f51c0075720 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.436+0000 7f51c6000700 1 --2- 172.21.15.40:0/1578978501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51c0103e30 0x7f51c0075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.436+0000 7f51c6000700 1 -- 172.21.15.40:0/1578978501 >> 172.21.15.40:0/1578978501 conn(0x7f51c007a580 msgr2=0x7f51c01021a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.436+0000 7f51c6000700 1 -- 172.21.15.40:0/1578978501 shutdown_connections 2024-04-26T23:59:37.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:37.436+0000 7f51c6000700 1 -- 172.21.15.40:0/1578978501 wait complete. 2024-04-26T23:59:37.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:37 smithi040 ceph-mon[28095]: pgmap v346: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.2 KiB/s wr, 16 op/s 2024-04-26T23:59:37.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:37 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1578978501' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T23:59:38.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:37 smithi080 ceph-mon[32362]: pgmap v346: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.2 KiB/s wr, 16 op/s 2024-04-26T23:59:38.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:37 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1578978501' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T23:59:38.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:37 smithi103 ceph-mon[32952]: pgmap v346: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.2 KiB/s wr, 16 op/s 2024-04-26T23:59:38.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:37 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1578978501' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T23:59:38.240 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3777720410},{"type":"v1","addr":"172.21.15.40:6801","nonce":3777720410}]},"active_addr":"172.21.15.40:6801/3777720410","active_change":"2024-04-26T23:50:44.546680+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.40:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2940103588}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":3266558063}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2451356116}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2409110486}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":3115468040}]}]} 2024-04-26T23:59:38.245 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T23:59:38.245 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T23:59:38.245 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T23:59:38.682 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:40.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:39 smithi080 ceph-mon[32362]: pgmap v347: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.1 KiB/s wr, 10 op/s 2024-04-26T23:59:40.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:39 smithi103 ceph-mon[32952]: pgmap v347: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.1 KiB/s wr, 10 op/s 2024-04-26T23:59:40.138 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:39 smithi040 ceph-mon[28095]: pgmap v347: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.1 KiB/s wr, 10 op/s 2024-04-26T23:59:40.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.981+0000 7f09d3c00700 1 -- 172.21.15.40:0/4132901363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc114300 msgr2=0x7f09cc1166f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:40.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.981+0000 7f09d3c00700 1 --2- 172.21.15.40:0/4132901363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc114300 0x7f09cc1166f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f09c400b0a0 tx=0x7f09c400b3b0 comp rx=0 tx=0).stop 2024-04-26T23:59:40.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.981+0000 7f09d3c00700 1 -- 172.21.15.40:0/4132901363 shutdown_connections 2024-04-26T23:59:40.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.981+0000 7f09d3c00700 1 --2- 172.21.15.40:0/4132901363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc114300 0x7f09cc1166f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:40.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.981+0000 7f09d3c00700 1 --2- 172.21.15.40:0/4132901363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09cc077950 0x7f09cc077d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:40.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.981+0000 7f09d3c00700 1 --2- 172.21.15.40:0/4132901363 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09cc078270 0x7f09cc113dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:40.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.981+0000 7f09d3c00700 1 -- 172.21.15.40:0/4132901363 >> 172.21.15.40:0/4132901363 conn(0x7f09cc101150 msgr2=0x7f09cc103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:40.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.981+0000 7f09d3c00700 1 -- 172.21.15.40:0/4132901363 shutdown_connections 2024-04-26T23:59:40.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.981+0000 7f09d3c00700 1 -- 172.21.15.40:0/4132901363 wait complete. 2024-04-26T23:59:40.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.982+0000 7f09d3c00700 1 Processor -- start 2024-04-26T23:59:40.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.982+0000 7f09d3c00700 1 -- start start 2024-04-26T23:59:40.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09d3c00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09cc077950 0x7f09cc1a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:40.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09d3c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc078270 0x7f09cc1a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:40.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09d3c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09cc114300 0x7f09cc1a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:40.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09d3c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc119c80 con 0x7f09cc078270 2024-04-26T23:59:40.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09d3c00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc119b00 con 0x7f09cc077950 2024-04-26T23:59:40.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09d3c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc119e00 con 0x7f09cc114300 2024-04-26T23:59:40.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09cbe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc078270 0x7f09cc1a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:40.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09cbe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc078270 0x7f09cc1a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37134/0 (socket says 172.21.15.40:37134) 2024-04-26T23:59:40.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09cbe00700 1 -- 172.21.15.40:0/4128844654 learned_addr learned my addr 172.21.15.40:0/4128844654 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:40.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09d1200700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09cc077950 0x7f09cc1a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:40.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09d1c00700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09cc114300 0x7f09cc1a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:40.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09cbe00700 1 -- 172.21.15.40:0/4128844654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09cc114300 msgr2=0x7f09cc1a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:40.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09cbe00700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09cc114300 0x7f09cc1a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:40.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.983+0000 7f09cbe00700 1 -- 172.21.15.40:0/4128844654 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09cc077950 msgr2=0x7f09cc1a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:40.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.984+0000 7f09cbe00700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09cc077950 0x7f09cc1a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:40.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.984+0000 7f09cbe00700 1 -- 172.21.15.40:0/4128844654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc1a51c0 con 0x7f09cc078270 2024-04-26T23:59:40.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.984+0000 7f09d1c00700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09cc114300 0x7f09cc1a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:40.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.984+0000 7f09d1200700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09cc077950 0x7f09cc1a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:40.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.984+0000 7f09cbe00700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc078270 0x7f09cc1a06f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f09b800eb30 tx=0x7f09b800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:40.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.984+0000 7f09c9600700 1 -- 172.21.15.40:0/4128844654 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b800cbd0 con 0x7f09cc078270 2024-04-26T23:59:40.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.984+0000 7f09c9600700 1 -- 172.21.15.40:0/4128844654 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09b80044d0 con 0x7f09cc078270 2024-04-26T23:59:40.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.984+0000 7f09d3c00700 1 -- 172.21.15.40:0/4128844654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09cc1a54b0 con 0x7f09cc078270 2024-04-26T23:59:40.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.984+0000 7f09c9600700 1 -- 172.21.15.40:0/4128844654 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b8003e30 con 0x7f09cc078270 2024-04-26T23:59:40.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.984+0000 7f09d3c00700 1 -- 172.21.15.40:0/4128844654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09cc1acd30 con 0x7f09cc078270 2024-04-26T23:59:40.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.986+0000 7f09d3c00700 1 -- 172.21.15.40:0/4128844654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09cc06a110 con 0x7f09cc078270 2024-04-26T23:59:40.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.988+0000 7f09c9600700 1 -- 172.21.15.40:0/4128844654 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f09b8004000 con 0x7f09cc078270 2024-04-26T23:59:40.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.989+0000 7f09c9600700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f09b40a3d50 0x7f09b40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:40.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.989+0000 7f09c9600700 1 -- 172.21.15.40:0/4128844654 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f09b80d0a80 con 0x7f09cc078270 2024-04-26T23:59:40.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.990+0000 7f09d1200700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f09b40a3d50 0x7f09b40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:40.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.990+0000 7f09d1200700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f09b40a3d50 0x7f09b40a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f09c0011490 tx=0x7f09c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:40.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:40.994+0000 7f09c9600700 1 -- 172.21.15.40:0/4128844654 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f09b80d2e20 con 0x7f09cc078270 2024-04-26T23:59:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.230+0000 7f09d3c00700 1 -- 172.21.15.40:0/4128844654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09cc1a1540 con 0x7f09cc078270 2024-04-26T23:59:41.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.233+0000 7f09c9600700 1 -- 172.21.15.40:0/4128844654 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24090 (secure 0 0 0) 0x7f09cc1a1540 con 0x7f09cc078270 2024-04-26T23:59:41.235 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:59:41.235 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:58:55.649955+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:59:41.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.236+0000 7f09d3c00700 1 -- 172.21.15.40:0/4128844654 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f09b40a3d50 msgr2=0x7f09b40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:41.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.236+0000 7f09d3c00700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f09b40a3d50 0x7f09b40a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f09c0011490 tx=0x7f09c0009260 comp rx=0 tx=0).stop 2024-04-26T23:59:41.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.236+0000 7f09d3c00700 1 -- 172.21.15.40:0/4128844654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc078270 msgr2=0x7f09cc1a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:41.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.236+0000 7f09d3c00700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc078270 0x7f09cc1a06f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f09b800eb30 tx=0x7f09b800ee40 comp rx=0 tx=0).stop 2024-04-26T23:59:41.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.237+0000 7f09d3c00700 1 -- 172.21.15.40:0/4128844654 shutdown_connections 2024-04-26T23:59:41.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.237+0000 7f09d3c00700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f09b40a3d50 0x7f09b40a6210 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:41.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.237+0000 7f09d3c00700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09cc114300 0x7f09cc1a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:41.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.237+0000 7f09d3c00700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09cc077950 0x7f09cc1a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:41.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.237+0000 7f09d3c00700 1 --2- 172.21.15.40:0/4128844654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc078270 0x7f09cc1a06f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:41.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.237+0000 7f09d3c00700 1 -- 172.21.15.40:0/4128844654 >> 172.21.15.40:0/4128844654 conn(0x7f09cc101150 msgr2=0x7f09cc103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:41.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.237+0000 7f09d3c00700 1 -- 172.21.15.40:0/4128844654 shutdown_connections 2024-04-26T23:59:41.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:41.237+0000 7f09d3c00700 1 -- 172.21.15.40:0/4128844654 wait complete. 2024-04-26T23:59:41.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:41 smithi040 ceph-mon[28095]: pgmap v348: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.2 KiB/s wr, 7 op/s 2024-04-26T23:59:41.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:41 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/4128844654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:42.031 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T23:59:42.031 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T23:59:42.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:41 smithi080 ceph-mon[32362]: pgmap v348: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.2 KiB/s wr, 7 op/s 2024-04-26T23:59:42.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:41 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/4128844654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:42.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:41 smithi103 ceph-mon[32952]: pgmap v348: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.2 KiB/s wr, 7 op/s 2024-04-26T23:59:42.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:41 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/4128844654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:43.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:43 smithi040 ceph-mon[28095]: pgmap v349: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2024-04-26T23:59:44.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:43 smithi080 ceph-mon[32362]: pgmap v349: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2024-04-26T23:59:44.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:43 smithi103 ceph-mon[32952]: pgmap v349: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2024-04-26T23:59:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.766+0000 7f839e600700 1 -- 172.21.15.40:0/2054450047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8398105640 msgr2=0x7f8398105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.766+0000 7f839e600700 1 --2- 172.21.15.40:0/2054450047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8398105640 0x7f8398105ac0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8384009990 tx=0x7f8384009ca0 comp rx=0 tx=0).stop 2024-04-26T23:59:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.767+0000 7f839e600700 1 -- 172.21.15.40:0/2054450047 shutdown_connections 2024-04-26T23:59:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.767+0000 7f839e600700 1 --2- 172.21.15.40:0/2054450047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8398106000 0x7f8398112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.767+0000 7f839e600700 1 --2- 172.21.15.40:0/2054450047 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f839810b640 0x7f839810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.767+0000 7f839e600700 1 --2- 172.21.15.40:0/2054450047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8398105640 0x7f8398105ac0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.767+0000 7f839e600700 1 -- 172.21.15.40:0/2054450047 >> 172.21.15.40:0/2054450047 conn(0x7f8398101130 msgr2=0x7f8398103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.767+0000 7f839e600700 1 -- 172.21.15.40:0/2054450047 shutdown_connections 2024-04-26T23:59:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.767+0000 7f839e600700 1 -- 172.21.15.40:0/2054450047 wait complete. 2024-04-26T23:59:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.768+0000 7f839e600700 1 Processor -- start 2024-04-26T23:59:44.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.768+0000 7f839e600700 1 -- start start 2024-04-26T23:59:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.769+0000 7f839e600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8398105640 0x7f839819bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.769+0000 7f839e600700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8398106000 0x7f839819c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.769+0000 7f839e600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839810b640 0x7f83981a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.769+0000 7f839e600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8398115890 con 0x7f8398105640 2024-04-26T23:59:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.769+0000 7f839e600700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f8398115710 con 0x7f8398106000 2024-04-26T23:59:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.769+0000 7f839e600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8398115a10 con 0x7f839810b640 2024-04-26T23:59:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.769+0000 7f8397400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8398105640 0x7f839819bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.769+0000 7f8396a00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8398106000 0x7f839819c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.769+0000 7f8397400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8398105640 0x7f839819bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37166/0 (socket says 172.21.15.40:37166) 2024-04-26T23:59:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.769+0000 7f8397400700 1 -- 172.21.15.40:0/2498391552 learned_addr learned my addr 172.21.15.40:0/2498391552 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:44.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.769+0000 7f8397e00700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839810b640 0x7f83981a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f8397400700 1 -- 172.21.15.40:0/2498391552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839810b640 msgr2=0x7f83981a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f8397400700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839810b640 0x7f83981a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f8397400700 1 -- 172.21.15.40:0/2498391552 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8398106000 msgr2=0x7f839819c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f8397400700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8398106000 0x7f839819c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f8397400700 1 -- 172.21.15.40:0/2498391552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83981a0d20 con 0x7f8398105640 2024-04-26T23:59:44.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f8397e00700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839810b640 0x7f83981a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f8396a00700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8398106000 0x7f839819c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f8397400700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8398105640 0x7f839819bd60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f838800e9a0 tx=0x7f838800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f838fe00700 1 -- 172.21.15.40:0/2498391552 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838800ca10 con 0x7f8398105640 2024-04-26T23:59:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f838fe00700 1 -- 172.21.15.40:0/2498391552 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f838800cbb0 con 0x7f8398105640 2024-04-26T23:59:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f839e600700 1 -- 172.21.15.40:0/2498391552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83981a1010 con 0x7f8398105640 2024-04-26T23:59:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f838fe00700 1 -- 172.21.15.40:0/2498391552 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388018680 con 0x7f8398105640 2024-04-26T23:59:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.770+0000 7f839e600700 1 -- 172.21.15.40:0/2498391552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83981a8890 con 0x7f8398105640 2024-04-26T23:59:44.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.772+0000 7f839e600700 1 -- 172.21.15.40:0/2498391552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f839806a110 con 0x7f8398105640 2024-04-26T23:59:44.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.774+0000 7f838fe00700 1 -- 172.21.15.40:0/2498391552 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8388010430 con 0x7f8398105640 2024-04-26T23:59:44.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.775+0000 7f838fe00700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f837c0a3d50 0x7f837c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:44.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.775+0000 7f838fe00700 1 -- 172.21.15.40:0/2498391552 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f83880d08c0 con 0x7f8398105640 2024-04-26T23:59:44.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.775+0000 7f8396a00700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f837c0a3d50 0x7f837c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:44.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.776+0000 7f8396a00700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f837c0a3d50 0x7f837c0a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8398103700 tx=0x7f838401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:44.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:44.780+0000 7f838fe00700 1 -- 172.21.15.40:0/2498391552 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f838809cea0 con 0x7f8398105640 2024-04-26T23:59:45.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.012+0000 7f839e600700 1 -- 172.21.15.40:0/2498391552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f839802e490 con 0x7f8398105640 2024-04-26T23:59:45.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.016+0000 7f838fe00700 1 -- 172.21.15.40:0/2498391552 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24090 (secure 0 0 0) 0x7f838809c7a0 con 0x7f8398105640 2024-04-26T23:59:45.019 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-26T23:59:45.019 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-26T23:58:55.649955+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:59:45.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.020+0000 7f839e600700 1 -- 172.21.15.40:0/2498391552 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f837c0a3d50 msgr2=0x7f837c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.020+0000 7f839e600700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f837c0a3d50 0x7f837c0a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8398103700 tx=0x7f838401a040 comp rx=0 tx=0).stop 2024-04-26T23:59:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.020+0000 7f839e600700 1 -- 172.21.15.40:0/2498391552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8398105640 msgr2=0x7f839819bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.020+0000 7f839e600700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8398105640 0x7f839819bd60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f838800e9a0 tx=0x7f838800ecb0 comp rx=0 tx=0).stop 2024-04-26T23:59:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.020+0000 7f839e600700 1 -- 172.21.15.40:0/2498391552 shutdown_connections 2024-04-26T23:59:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.020+0000 7f839e600700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f837c0a3d50 0x7f837c0a6210 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:45.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.020+0000 7f839e600700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f839810b640 0x7f83981a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:45.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.020+0000 7f839e600700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8398105640 0x7f839819bd60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:45.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.020+0000 7f839e600700 1 --2- 172.21.15.40:0/2498391552 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8398106000 0x7f839819c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:45.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.020+0000 7f839e600700 1 -- 172.21.15.40:0/2498391552 >> 172.21.15.40:0/2498391552 conn(0x7f8398101130 msgr2=0x7f8398101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:45.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.020+0000 7f839e600700 1 -- 172.21.15.40:0/2498391552 shutdown_connections 2024-04-26T23:59:45.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:45.020+0000 7f839e600700 1 -- 172.21.15.40:0/2498391552 wait complete. 2024-04-26T23:59:45.873 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T23:59:45.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T23:59:45.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T23:59:45.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T23:59:45.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T23:59:45.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T23:59:45.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T23:59:45.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T23:59:45.877 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-26T23:59:45.877 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-26T23:59:45.878 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-26T23:59:45.878 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-26T23:59:46.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:45 smithi080 ceph-mon[32362]: pgmap v350: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T23:59:46.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:45 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2498391552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:46.053 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:45 smithi040 ceph-mon[28095]: pgmap v350: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T23:59:46.054 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:45 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2498391552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:46.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:45 smithi103 ceph-mon[32952]: pgmap v350: 129 pgs: 129 active+clean; 630 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T23:59:46.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:45 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2498391552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:59:46.935 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:46.954 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:46.968 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:46.993 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:47.904 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:47 smithi040 ceph-mon[28095]: pgmap v351: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-26T23:59:48.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:47 smithi080 ceph-mon[32362]: pgmap v351: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-26T23:59:48.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:47 smithi103 ceph-mon[32952]: pgmap v351: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-26T23:59:49.462 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:49.463 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:49.463 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:49.558 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:49.560 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:49.570 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:49.900 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:49.902 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:59:50.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:49 smithi080 ceph-mon[32362]: pgmap v352: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T23:59:50.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:49 smithi103 ceph-mon[32952]: pgmap v352: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T23:59:50.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:49 smithi040 ceph-mon[28095]: pgmap v352: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T23:59:51.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.577+0000 7fbfaca00700 1 -- 172.21.15.40:0/3735756830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfa810fe20 msgr2=0x7fbfa8112210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:51.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.577+0000 7fbfaca00700 1 --2- 172.21.15.40:0/3735756830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfa810fe20 0x7fbfa8112210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000b0a0 tx=0x7fbfa000b3b0 comp rx=0 tx=0).stop 2024-04-26T23:59:51.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.578+0000 7fbfaca00700 1 -- 172.21.15.40:0/3735756830 shutdown_connections 2024-04-26T23:59:51.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.578+0000 7fbfaca00700 1 --2- 172.21.15.40:0/3735756830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfa810fe20 0x7fbfa8112210 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:51.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.578+0000 7fbfaca00700 1 --2- 172.21.15.40:0/3735756830 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfa8075650 0x7fbfa8075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.578+0000 7fbfaca00700 1 --2- 172.21.15.40:0/3735756830 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfa8075f70 0x7fbfa810f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.578+0000 7fbfaca00700 1 -- 172.21.15.40:0/3735756830 >> 172.21.15.40:0/3735756830 conn(0x7fbfa807a580 msgr2=0x7fbfa807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.578+0000 7fbfaca00700 1 -- 172.21.15.40:0/3735756830 shutdown_connections 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.578+0000 7fbfaca00700 1 -- 172.21.15.40:0/3735756830 wait complete. 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.580+0000 7fbfaca00700 1 Processor -- start 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.580+0000 7fbfaca00700 1 -- start start 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.580+0000 7fbfaca00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfa8075650 0x7fbfa81a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.580+0000 7fbfaca00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfa8075f70 0x7fbfa81a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.580+0000 7fbfaca00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfa810fe20 0x7fbfa819baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.580+0000 7fbfaca00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa8115800 con 0x7fbfa8075650 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.580+0000 7fbfaca00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa8115680 con 0x7fbfa8075f70 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.580+0000 7fbfaca00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa8115980 con 0x7fbfa810fe20 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.581+0000 7fbfa5e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfa8075650 0x7fbfa81a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:51.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.581+0000 7fbfa5400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfa8075f70 0x7fbfa81a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.581+0000 7fbfa5e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfa8075650 0x7fbfa81a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41712/0 (socket says 172.21.15.40:41712) 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.581+0000 7fbfa5e00700 1 -- 172.21.15.40:0/2329506127 learned_addr learned my addr 172.21.15.40:0/2329506127 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.581+0000 7fbfa6800700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfa810fe20 0x7fbfa819baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.581+0000 7fbfa5e00700 1 -- 172.21.15.40:0/2329506127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfa810fe20 msgr2=0x7fbfa819baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.581+0000 7fbfa5e00700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfa810fe20 0x7fbfa819baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.581+0000 7fbfa5e00700 1 -- 172.21.15.40:0/2329506127 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfa8075f70 msgr2=0x7fbfa81a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.581+0000 7fbfa5e00700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfa8075f70 0x7fbfa81a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.581+0000 7fbfa5e00700 1 -- 172.21.15.40:0/2329506127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa819c200 con 0x7fbfa8075650 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.581+0000 7fbfa5400700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfa8075f70 0x7fbfa81a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.582+0000 7fbfa5e00700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfa8075650 0x7fbfa81a18e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800e9a0 tx=0x7fbf9800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.582+0000 7fbf9ea00700 1 -- 172.21.15.40:0/2329506127 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9800ca10 con 0x7fbfa8075650 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.582+0000 7fbfaca00700 1 -- 172.21.15.40:0/2329506127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa819c4f0 con 0x7fbfa8075650 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.582+0000 7fbf9ea00700 1 -- 172.21.15.40:0/2329506127 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf9800cbb0 con 0x7fbfa8075650 2024-04-26T23:59:51.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.582+0000 7fbf9ea00700 1 -- 172.21.15.40:0/2329506127 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf98018680 con 0x7fbfa8075650 2024-04-26T23:59:51.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.582+0000 7fbfaca00700 1 -- 172.21.15.40:0/2329506127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa81a87b0 con 0x7fbfa8075650 2024-04-26T23:59:51.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.583+0000 7fbfaca00700 1 -- 172.21.15.40:0/2329506127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbfa8063bb0 con 0x7fbfa8075650 2024-04-26T23:59:51.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.585+0000 7fbf9ea00700 1 -- 172.21.15.40:0/2329506127 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbf98010430 con 0x7fbfa8075650 2024-04-26T23:59:51.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.587+0000 7fbf9ea00700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbf8c0a3d50 0x7fbf8c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:51.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.587+0000 7fbf9ea00700 1 -- 172.21.15.40:0/2329506127 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fbf980d0ed0 con 0x7fbfa8075650 2024-04-26T23:59:51.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.588+0000 7fbfa5400700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbf8c0a3d50 0x7fbf8c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:51.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.588+0000 7fbf9ea00700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825] conn(0x7fbf8c0aea10 0x7fbf8c0b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:51.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.588+0000 7fbf9ea00700 1 -- 172.21.15.40:0/2329506127 --> [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf8c0b1520 con 0x7fbf8c0aea10 2024-04-26T23:59:51.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.588+0000 7fbf9ea00700 1 -- 172.21.15.40:0/2329506127 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fbf980d1280 con 0x7fbfa8075650 2024-04-26T23:59:51.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.588+0000 7fbfa6800700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825] conn(0x7fbf8c0aea10 0x7fbf8c0b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:51.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.588+0000 7fbfa5400700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbf8c0a3d50 0x7fbf8c0a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8111ae0 tx=0x7fbf94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:51.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.589+0000 7fbfa6800700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825] conn(0x7fbf8c0aea10 0x7fbf8c0b0e50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:51.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.590+0000 7fbf9ea00700 1 -- 172.21.15.40:0/2329506127 <== osd.9 v2:172.21.15.40:6826/1030365825 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbf8c0b1520 con 0x7fbf8c0aea10 2024-04-26T23:59:51.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.608+0000 7fbfaca00700 1 -- 172.21.15.40:0/2329506127 --> [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbfa8063fb0 con 0x7fbf8c0aea10 2024-04-26T23:59:51.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.609+0000 7fbf9ea00700 1 -- 172.21.15.40:0/2329506127 <== osd.9 v2:172.21.15.40:6826/1030365825 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbfa8063fb0 con 0x7fbf8c0aea10 2024-04-26T23:59:51.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.609+0000 7fbfaca00700 1 -- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825] conn(0x7fbf8c0aea10 msgr2=0x7fbf8c0b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:51.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.609+0000 7fbfaca00700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825] conn(0x7fbf8c0aea10 0x7fbf8c0b0e50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:51.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.609+0000 7fbfaca00700 1 -- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbf8c0a3d50 msgr2=0x7fbf8c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:51.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.609+0000 7fbfaca00700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbf8c0a3d50 0x7fbf8c0a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8111ae0 tx=0x7fbf94009260 comp rx=0 tx=0).stop 2024-04-26T23:59:51.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.609+0000 7fbfaca00700 1 -- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfa8075650 msgr2=0x7fbfa81a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:51.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.609+0000 7fbfaca00700 1 --2- 172.21.15.40:0/2329506127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfa8075650 0x7fbfa81a18e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800e9a0 tx=0x7fbf9800ecb0 comp rx=0 tx=0).stop 2024-04-26T23:59:51.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.609+0000 7fbfa6800700 1 -- 172.21.15.40:0/2329506127 reap_dead start 2024-04-26T23:59:51.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.610+0000 7fbfaca00700 1 -- 172.21.15.40:0/2329506127 shutdown_connections 2024-04-26T23:59:51.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.610+0000 7fbfaca00700 1 -- 172.21.15.40:0/2329506127 >> 172.21.15.40:0/2329506127 conn(0x7fbfa807a580 msgr2=0x7fbfa81005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:51.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.610+0000 7fbfaca00700 1 -- 172.21.15.40:0/2329506127 shutdown_connections 2024-04-26T23:59:51.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:51.610+0000 7fbfaca00700 1 -- 172.21.15.40:0/2329506127 wait complete. 2024-04-26T23:59:52.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:51 smithi103 ceph-mon[32952]: pgmap v353: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T23:59:52.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:51 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:59:52.135 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:51 smithi040 ceph-mon[28095]: pgmap v353: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T23:59:52.135 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:51 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:59:52.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:51 smithi080 ceph-mon[32362]: pgmap v353: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T23:59:52.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:51 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:59:54.379 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:54 smithi040 ceph-mon[28095]: pgmap v354: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T23:59:54.379 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:59:54.379 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:59:54.379 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:59:54.379 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:59:54.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:54 smithi080 ceph-mon[32362]: pgmap v354: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T23:59:54.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:54 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:59:54.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:54 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:59:54.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:54 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:59:54.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:54 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:59:54.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:54 smithi103 ceph-mon[32952]: pgmap v354: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-26T23:59:54.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:59:54.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:59:54.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:59:54.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-26T23:59:54.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.347+0000 7fc632800700 1 -- 172.21.15.40:0/3147112602 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc62c105640 msgr2=0x7fc62c105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:54.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.347+0000 7fc632800700 1 --2- 172.21.15.40:0/3147112602 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc62c105640 0x7fc62c105ac0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc6180098f0 tx=0x7fc618009c00 comp rx=0 tx=0).stop 2024-04-26T23:59:54.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.348+0000 7fc632800700 1 -- 172.21.15.40:0/3147112602 shutdown_connections 2024-04-26T23:59:54.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.348+0000 7fc632800700 1 --2- 172.21.15.40:0/3147112602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc62c106000 0x7fc62c112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:54.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.348+0000 7fc632800700 1 --2- 172.21.15.40:0/3147112602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62c10b640 0x7fc62c10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:54.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.348+0000 7fc632800700 1 --2- 172.21.15.40:0/3147112602 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc62c105640 0x7fc62c105ac0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:54.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.348+0000 7fc632800700 1 -- 172.21.15.40:0/3147112602 >> 172.21.15.40:0/3147112602 conn(0x7fc62c101130 msgr2=0x7fc62c103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:54.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.348+0000 7fc632800700 1 -- 172.21.15.40:0/3147112602 shutdown_connections 2024-04-26T23:59:54.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.348+0000 7fc632800700 1 -- 172.21.15.40:0/3147112602 wait complete. 2024-04-26T23:59:54.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.349+0000 7fc632800700 1 Processor -- start 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.349+0000 7fc632800700 1 -- start start 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.349+0000 7fc632800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc62c105640 0x7fc62c19bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.349+0000 7fc632800700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc62c106000 0x7fc62c19c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc632800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62c10b640 0x7fc62c1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc632800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c115890 con 0x7fc62c10b640 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc632800700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c115710 con 0x7fc62c106000 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc632800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c115a10 con 0x7fc62c105640 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc62be00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62c10b640 0x7fc62c1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc62b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc62c105640 0x7fc62c19bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc62aa00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc62c106000 0x7fc62c19c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc62b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc62c105640 0x7fc62c19bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60736/0 (socket says 172.21.15.40:60736) 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc62b400700 1 -- 172.21.15.40:0/2232920998 learned_addr learned my addr 172.21.15.40:0/2232920998 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc62b400700 1 -- 172.21.15.40:0/2232920998 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc62c106000 msgr2=0x7fc62c19c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:54.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc62b400700 1 --2- 172.21.15.40:0/2232920998 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc62c106000 0x7fc62c19c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc62b400700 1 -- 172.21.15.40:0/2232920998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62c10b640 msgr2=0x7fc62c1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc62b400700 1 --2- 172.21.15.40:0/2232920998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62c10b640 0x7fc62c1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.350+0000 7fc62b400700 1 -- 172.21.15.40:0/2232920998 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62c1a0d70 con 0x7fc62c105640 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.351+0000 7fc62b400700 1 --2- 172.21.15.40:0/2232920998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc62c105640 0x7fc62c19bd60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc61c00e9a0 tx=0x7fc61c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.351+0000 7fc623e00700 1 -- 172.21.15.40:0/2232920998 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c00ca10 con 0x7fc62c105640 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.351+0000 7fc623e00700 1 -- 172.21.15.40:0/2232920998 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc61c00cbb0 con 0x7fc62c105640 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.351+0000 7fc623e00700 1 -- 172.21.15.40:0/2232920998 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c018680 con 0x7fc62c105640 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.351+0000 7fc632800700 1 -- 172.21.15.40:0/2232920998 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc62c1a1060 con 0x7fc62c105640 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.351+0000 7fc632800700 1 -- 172.21.15.40:0/2232920998 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc62c1a88e0 con 0x7fc62c105640 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.352+0000 7fc632800700 1 -- 172.21.15.40:0/2232920998 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc62c063bb0 con 0x7fc62c105640 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.354+0000 7fc623e00700 1 -- 172.21.15.40:0/2232920998 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc61c010430 con 0x7fc62c105640 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.355+0000 7fc623e00700 1 --2- 172.21.15.40:0/2232920998 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6100a3d50 0x7fc6100a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.355+0000 7fc623e00700 1 -- 172.21.15.40:0/2232920998 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc61c0d0990 con 0x7fc62c105640 2024-04-26T23:59:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.356+0000 7fc62aa00700 1 --2- 172.21.15.40:0/2232920998 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6100a3d50 0x7fc6100a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:54.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.356+0000 7fc623e00700 1 --2- 172.21.15.40:0/2232920998 >> [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332] conn(0x7fc6100aea10 0x7fc6100b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:54.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.356+0000 7fc623e00700 1 -- 172.21.15.40:0/2232920998 --> [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc61c01ce70 con 0x7fc6100aea10 2024-04-26T23:59:54.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.356+0000 7fc623e00700 1 -- 172.21.15.40:0/2232920998 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fc61c0d0d40 con 0x7fc62c105640 2024-04-26T23:59:54.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.356+0000 7fc62be00700 1 --2- 172.21.15.40:0/2232920998 >> [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332] conn(0x7fc6100aea10 0x7fc6100b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:54.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.357+0000 7fc62aa00700 1 --2- 172.21.15.40:0/2232920998 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6100a3d50 0x7fc6100a6210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc6180098c0 tx=0x7fc618019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:54.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.357+0000 7fc62be00700 1 --2- 172.21.15.40:0/2232920998 >> [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332] conn(0x7fc6100aea10 0x7fc6100b0e50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:54.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.358+0000 7fc623e00700 1 -- 172.21.15.40:0/2232920998 <== osd.7 v2:172.21.15.80:6816/1231184332 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc61c01ce70 con 0x7fc6100aea10 2024-04-26T23:59:54.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.376+0000 7fc632800700 1 -- 172.21.15.40:0/2232920998 --> [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc62c063fb0 con 0x7fc6100aea10 2024-04-26T23:59:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.377+0000 7fc623e00700 1 -- 172.21.15.40:0/2232920998 <== osd.7 v2:172.21.15.80:6816/1231184332 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc62c063fb0 con 0x7fc6100aea10 2024-04-26T23:59:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.378+0000 7fc632800700 1 -- 172.21.15.40:0/2232920998 >> [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332] conn(0x7fc6100aea10 msgr2=0x7fc6100b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.378+0000 7fc632800700 1 --2- 172.21.15.40:0/2232920998 >> [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332] conn(0x7fc6100aea10 0x7fc6100b0e50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.378+0000 7fc632800700 1 -- 172.21.15.40:0/2232920998 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6100a3d50 msgr2=0x7fc6100a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.378+0000 7fc632800700 1 --2- 172.21.15.40:0/2232920998 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6100a3d50 0x7fc6100a6210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc6180098c0 tx=0x7fc618019040 comp rx=0 tx=0).stop 2024-04-26T23:59:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.378+0000 7fc632800700 1 -- 172.21.15.40:0/2232920998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc62c105640 msgr2=0x7fc62c19bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.378+0000 7fc632800700 1 --2- 172.21.15.40:0/2232920998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc62c105640 0x7fc62c19bd60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc61c00e9a0 tx=0x7fc61c00ecb0 comp rx=0 tx=0).stop 2024-04-26T23:59:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.378+0000 7fc62be00700 1 -- 172.21.15.40:0/2232920998 reap_dead start 2024-04-26T23:59:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.378+0000 7fc632800700 1 -- 172.21.15.40:0/2232920998 shutdown_connections 2024-04-26T23:59:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.378+0000 7fc632800700 1 -- 172.21.15.40:0/2232920998 >> 172.21.15.40:0/2232920998 conn(0x7fc62c101130 msgr2=0x7fc62c101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.378+0000 7fc632800700 1 -- 172.21.15.40:0/2232920998 shutdown_connections 2024-04-26T23:59:54.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:54.378+0000 7fc632800700 1 -- 172.21.15.40:0/2232920998 wait complete. 2024-04-26T23:59:56.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:56 smithi040 ceph-mon[28095]: pgmap v355: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T23:59:56.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:56 smithi080 ceph-mon[32362]: pgmap v355: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T23:59:56.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:56 smithi103 ceph-mon[32952]: pgmap v355: 129 pgs: 129 active+clean; 634 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T23:59:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.110+0000 7ff622400700 1 -- 172.21.15.40:0/3562915862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff61c10a320 msgr2=0x7ff61c1167c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.110+0000 7ff622400700 1 --2- 172.21.15.40:0/3562915862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff61c10a320 0x7ff61c1167c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff614009f90 tx=0x7ff61400b330 comp rx=0 tx=0).stop 2024-04-26T23:59:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.110+0000 7ff622400700 1 -- 172.21.15.40:0/3562915862 shutdown_connections 2024-04-26T23:59:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.110+0000 7ff622400700 1 --2- 172.21.15.40:0/3562915862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff61c10a320 0x7ff61c1167c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.110+0000 7ff622400700 1 --2- 172.21.15.40:0/3562915862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff61c10e830 0x7ff61c10ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.110+0000 7ff622400700 1 --2- 172.21.15.40:0/3562915862 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff61c109960 0x7ff61c109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.110+0000 7ff622400700 1 -- 172.21.15.40:0/3562915862 >> 172.21.15.40:0/3562915862 conn(0x7ff61c0791b0 msgr2=0x7ff61c108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.110+0000 7ff622400700 1 -- 172.21.15.40:0/3562915862 shutdown_connections 2024-04-26T23:59:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.110+0000 7ff622400700 1 -- 172.21.15.40:0/3562915862 wait complete. 2024-04-26T23:59:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.112+0000 7ff622400700 1 Processor -- start 2024-04-26T23:59:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.112+0000 7ff622400700 1 -- start start 2024-04-26T23:59:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.112+0000 7ff622400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff61c109960 0x7ff61c19ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.112+0000 7ff622400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff61c10a320 0x7ff61c1a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.112+0000 7ff622400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff61c10e830 0x7ff61c1a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.112+0000 7ff622400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff61c119b50 con 0x7ff61c109960 2024-04-26T23:59:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.112+0000 7ff622400700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7ff61c1199d0 con 0x7ff61c10e830 2024-04-26T23:59:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.112+0000 7ff622400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff61c119cd0 con 0x7ff61c10a320 2024-04-26T23:59:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.113+0000 7ff621000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff61c109960 0x7ff61c19ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.113+0000 7ff621000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff61c109960 0x7ff61c19ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41726/0 (socket says 172.21.15.40:41726) 2024-04-26T23:59:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.113+0000 7ff621000700 1 -- 172.21.15.40:0/701524088 learned_addr learned my addr 172.21.15.40:0/701524088 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.113+0000 7ff621a00700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff61c10e830 0x7ff61c1a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.113+0000 7ff61be00700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff61c10a320 0x7ff61c1a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.113+0000 7ff621000700 1 -- 172.21.15.40:0/701524088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff61c10a320 msgr2=0x7ff61c1a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.113+0000 7ff621000700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff61c10a320 0x7ff61c1a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.113+0000 7ff621000700 1 -- 172.21.15.40:0/701524088 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff61c10e830 msgr2=0x7ff61c1a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.113+0000 7ff621000700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff61c10e830 0x7ff61c1a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.113+0000 7ff621000700 1 -- 172.21.15.40:0/701524088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61c1a4fa0 con 0x7ff61c109960 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.114+0000 7ff621000700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff61c109960 0x7ff61c19ffe0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff61000e9a0 tx=0x7ff61000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.114+0000 7ff619600700 1 -- 172.21.15.40:0/701524088 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61000ca10 con 0x7ff61c109960 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.114+0000 7ff622400700 1 -- 172.21.15.40:0/701524088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff61c1a5290 con 0x7ff61c109960 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.114+0000 7ff619600700 1 -- 172.21.15.40:0/701524088 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff61000cbb0 con 0x7ff61c109960 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.114+0000 7ff619600700 1 -- 172.21.15.40:0/701524088 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610018680 con 0x7ff61c109960 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.114+0000 7ff622400700 1 -- 172.21.15.40:0/701524088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff61c1acb10 con 0x7ff61c109960 2024-04-26T23:59:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.115+0000 7ff622400700 1 -- 172.21.15.40:0/701524088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff61c063b70 con 0x7ff61c109960 2024-04-26T23:59:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.117+0000 7ff619600700 1 -- 172.21.15.40:0/701524088 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff610010430 con 0x7ff61c109960 2024-04-26T23:59:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.119+0000 7ff619600700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff6040a3cd0 0x7ff6040a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.119+0000 7ff619600700 1 -- 172.21.15.40:0/701524088 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff6100d0e90 con 0x7ff61c109960 2024-04-26T23:59:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.119+0000 7ff61be00700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff6040a3cd0 0x7ff6040a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.119+0000 7ff619600700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397] conn(0x7ff6040ae990 0x7ff6040b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:59:58.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.119+0000 7ff619600700 1 -- 172.21.15.40:0/701524088 --> [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff610019ec0 con 0x7ff6040ae990 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.119+0000 7ff619600700 1 -- 172.21.15.40:0/701524088 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7ff6100d1240 con 0x7ff61c109960 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.119+0000 7ff621a00700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397] conn(0x7ff6040ae990 0x7ff6040b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.120+0000 7ff61be00700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff6040a3cd0 0x7ff6040a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff608009750 tx=0x7ff608009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.120+0000 7ff621a00700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397] conn(0x7ff6040ae990 0x7ff6040b0dd0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.121+0000 7ff619600700 1 -- 172.21.15.40:0/701524088 <== osd.11 v2:172.21.15.103:6824/2714925397 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff610019ec0 con 0x7ff6040ae990 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.139+0000 7ff622400700 1 -- 172.21.15.40:0/701524088 --> [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff61c063ff0 con 0x7ff6040ae990 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.140+0000 7ff619600700 1 -- 172.21.15.40:0/701524088 <== osd.11 v2:172.21.15.103:6824/2714925397 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff61c063ff0 con 0x7ff6040ae990 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.140+0000 7ff622400700 1 -- 172.21.15.40:0/701524088 >> [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397] conn(0x7ff6040ae990 msgr2=0x7ff6040b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.140+0000 7ff622400700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397] conn(0x7ff6040ae990 0x7ff6040b0dd0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.141+0000 7ff622400700 1 -- 172.21.15.40:0/701524088 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff6040a3cd0 msgr2=0x7ff6040a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.141+0000 7ff622400700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff6040a3cd0 0x7ff6040a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff608009750 tx=0x7ff608009310 comp rx=0 tx=0).stop 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.141+0000 7ff622400700 1 -- 172.21.15.40:0/701524088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff61c109960 msgr2=0x7ff61c19ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:59:58.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.141+0000 7ff622400700 1 --2- 172.21.15.40:0/701524088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff61c109960 0x7ff61c19ffe0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff61000e9a0 tx=0x7ff61000ecb0 comp rx=0 tx=0).stop 2024-04-26T23:59:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.141+0000 7ff621a00700 1 -- 172.21.15.40:0/701524088 reap_dead start 2024-04-26T23:59:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.141+0000 7ff622400700 1 -- 172.21.15.40:0/701524088 shutdown_connections 2024-04-26T23:59:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.141+0000 7ff622400700 1 -- 172.21.15.40:0/701524088 >> 172.21.15.40:0/701524088 conn(0x7ff61c0791b0 msgr2=0x7ff61c079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:59:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.141+0000 7ff622400700 1 -- 172.21.15.40:0/701524088 shutdown_connections 2024-04-26T23:59:58.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-26T23:59:57.141+0000 7ff622400700 1 -- 172.21.15.40:0/701524088 wait complete. 2024-04-26T23:59:58.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 26 23:59:58 smithi040 ceph-mon[28095]: pgmap v356: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-26T23:59:58.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 26 23:59:58 smithi080 ceph-mon[32362]: pgmap v356: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-26T23:59:58.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 26 23:59:58 smithi103 ceph-mon[32952]: pgmap v356: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-26T23:59:59.317 INFO:teuthology.orchestra.run.smithi040.stdout:347892351003 2024-04-26T23:59:59.318 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-27T00:00:00.299 INFO:teuthology.orchestra.run.smithi040.stdout:223338299441 2024-04-27T00:00:00.299 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-27T00:00:00.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:00 smithi040 ceph-mon[28095]: pgmap v357: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-27T00:00:00.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:00 smithi040 ceph-mon[28095]: overall HEALTH_OK 2024-04-27T00:00:00.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:00 smithi080 ceph-mon[32362]: pgmap v357: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-27T00:00:00.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:00 smithi080 ceph-mon[32362]: overall HEALTH_OK 2024-04-27T00:00:00.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:00 smithi103 ceph-mon[32952]: pgmap v357: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-27T00:00:00.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:00 smithi103 ceph-mon[32952]: overall HEALTH_OK 2024-04-27T00:00:02.217 INFO:teuthology.orchestra.run.smithi040.stdout:292057776165 2024-04-27T00:00:02.217 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-27T00:00:02.455 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:02 smithi040 ceph-mon[28095]: pgmap v358: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-27T00:00:02.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:02 smithi080 ceph-mon[32362]: pgmap v358: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-27T00:00:02.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:02 smithi103 ceph-mon[32952]: pgmap v358: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-27T00:00:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.815+0000 7fe4cb200700 1 -- 172.21.15.40:0/2025394071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4c41094a0 msgr2=0x7fe4c4109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.815+0000 7fe4cb200700 1 --2- 172.21.15.40:0/2025394071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4c41094a0 0x7fe4c4109880 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4009990 tx=0x7fe4b4009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.815+0000 7fe4cb200700 1 -- 172.21.15.40:0/2025394071 shutdown_connections 2024-04-27T00:00:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.815+0000 7fe4cb200700 1 --2- 172.21.15.40:0/2025394071 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4c406bad0 0x7fe4c41102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.815+0000 7fe4cb200700 1 --2- 172.21.15.40:0/2025394071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4c41094a0 0x7fe4c4109880 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.815+0000 7fe4cb200700 1 --2- 172.21.15.40:0/2025394071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4c406b130 0x7fe4c406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:02.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.815+0000 7fe4cb200700 1 -- 172.21.15.40:0/2025394071 >> 172.21.15.40:0/2025394071 conn(0x7fe4c40792d0 msgr2=0x7fe4c40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:02.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.816+0000 7fe4cb200700 1 -- 172.21.15.40:0/2025394071 shutdown_connections 2024-04-27T00:00:02.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.816+0000 7fe4cb200700 1 -- 172.21.15.40:0/2025394071 wait complete. 2024-04-27T00:00:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.817+0000 7fe4cb200700 1 Processor -- start 2024-04-27T00:00:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.817+0000 7fe4cb200700 1 -- start start 2024-04-27T00:00:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.818+0000 7fe4cb200700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4c406b130 0x7fe4c419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.818+0000 7fe4cb200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4c406bad0 0x7fe4c419fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.818+0000 7fe4cb200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4c41094a0 0x7fe4c41999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.818+0000 7fe4cb200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c41135e0 con 0x7fe4c41094a0 2024-04-27T00:00:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.818+0000 7fe4cb200700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c4113460 con 0x7fe4c406b130 2024-04-27T00:00:02.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.818+0000 7fe4cb200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c4113760 con 0x7fe4c406bad0 2024-04-27T00:00:02.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.818+0000 7fe4c3e00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4c406b130 0x7fe4c419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:02.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.818+0000 7fe4c3e00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4c406b130 0x7fe4c419f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:57200/0 (socket says 172.21.15.40:57200) 2024-04-27T00:00:02.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.818+0000 7fe4c3e00700 1 -- 172.21.15.40:0/3891503330 learned_addr learned my addr 172.21.15.40:0/3891503330 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:02.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.818+0000 7fe4c3400700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4c406bad0 0x7fe4c419fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:02.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.819+0000 7fe4c9200700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4c41094a0 0x7fe4c41999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.819+0000 7fe4c3e00700 1 -- 172.21.15.40:0/3891503330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4c406bad0 msgr2=0x7fe4c419fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.819+0000 7fe4c3e00700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4c406bad0 0x7fe4c419fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.819+0000 7fe4c3e00700 1 -- 172.21.15.40:0/3891503330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4c41094a0 msgr2=0x7fe4c41999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.819+0000 7fe4c3e00700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4c41094a0 0x7fe4c41999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.819+0000 7fe4c3e00700 1 -- 172.21.15.40:0/3891503330 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c419a100 con 0x7fe4c406b130 2024-04-27T00:00:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.819+0000 7fe4c3400700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4c406bad0 0x7fe4c419fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.819+0000 7fe4c3e00700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4c406b130 0x7fe4c419f7e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4000c00 tx=0x7fe4b40057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:02.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.820+0000 7fe4c0c00700 1 -- 172.21.15.40:0/3891503330 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4027070 con 0x7fe4c406b130 2024-04-27T00:00:02.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.820+0000 7fe4cb200700 1 -- 172.21.15.40:0/3891503330 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c419a390 con 0x7fe4c406b130 2024-04-27T00:00:02.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.820+0000 7fe4c0c00700 1 -- 172.21.15.40:0/3891503330 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4b4022470 con 0x7fe4c406b130 2024-04-27T00:00:02.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.820+0000 7fe4c0c00700 1 -- 172.21.15.40:0/3891503330 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4017650 con 0x7fe4c406b130 2024-04-27T00:00:02.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.820+0000 7fe4cb200700 1 -- 172.21.15.40:0/3891503330 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c41a66b0 con 0x7fe4c406b130 2024-04-27T00:00:02.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.821+0000 7fe4cb200700 1 -- 172.21.15.40:0/3891503330 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe4c4063bb0 con 0x7fe4c406b130 2024-04-27T00:00:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.823+0000 7fe4c0c00700 1 -- 172.21.15.40:0/3891503330 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe4b402a070 con 0x7fe4c406b130 2024-04-27T00:00:02.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.825+0000 7fe4c0c00700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe4a80a3cd0 0x7fe4a80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:02.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.825+0000 7fe4c0c00700 1 -- 172.21.15.40:0/3891503330 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe4b40d1690 con 0x7fe4c406b130 2024-04-27T00:00:02.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.825+0000 7fe4c3400700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe4a80a3cd0 0x7fe4a80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:02.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.825+0000 7fe4c0c00700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790] conn(0x7fe4a80ae990 0x7fe4a80b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:02.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.825+0000 7fe4c0c00700 1 -- 172.21.15.40:0/3891503330 --> [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe4a80b14a0 con 0x7fe4a80ae990 2024-04-27T00:00:02.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.825+0000 7fe4c0c00700 1 -- 172.21.15.40:0/3891503330 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fe4b40d1a40 con 0x7fe4c406b130 2024-04-27T00:00:02.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.826+0000 7fe4c9200700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790] conn(0x7fe4a80ae990 0x7fe4a80b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:02.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.826+0000 7fe4c3400700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe4a80a3cd0 0x7fe4a80a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009680 tx=0x7fe4b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:02.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.826+0000 7fe4c9200700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790] conn(0x7fe4a80ae990 0x7fe4a80b0dd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:02.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.827+0000 7fe4c0c00700 1 -- 172.21.15.40:0/3891503330 <== osd.4 v2:172.21.15.80:6808/3602586790 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe4a80b14a0 con 0x7fe4a80ae990 2024-04-27T00:00:02.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.845+0000 7fe4cb200700 1 -- 172.21.15.40:0/3891503330 --> [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe4c4063fb0 con 0x7fe4a80ae990 2024-04-27T00:00:02.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.846+0000 7fe4c0c00700 1 -- 172.21.15.40:0/3891503330 <== osd.4 v2:172.21.15.80:6808/3602586790 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe4c4063fb0 con 0x7fe4a80ae990 2024-04-27T00:00:02.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.846+0000 7fe4cb200700 1 -- 172.21.15.40:0/3891503330 >> [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790] conn(0x7fe4a80ae990 msgr2=0x7fe4a80b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:02.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.846+0000 7fe4cb200700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790] conn(0x7fe4a80ae990 0x7fe4a80b0dd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:02.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.847+0000 7fe4cb200700 1 -- 172.21.15.40:0/3891503330 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe4a80a3cd0 msgr2=0x7fe4a80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:02.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.847+0000 7fe4cb200700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe4a80a3cd0 0x7fe4a80a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009680 tx=0x7fe4b0009260 comp rx=0 tx=0).stop 2024-04-27T00:00:02.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.847+0000 7fe4cb200700 1 -- 172.21.15.40:0/3891503330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4c406b130 msgr2=0x7fe4c419f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:02.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.847+0000 7fe4cb200700 1 --2- 172.21.15.40:0/3891503330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4c406b130 0x7fe4c419f7e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4000c00 tx=0x7fe4b40057c0 comp rx=0 tx=0).stop 2024-04-27T00:00:02.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.847+0000 7fe4c9200700 1 -- 172.21.15.40:0/3891503330 reap_dead start 2024-04-27T00:00:02.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.847+0000 7fe4cb200700 1 -- 172.21.15.40:0/3891503330 shutdown_connections 2024-04-27T00:00:02.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.847+0000 7fe4cb200700 1 -- 172.21.15.40:0/3891503330 >> 172.21.15.40:0/3891503330 conn(0x7fe4c40792d0 msgr2=0x7fe4c41004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:02.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.847+0000 7fe4cb200700 1 -- 172.21.15.40:0/3891503330 shutdown_connections 2024-04-27T00:00:02.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:02.847+0000 7fe4cb200700 1 -- 172.21.15.40:0/3891503330 wait complete. 2024-04-27T00:00:03.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.090+0000 7f89d0400700 1 -- 172.21.15.40:0/3691306038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89c8109490 msgr2=0x7f89c8109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:03.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.090+0000 7f89d0400700 1 --2- 172.21.15.40:0/3691306038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89c8109490 0x7f89c8109870 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f89bc009990 tx=0x7f89bc009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:03.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.090+0000 7f89d0400700 1 -- 172.21.15.40:0/3691306038 shutdown_connections 2024-04-27T00:00:03.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.091+0000 7f89d0400700 1 --2- 172.21.15.40:0/3691306038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89c806c360 0x7f89c81102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:03.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.091+0000 7f89d0400700 1 --2- 172.21.15.40:0/3691306038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89c8109490 0x7f89c8109870 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:03.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.091+0000 7f89d0400700 1 --2- 172.21.15.40:0/3691306038 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89c806b9c0 0x7f89c806be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:03.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.091+0000 7f89d0400700 1 -- 172.21.15.40:0/3691306038 >> 172.21.15.40:0/3691306038 conn(0x7f89c8079290 msgr2=0x7f89c80796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:03.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.091+0000 7f89d0400700 1 -- 172.21.15.40:0/3691306038 shutdown_connections 2024-04-27T00:00:03.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.091+0000 7f89d0400700 1 -- 172.21.15.40:0/3691306038 wait complete. 2024-04-27T00:00:03.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.092+0000 7f89d0400700 1 Processor -- start 2024-04-27T00:00:03.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.092+0000 7f89d0400700 1 -- start start 2024-04-27T00:00:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.093+0000 7f89d0400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89c806b9c0 0x7f89c819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.093+0000 7f89d0400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89c806c360 0x7f89c819c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.093+0000 7f89d0400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89c8109490 0x7f89c81a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.093+0000 7f89d0400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f89c8113580 con 0x7f89c806b9c0 2024-04-27T00:00:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.093+0000 7f89d0400700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f89c8113400 con 0x7f89c8109490 2024-04-27T00:00:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.093+0000 7f89d0400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89c8113700 con 0x7f89c806c360 2024-04-27T00:00:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.093+0000 7f89cda00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89c806b9c0 0x7f89c819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.093+0000 7f89cda00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89c806b9c0 0x7f89c819bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37854/0 (socket says 172.21.15.40:37854) 2024-04-27T00:00:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.093+0000 7f89cda00700 1 -- 172.21.15.40:0/3403733782 learned_addr learned my addr 172.21.15.40:0/3403733782 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.093+0000 7f89cd000700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89c806c360 0x7f89c819c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:03.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.093+0000 7f89ce400700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89c8109490 0x7f89c81a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:03.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.094+0000 7f89cda00700 1 -- 172.21.15.40:0/3403733782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89c806c360 msgr2=0x7f89c819c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:03.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.094+0000 7f89cda00700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89c806c360 0x7f89c819c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:03.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.094+0000 7f89cda00700 1 -- 172.21.15.40:0/3403733782 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89c8109490 msgr2=0x7f89c81a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:03.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.094+0000 7f89cda00700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89c8109490 0x7f89c81a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:03.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.094+0000 7f89cda00700 1 -- 172.21.15.40:0/3403733782 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c81a0d20 con 0x7f89c806b9c0 2024-04-27T00:00:03.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.094+0000 7f89cda00700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89c806b9c0 0x7f89c819bd10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0057c0 tx=0x7f89bc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:03.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.094+0000 7f89c6000700 1 -- 172.21.15.40:0/3403733782 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc026070 con 0x7f89c806b9c0 2024-04-27T00:00:03.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.094+0000 7f89c6000700 1 -- 172.21.15.40:0/3403733782 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89bc021410 con 0x7f89c806b9c0 2024-04-27T00:00:03.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.094+0000 7f89d0400700 1 -- 172.21.15.40:0/3403733782 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c81a0fb0 con 0x7f89c806b9c0 2024-04-27T00:00:03.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.094+0000 7f89c6000700 1 -- 172.21.15.40:0/3403733782 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc017650 con 0x7f89c806b9c0 2024-04-27T00:00:03.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.094+0000 7f89d0400700 1 -- 172.21.15.40:0/3403733782 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c81a8890 con 0x7f89c806b9c0 2024-04-27T00:00:03.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.095+0000 7f89d0400700 1 -- 172.21.15.40:0/3403733782 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f89c8063bb0 con 0x7f89c806b9c0 2024-04-27T00:00:03.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.098+0000 7f89c6000700 1 -- 172.21.15.40:0/3403733782 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f89bc029070 con 0x7f89c806b9c0 2024-04-27T00:00:03.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.099+0000 7f89c6000700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f89b00a3cd0 0x7f89b00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:03.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.099+0000 7f89c6000700 1 -- 172.21.15.40:0/3403733782 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f89bc0d18c0 con 0x7f89c806b9c0 2024-04-27T00:00:03.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.099+0000 7f89cd000700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f89b00a3cd0 0x7f89b00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:03.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.099+0000 7f89c6000700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511] conn(0x7f89b00ae860 0x7f89b00b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:03.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.099+0000 7f89c6000700 1 -- 172.21.15.40:0/3403733782 --> [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89b00b1370 con 0x7f89b00ae860 2024-04-27T00:00:03.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.099+0000 7f89c6000700 1 -- 172.21.15.40:0/3403733782 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f89bc0d1c70 con 0x7f89c806b9c0 2024-04-27T00:00:03.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.100+0000 7f89ce400700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511] conn(0x7f89b00ae860 0x7f89b00b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:03.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.100+0000 7f89cd000700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f89b00a3cd0 0x7f89b00a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f89c8079940 tx=0x7f89b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:03.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.101+0000 7f89ce400700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511] conn(0x7f89b00ae860 0x7f89b00b0ca0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:03.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.102+0000 7f89c6000700 1 -- 172.21.15.40:0/3403733782 <== osd.8 v2:172.21.15.103:6816/4109830511 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b00b1370 con 0x7f89b00ae860 2024-04-27T00:00:03.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.120+0000 7f89d0400700 1 -- 172.21.15.40:0/3403733782 --> [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f89c8064030 con 0x7f89b00ae860 2024-04-27T00:00:03.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.121+0000 7f89c6000700 1 -- 172.21.15.40:0/3403733782 <== osd.8 v2:172.21.15.103:6816/4109830511 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f89c8064030 con 0x7f89b00ae860 2024-04-27T00:00:03.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.121+0000 7f89d0400700 1 -- 172.21.15.40:0/3403733782 >> [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511] conn(0x7f89b00ae860 msgr2=0x7f89b00b0ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:03.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.121+0000 7f89d0400700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511] conn(0x7f89b00ae860 0x7f89b00b0ca0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:03.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.122+0000 7f89d0400700 1 -- 172.21.15.40:0/3403733782 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f89b00a3cd0 msgr2=0x7f89b00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:03.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.122+0000 7f89d0400700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f89b00a3cd0 0x7f89b00a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f89c8079940 tx=0x7f89b4009260 comp rx=0 tx=0).stop 2024-04-27T00:00:03.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.122+0000 7f89d0400700 1 -- 172.21.15.40:0/3403733782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89c806b9c0 msgr2=0x7f89c819bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:03.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.122+0000 7f89d0400700 1 --2- 172.21.15.40:0/3403733782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89c806b9c0 0x7f89c819bd10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0057c0 tx=0x7f89bc009580 comp rx=0 tx=0).stop 2024-04-27T00:00:03.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.122+0000 7f89ce400700 1 -- 172.21.15.40:0/3403733782 reap_dead start 2024-04-27T00:00:03.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.122+0000 7f89d0400700 1 -- 172.21.15.40:0/3403733782 shutdown_connections 2024-04-27T00:00:03.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.122+0000 7f89d0400700 1 -- 172.21.15.40:0/3403733782 >> 172.21.15.40:0/3403733782 conn(0x7f89c8079290 msgr2=0x7f89c810e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:03.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.122+0000 7f89d0400700 1 -- 172.21.15.40:0/3403733782 shutdown_connections 2024-04-27T00:00:03.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:03.122+0000 7f89d0400700 1 -- 172.21.15.40:0/3403733782 wait complete. 2024-04-27T00:00:04.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:04 smithi040 ceph-mon[28095]: pgmap v359: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-27T00:00:04.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:04 smithi080 ceph-mon[32362]: pgmap v359: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-27T00:00:04.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:04 smithi103 ceph-mon[32952]: pgmap v359: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-27T00:00:05.279 INFO:teuthology.orchestra.run.smithi040.stdout:257698037807 2024-04-27T00:00:05.279 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-27T00:00:05.482 INFO:teuthology.orchestra.run.smithi040.stdout:146028888134 2024-04-27T00:00:05.483 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-27T00:00:06.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:06 smithi040 ceph-mon[28095]: pgmap v360: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-27T00:00:06.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:06 smithi080 ceph-mon[32362]: pgmap v360: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-27T00:00:06.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:06 smithi103 ceph-mon[32952]: pgmap v360: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-27T00:00:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.146+0000 7f6596600700 1 -- 172.21.15.40:0/2096371764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6590109480 msgr2=0x7f6590109860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.146+0000 7f6596600700 1 --2- 172.21.15.40:0/2096371764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6590109480 0x7f6590109860 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6580009990 tx=0x7f6580009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.146+0000 7f6596600700 1 -- 172.21.15.40:0/2096371764 shutdown_connections 2024-04-27T00:00:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.146+0000 7f6596600700 1 --2- 172.21.15.40:0/2096371764 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6590103ea0 0x7f65901102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.146+0000 7f6596600700 1 --2- 172.21.15.40:0/2096371764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6590109480 0x7f6590109860 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.146+0000 7f6596600700 1 --2- 172.21.15.40:0/2096371764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65901034e0 0x7f6590103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.146+0000 7f6596600700 1 -- 172.21.15.40:0/2096371764 >> 172.21.15.40:0/2096371764 conn(0x7f659007a5e0 msgr2=0x7f659007a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.147+0000 7f6596600700 1 -- 172.21.15.40:0/2096371764 shutdown_connections 2024-04-27T00:00:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.147+0000 7f6596600700 1 -- 172.21.15.40:0/2096371764 wait complete. 2024-04-27T00:00:06.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.148+0000 7f6596600700 1 Processor -- start 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.148+0000 7f6596600700 1 -- start start 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.148+0000 7f6596600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65901034e0 0x7f659019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f6596600700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6590103ea0 0x7f659019fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f6596600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6590109480 0x7f6590199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f6596600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6590113600 con 0x7f6590109480 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f6596600700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f6590113480 con 0x7f6590103ea0 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f6596600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6590113780 con 0x7f65901034e0 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f658fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6590109480 0x7f6590199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f658f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65901034e0 0x7f659019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f658ea00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6590103ea0 0x7f659019fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f658f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65901034e0 0x7f659019f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37714/0 (socket says 172.21.15.40:37714) 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f658fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6590109480 0x7f6590199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37872/0 (socket says 172.21.15.40:37872) 2024-04-27T00:00:06.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f658f400700 1 -- 172.21.15.40:0/4007487129 learned_addr learned my addr 172.21.15.40:0/4007487129 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f658f400700 1 -- 172.21.15.40:0/4007487129 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6590103ea0 msgr2=0x7f659019fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f658f400700 1 --2- 172.21.15.40:0/4007487129 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6590103ea0 0x7f659019fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f658f400700 1 -- 172.21.15.40:0/4007487129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6590109480 msgr2=0x7f6590199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f658f400700 1 --2- 172.21.15.40:0/4007487129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6590109480 0x7f6590199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.149+0000 7f658f400700 1 -- 172.21.15.40:0/4007487129 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659019a0c0 con 0x7f65901034e0 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.150+0000 7f658f400700 1 --2- 172.21.15.40:0/4007487129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65901034e0 0x7f659019f7a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6580000c00 tx=0x7f65800057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.150+0000 7f6587e00700 1 -- 172.21.15.40:0/4007487129 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580027070 con 0x7f65901034e0 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.150+0000 7f6587e00700 1 -- 172.21.15.40:0/4007487129 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6580022470 con 0x7f65901034e0 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.150+0000 7f6587e00700 1 -- 172.21.15.40:0/4007487129 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580017650 con 0x7f65901034e0 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.150+0000 7f6596600700 1 -- 172.21.15.40:0/4007487129 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f659019a350 con 0x7f65901034e0 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.150+0000 7f6596600700 1 -- 172.21.15.40:0/4007487129 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6590075650 con 0x7f65901034e0 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.151+0000 7f6596600700 1 -- 172.21.15.40:0/4007487129 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6590063bb0 con 0x7f65901034e0 2024-04-27T00:00:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.153+0000 7f6587e00700 1 -- 172.21.15.40:0/4007487129 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f658002a070 con 0x7f65901034e0 2024-04-27T00:00:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.155+0000 7f6587e00700 1 --2- 172.21.15.40:0/4007487129 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f65740a3d50 0x7f65740a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.155+0000 7f6587e00700 1 -- 172.21.15.40:0/4007487129 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f65800d1600 con 0x7f65901034e0 2024-04-27T00:00:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.155+0000 7f658ea00700 1 --2- 172.21.15.40:0/4007487129 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f65740a3d50 0x7f65740a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.156+0000 7f6587e00700 1 --2- 172.21.15.40:0/4007487129 >> [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951] conn(0x7f65740ae8e0 0x7f65740b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.156+0000 7f6587e00700 1 -- 172.21.15.40:0/4007487129 --> [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6580001d30 con 0x7f65740ae8e0 2024-04-27T00:00:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.156+0000 7f6587e00700 1 -- 172.21.15.40:0/4007487129 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f65800d19b0 con 0x7f65901034e0 2024-04-27T00:00:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.156+0000 7f658fe00700 1 --2- 172.21.15.40:0/4007487129 >> [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951] conn(0x7f65740ae8e0 0x7f65740b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.156+0000 7f658ea00700 1 --2- 172.21.15.40:0/4007487129 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f65740a3d50 0x7f65740a6210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f659007acc0 tx=0x7f657c012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.157+0000 7f658fe00700 1 --2- 172.21.15.40:0/4007487129 >> [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951] conn(0x7f65740ae8e0 0x7f65740b0d20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:06.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.158+0000 7f6587e00700 1 -- 172.21.15.40:0/4007487129 <== osd.10 v2:172.21.15.80:6824/1789667951 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6580001d30 con 0x7f65740ae8e0 2024-04-27T00:00:06.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.176+0000 7f6596600700 1 -- 172.21.15.40:0/4007487129 --> [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6590064080 con 0x7f65740ae8e0 2024-04-27T00:00:06.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.177+0000 7f6587e00700 1 -- 172.21.15.40:0/4007487129 <== osd.10 v2:172.21.15.80:6824/1789667951 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6590064080 con 0x7f65740ae8e0 2024-04-27T00:00:06.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.177+0000 7f6596600700 1 -- 172.21.15.40:0/4007487129 >> [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951] conn(0x7f65740ae8e0 msgr2=0x7f65740b0d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:06.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.177+0000 7f6596600700 1 --2- 172.21.15.40:0/4007487129 >> [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951] conn(0x7f65740ae8e0 0x7f65740b0d20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:06.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.178+0000 7f6596600700 1 -- 172.21.15.40:0/4007487129 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f65740a3d50 msgr2=0x7f65740a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:06.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.178+0000 7f6596600700 1 --2- 172.21.15.40:0/4007487129 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f65740a3d50 0x7f65740a6210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f659007acc0 tx=0x7f657c012040 comp rx=0 tx=0).stop 2024-04-27T00:00:06.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.178+0000 7f6596600700 1 -- 172.21.15.40:0/4007487129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65901034e0 msgr2=0x7f659019f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:06.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.178+0000 7f6596600700 1 --2- 172.21.15.40:0/4007487129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65901034e0 0x7f659019f7a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6580000c00 tx=0x7f65800057c0 comp rx=0 tx=0).stop 2024-04-27T00:00:06.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.178+0000 7f658fe00700 1 -- 172.21.15.40:0/4007487129 reap_dead start 2024-04-27T00:00:06.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.178+0000 7f6596600700 1 -- 172.21.15.40:0/4007487129 shutdown_connections 2024-04-27T00:00:06.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.178+0000 7f6596600700 1 -- 172.21.15.40:0/4007487129 >> 172.21.15.40:0/4007487129 conn(0x7f659007a5e0 msgr2=0x7f6590101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:06.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.178+0000 7f6596600700 1 -- 172.21.15.40:0/4007487129 shutdown_connections 2024-04-27T00:00:06.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:06.178+0000 7f6596600700 1 -- 172.21.15.40:0/4007487129 wait complete. 2024-04-27T00:00:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.634+0000 7f2a07800700 1 -- 172.21.15.40:0/2687546945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a0006c490 msgr2=0x7f2a00110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.634+0000 7f2a07800700 1 --2- 172.21.15.40:0/2687546945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a0006c490 0x7f2a00110300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f29f800b0a0 tx=0x7f29f800b3b0 comp rx=0 tx=0).stop 2024-04-27T00:00:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.634+0000 7f2a07800700 1 -- 172.21.15.40:0/2687546945 shutdown_connections 2024-04-27T00:00:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.634+0000 7f2a07800700 1 --2- 172.21.15.40:0/2687546945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a0006c490 0x7f2a00110300 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.634+0000 7f2a07800700 1 --2- 172.21.15.40:0/2687546945 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a001094c0 0x7f2a001098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.634+0000 7f2a07800700 1 --2- 172.21.15.40:0/2687546945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a0006baf0 0x7f2a0006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.634+0000 7f2a07800700 1 -- 172.21.15.40:0/2687546945 >> 172.21.15.40:0/2687546945 conn(0x7f2a000792d0 msgr2=0x7f2a000796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.634+0000 7f2a07800700 1 -- 172.21.15.40:0/2687546945 shutdown_connections 2024-04-27T00:00:07.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.634+0000 7f2a07800700 1 -- 172.21.15.40:0/2687546945 wait complete. 2024-04-27T00:00:07.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.635+0000 7f2a07800700 1 Processor -- start 2024-04-27T00:00:07.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.635+0000 7f2a07800700 1 -- start start 2024-04-27T00:00:07.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.636+0000 7f2a07800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a0006baf0 0x7f2a0019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:07.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.636+0000 7f2a07800700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0006c490 0x7f2a0019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:07.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.636+0000 7f2a07800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a001094c0 0x7f2a00199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:07.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.636+0000 7f2a07800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001135e0 con 0x7f2a001094c0 2024-04-27T00:00:07.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.636+0000 7f2a07800700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f2a00113460 con 0x7f2a0006c490 2024-04-27T00:00:07.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.636+0000 7f2a07800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2a00113760 con 0x7f2a0006baf0 2024-04-27T00:00:07.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.636+0000 7f2a04e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a0006baf0 0x7f2a0019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:07.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.636+0000 7f2a05800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a001094c0 0x7f2a00199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:07.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.636+0000 7f2a04e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a0006baf0 0x7f2a0019f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37730/0 (socket says 172.21.15.40:37730) 2024-04-27T00:00:07.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.636+0000 7f2a04e00700 1 -- 172.21.15.40:0/3822073371 learned_addr learned my addr 172.21.15.40:0/3822073371 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:07.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.636+0000 7f29ffe00700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0006c490 0x7f2a0019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:07.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.637+0000 7f2a04e00700 1 -- 172.21.15.40:0/3822073371 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0006c490 msgr2=0x7f2a0019fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:07.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.637+0000 7f2a04e00700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0006c490 0x7f2a0019fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:07.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.637+0000 7f2a04e00700 1 -- 172.21.15.40:0/3822073371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a001094c0 msgr2=0x7f2a00199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:07.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.637+0000 7f2a04e00700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a001094c0 0x7f2a00199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:07.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.637+0000 7f2a04e00700 1 -- 172.21.15.40:0/3822073371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0019a060 con 0x7f2a0006baf0 2024-04-27T00:00:07.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.637+0000 7f29ffe00700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2a0006c490 0x7f2a0019fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:07.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.637+0000 7f2a05800700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a001094c0 0x7f2a00199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:07.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.637+0000 7f2a04e00700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a0006baf0 0x7f2a0019f740 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f29f400e9a0 tx=0x7f29f400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:07.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.637+0000 7f29fd600700 1 -- 172.21.15.40:0/3822073371 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f400ca10 con 0x7f2a0006baf0 2024-04-27T00:00:07.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.637+0000 7f29fd600700 1 -- 172.21.15.40:0/3822073371 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29f400cbb0 con 0x7f2a0006baf0 2024-04-27T00:00:07.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.637+0000 7f2a07800700 1 -- 172.21.15.40:0/3822073371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a0019a350 con 0x7f2a0006baf0 2024-04-27T00:00:07.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.638+0000 7f29fd600700 1 -- 172.21.15.40:0/3822073371 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f4018680 con 0x7f2a0006baf0 2024-04-27T00:00:07.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.638+0000 7f2a07800700 1 -- 172.21.15.40:0/3822073371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a001a6610 con 0x7f2a0006baf0 2024-04-27T00:00:07.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.639+0000 7f2a07800700 1 -- 172.21.15.40:0/3822073371 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2a00063b70 con 0x7f2a0006baf0 2024-04-27T00:00:07.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.641+0000 7f29fd600700 1 -- 172.21.15.40:0/3822073371 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f29f4010430 con 0x7f2a0006baf0 2024-04-27T00:00:07.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.642+0000 7f29fd600700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29e80a3cd0 0x7f29e80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:07.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.642+0000 7f29fd600700 1 -- 172.21.15.40:0/3822073371 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f29f40d0e90 con 0x7f2a0006baf0 2024-04-27T00:00:07.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.643+0000 7f29ffe00700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29e80a3cd0 0x7f29e80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:07.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.643+0000 7f29fd600700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093] conn(0x7f29e80ae990 0x7f29e80b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:07.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.643+0000 7f29fd600700 1 -- 172.21.15.40:0/3822073371 --> [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29f4019ec0 con 0x7f29e80ae990 2024-04-27T00:00:07.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.643+0000 7f29fd600700 1 -- 172.21.15.40:0/3822073371 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f29f40d1240 con 0x7f2a0006baf0 2024-04-27T00:00:07.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.643+0000 7f2a05800700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093] conn(0x7f29e80ae990 0x7f29e80b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:07.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.643+0000 7f29ffe00700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29e80a3cd0 0x7f29e80a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2a00079940 tx=0x7f29ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:07.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.644+0000 7f2a05800700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093] conn(0x7f29e80ae990 0x7f29e80b0dd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:07.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.645+0000 7f29fd600700 1 -- 172.21.15.40:0/3822073371 <== osd.1 v2:172.21.15.80:6800/3368601093 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f29f4019ec0 con 0x7f29e80ae990 2024-04-27T00:00:07.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.663+0000 7f2a07800700 1 -- 172.21.15.40:0/3822073371 --> [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2a00063fa0 con 0x7f29e80ae990 2024-04-27T00:00:07.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.664+0000 7f29fd600700 1 -- 172.21.15.40:0/3822073371 <== osd.1 v2:172.21.15.80:6800/3368601093 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2a00063fa0 con 0x7f29e80ae990 2024-04-27T00:00:07.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.664+0000 7f2a07800700 1 -- 172.21.15.40:0/3822073371 >> [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093] conn(0x7f29e80ae990 msgr2=0x7f29e80b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:07.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.664+0000 7f2a07800700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093] conn(0x7f29e80ae990 0x7f29e80b0dd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:07.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.665+0000 7f2a07800700 1 -- 172.21.15.40:0/3822073371 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29e80a3cd0 msgr2=0x7f29e80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:07.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.665+0000 7f2a07800700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29e80a3cd0 0x7f29e80a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2a00079940 tx=0x7f29ec009260 comp rx=0 tx=0).stop 2024-04-27T00:00:07.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.665+0000 7f2a07800700 1 -- 172.21.15.40:0/3822073371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a0006baf0 msgr2=0x7f2a0019f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:07.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.665+0000 7f2a07800700 1 --2- 172.21.15.40:0/3822073371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a0006baf0 0x7f2a0019f740 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f29f400e9a0 tx=0x7f29f400ecb0 comp rx=0 tx=0).stop 2024-04-27T00:00:07.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.665+0000 7f2a05800700 1 -- 172.21.15.40:0/3822073371 reap_dead start 2024-04-27T00:00:07.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.665+0000 7f2a07800700 1 -- 172.21.15.40:0/3822073371 shutdown_connections 2024-04-27T00:00:07.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.665+0000 7f2a07800700 1 -- 172.21.15.40:0/3822073371 >> 172.21.15.40:0/3822073371 conn(0x7f2a000792d0 msgr2=0x7f2a00100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:07.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.665+0000 7f2a07800700 1 -- 172.21.15.40:0/3822073371 shutdown_connections 2024-04-27T00:00:07.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:07.665+0000 7f2a07800700 1 -- 172.21.15.40:0/3822073371 wait complete. 2024-04-27T00:00:08.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:08 smithi080 ceph-mon[32362]: pgmap v361: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-27T00:00:08.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:08 smithi103 ceph-mon[32952]: pgmap v361: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-27T00:00:08.680 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:08 smithi040 ceph-mon[28095]: pgmap v361: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-27T00:00:08.757 INFO:teuthology.orchestra.run.smithi040.stdout:313532612642 2024-04-27T00:00:08.757 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-27T00:00:09.611 INFO:teuthology.orchestra.run.smithi040.stdout:55834574934 2024-04-27T00:00:09.612 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-27T00:00:10.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.344+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1183023854 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a581024b0 msgr2=0x7f5a58102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:10.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.344+0000 7f5a5ce00700 1 --2- 172.21.15.40:0/1183023854 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a581024b0 0x7f5a58102890 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5a48009990 tx=0x7f5a48009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.344+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1183023854 shutdown_connections 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.344+0000 7f5a5ce00700 1 --2- 172.21.15.40:0/1183023854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a58107590 0x7f5a5810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.344+0000 7f5a5ce00700 1 --2- 172.21.15.40:0/1183023854 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a581024b0 0x7f5a58102890 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.344+0000 7f5a5ce00700 1 --2- 172.21.15.40:0/1183023854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a58102e60 0x7f5a58106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.344+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1183023854 >> 172.21.15.40:0/1183023854 conn(0x7f5a580fdd40 msgr2=0x7f5a58100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.344+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1183023854 shutdown_connections 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.344+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1183023854 wait complete. 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.346+0000 7f5a5ce00700 1 Processor -- start 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.346+0000 7f5a5ce00700 1 -- start start 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.346+0000 7f5a5ce00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a581024b0 0x7f5a581a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.346+0000 7f5a5ce00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a58102e60 0x7f5a581a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.346+0000 7f5a5ce00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a58107590 0x7f5a5810e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:10.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.346+0000 7f5a5ce00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5806bc90 con 0x7f5a581024b0 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.346+0000 7f5a5ce00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5806bb10 con 0x7f5a58107590 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.346+0000 7f5a5ce00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5806be10 con 0x7f5a58102e60 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.347+0000 7f5a56a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a58102e60 0x7f5a581a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.347+0000 7f5a57400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a581024b0 0x7f5a581a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.347+0000 7f5a56a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a58102e60 0x7f5a581a1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37742/0 (socket says 172.21.15.40:37742) 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.347+0000 7f5a56a00700 1 -- 172.21.15.40:0/1787668514 learned_addr learned my addr 172.21.15.40:0/1787668514 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.347+0000 7f5a57e00700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a58107590 0x7f5a5810e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.347+0000 7f5a56a00700 1 -- 172.21.15.40:0/1787668514 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a58107590 msgr2=0x7f5a5810e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.347+0000 7f5a56a00700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a58107590 0x7f5a5810e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.347+0000 7f5a56a00700 1 -- 172.21.15.40:0/1787668514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a581024b0 msgr2=0x7f5a581a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.347+0000 7f5a56a00700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a581024b0 0x7f5a581a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.347+0000 7f5a56a00700 1 -- 172.21.15.40:0/1787668514 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5810ecd0 con 0x7f5a58102e60 2024-04-27T00:00:10.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.347+0000 7f5a57e00700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5a58107590 0x7f5a5810e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:00:10.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.348+0000 7f5a56a00700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a58102e60 0x7f5a581a1e60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400e9f0 tx=0x7f5a4400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:10.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.348+0000 7f5a4fe00700 1 -- 172.21.15.40:0/1787668514 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4400cbd0 con 0x7f5a58102e60 2024-04-27T00:00:10.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.348+0000 7f5a4fe00700 1 -- 172.21.15.40:0/1787668514 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a4400cd70 con 0x7f5a58102e60 2024-04-27T00:00:10.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.348+0000 7f5a4fe00700 1 -- 172.21.15.40:0/1787668514 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44005650 con 0x7f5a58102e60 2024-04-27T00:00:10.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.348+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1787668514 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a5810efc0 con 0x7f5a58102e60 2024-04-27T00:00:10.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.348+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1787668514 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a581a87a0 con 0x7f5a58102e60 2024-04-27T00:00:10.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.349+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1787668514 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5a58063bb0 con 0x7f5a58102e60 2024-04-27T00:00:10.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.352+0000 7f5a4fe00700 1 -- 172.21.15.40:0/1787668514 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5a44004750 con 0x7f5a58102e60 2024-04-27T00:00:10.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.353+0000 7f5a4fe00700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5a3c0a3cd0 0x7f5a3c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.353+0000 7f5a4fe00700 1 -- 172.21.15.40:0/1787668514 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f5a440d0e00 con 0x7f5a58102e60 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.354+0000 7f5a57400700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5a3c0a3cd0 0x7f5a3c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.354+0000 7f5a4fe00700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516] conn(0x7f5a3c0ae990 0x7f5a3c0b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.354+0000 7f5a4fe00700 1 -- 172.21.15.40:0/1787668514 --> [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a3c0b14a0 con 0x7f5a3c0ae990 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.354+0000 7f5a4fe00700 1 -- 172.21.15.40:0/1787668514 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f5a440d11b0 con 0x7f5a58102e60 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.354+0000 7f5a57e00700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516] conn(0x7f5a3c0ae990 0x7f5a3c0b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.355+0000 7f5a57400700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5a3c0a3cd0 0x7f5a3c0a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800b790 tx=0x7f5a48005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.355+0000 7f5a57e00700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516] conn(0x7f5a3c0ae990 0x7f5a3c0b0dd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.357+0000 7f5a4fe00700 1 -- 172.21.15.40:0/1787668514 <== osd.6 v2:172.21.15.40:6818/3383463516 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5a3c0b14a0 con 0x7f5a3c0ae990 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.377+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1787668514 --> [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5a58064030 con 0x7f5a3c0ae990 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.377+0000 7f5a4fe00700 1 -- 172.21.15.40:0/1787668514 <== osd.6 v2:172.21.15.40:6818/3383463516 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5a58064030 con 0x7f5a3c0ae990 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.378+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1787668514 >> [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516] conn(0x7f5a3c0ae990 msgr2=0x7f5a3c0b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:10.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.378+0000 7f5a5ce00700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516] conn(0x7f5a3c0ae990 0x7f5a3c0b0dd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:10.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.378+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1787668514 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5a3c0a3cd0 msgr2=0x7f5a3c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:10.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.378+0000 7f5a5ce00700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5a3c0a3cd0 0x7f5a3c0a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800b790 tx=0x7f5a48005fd0 comp rx=0 tx=0).stop 2024-04-27T00:00:10.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.378+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1787668514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a58102e60 msgr2=0x7f5a581a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:10.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.378+0000 7f5a5ce00700 1 --2- 172.21.15.40:0/1787668514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a58102e60 0x7f5a581a1e60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400e9f0 tx=0x7f5a4400ed00 comp rx=0 tx=0).stop 2024-04-27T00:00:10.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.378+0000 7f5a57e00700 1 -- 172.21.15.40:0/1787668514 reap_dead start 2024-04-27T00:00:10.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.378+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1787668514 shutdown_connections 2024-04-27T00:00:10.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.378+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1787668514 >> 172.21.15.40:0/1787668514 conn(0x7f5a580fdd40 msgr2=0x7f5a58112a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:10.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.379+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1787668514 shutdown_connections 2024-04-27T00:00:10.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:09.379+0000 7f5a5ce00700 1 -- 172.21.15.40:0/1787668514 wait complete. 2024-04-27T00:00:10.376 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:10 smithi040 ceph-mon[28095]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:10.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:10 smithi080 ceph-mon[32362]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:10.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:10 smithi103 ceph-mon[32952]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:11.270 INFO:teuthology.orchestra.run.smithi040.stdout:201863462971 2024-04-27T00:00:11.270 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-27T00:00:12.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.082+0000 7ff56f000700 1 -- 172.21.15.40:0/2925616344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56806b130 msgr2=0x7ff56806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:12.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.082+0000 7ff56f000700 1 --2- 172.21.15.40:0/2925616344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56806b130 0x7ff56806b590 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff554009990 tx=0x7ff554009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:12.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.082+0000 7ff56f000700 1 -- 172.21.15.40:0/2925616344 shutdown_connections 2024-04-27T00:00:12.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.082+0000 7ff56f000700 1 --2- 172.21.15.40:0/2925616344 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff56806bad0 0x7ff5681102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:12.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.082+0000 7ff56f000700 1 --2- 172.21.15.40:0/2925616344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5681094a0 0x7ff568109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:12.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.082+0000 7ff56f000700 1 --2- 172.21.15.40:0/2925616344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56806b130 0x7ff56806b590 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:12.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.082+0000 7ff56f000700 1 -- 172.21.15.40:0/2925616344 >> 172.21.15.40:0/2925616344 conn(0x7ff5680792d0 msgr2=0x7ff5680796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:12.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.082+0000 7ff56f000700 1 -- 172.21.15.40:0/2925616344 shutdown_connections 2024-04-27T00:00:12.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.082+0000 7ff56f000700 1 -- 172.21.15.40:0/2925616344 wait complete. 2024-04-27T00:00:12.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.083+0000 7ff56f000700 1 Processor -- start 2024-04-27T00:00:12.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.083+0000 7ff56f000700 1 -- start start 2024-04-27T00:00:12.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.083+0000 7ff56f000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56806b130 0x7ff5681023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:12.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff56f000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56806bad0 0x7ff568102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff56f000700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5681094a0 0x7ff568102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff56f000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff5681135d0 con 0x7ff56806bad0 2024-04-27T00:00:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff56f000700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7ff568113450 con 0x7ff5681094a0 2024-04-27T00:00:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff56f000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff568113750 con 0x7ff56806b130 2024-04-27T00:00:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff567400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56806bad0 0x7ff568102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff56d000700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5681094a0 0x7ff568102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff567400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56806bad0 0x7ff568102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59734/0 (socket says 172.21.15.40:59734) 2024-04-27T00:00:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff567400700 1 -- 172.21.15.40:0/4161689977 learned_addr learned my addr 172.21.15.40:0/4161689977 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff567e00700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56806b130 0x7ff5681023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff56d000700 1 -- 172.21.15.40:0/4161689977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56806b130 msgr2=0x7ff5681023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff56d000700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff56806b130 0x7ff5681023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff56d000700 1 -- 172.21.15.40:0/4161689977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56806bad0 msgr2=0x7ff568102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:12.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff56d000700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56806bad0 0x7ff568102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:12.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff56d000700 1 -- 172.21.15.40:0/4161689977 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5681a8000 con 0x7ff5681094a0 2024-04-27T00:00:12.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.084+0000 7ff567400700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff56806bad0 0x7ff568102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:12.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.085+0000 7ff56d000700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5681094a0 0x7ff568102e70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff56000efc0 tx=0x7ff56000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:12.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.085+0000 7ff564c00700 1 -- 172.21.15.40:0/4161689977 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5600092e0 con 0x7ff5681094a0 2024-04-27T00:00:12.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.085+0000 7ff564c00700 1 -- 172.21.15.40:0/4161689977 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff560004bb0 con 0x7ff5681094a0 2024-04-27T00:00:12.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.085+0000 7ff56f000700 1 -- 172.21.15.40:0/4161689977 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5681a8290 con 0x7ff5681094a0 2024-04-27T00:00:12.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.085+0000 7ff564c00700 1 -- 172.21.15.40:0/4161689977 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560004020 con 0x7ff5681094a0 2024-04-27T00:00:12.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.085+0000 7ff56f000700 1 -- 172.21.15.40:0/4161689977 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5681a87d0 con 0x7ff5681094a0 2024-04-27T00:00:12.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.086+0000 7ff56f000700 1 -- 172.21.15.40:0/4161689977 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff568063b30 con 0x7ff5681094a0 2024-04-27T00:00:12.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.088+0000 7ff564c00700 1 -- 172.21.15.40:0/4161689977 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff56001b030 con 0x7ff5681094a0 2024-04-27T00:00:12.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.090+0000 7ff564c00700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff54c0a3d50 0x7ff54c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:12.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.090+0000 7ff564c00700 1 -- 172.21.15.40:0/4161689977 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff5600d0280 con 0x7ff5681094a0 2024-04-27T00:00:12.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.090+0000 7ff567e00700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff54c0a3d50 0x7ff54c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:12.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.090+0000 7ff564c00700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438] conn(0x7ff54c0aea10 0x7ff54c0b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:12.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.090+0000 7ff564c00700 1 -- 172.21.15.40:0/4161689977 --> [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff56001be50 con 0x7ff54c0aea10 2024-04-27T00:00:12.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.090+0000 7ff564c00700 1 -- 172.21.15.40:0/4161689977 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7ff5600d7050 con 0x7ff5681094a0 2024-04-27T00:00:12.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.091+0000 7ff567400700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438] conn(0x7ff54c0aea10 0x7ff54c0b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:12.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.091+0000 7ff567e00700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff54c0a3d50 0x7ff54c0a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff558009be0 tx=0x7ff558009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:12.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.091+0000 7ff567400700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438] conn(0x7ff54c0aea10 0x7ff54c0b0e50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:12.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.092+0000 7ff564c00700 1 -- 172.21.15.40:0/4161689977 <== osd.5 v2:172.21.15.103:6808/63420438 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff56001be50 con 0x7ff54c0aea10 2024-04-27T00:00:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.111+0000 7ff56f000700 1 -- 172.21.15.40:0/4161689977 --> [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff568063fb0 con 0x7ff54c0aea10 2024-04-27T00:00:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.112+0000 7ff564c00700 1 -- 172.21.15.40:0/4161689977 <== osd.5 v2:172.21.15.103:6808/63420438 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff568063fb0 con 0x7ff54c0aea10 2024-04-27T00:00:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.112+0000 7ff56f000700 1 -- 172.21.15.40:0/4161689977 >> [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438] conn(0x7ff54c0aea10 msgr2=0x7ff54c0b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.112+0000 7ff56f000700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438] conn(0x7ff54c0aea10 0x7ff54c0b0e50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.113+0000 7ff56f000700 1 -- 172.21.15.40:0/4161689977 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff54c0a3d50 msgr2=0x7ff54c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.113+0000 7ff56f000700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff54c0a3d50 0x7ff54c0a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff558009be0 tx=0x7ff558009260 comp rx=0 tx=0).stop 2024-04-27T00:00:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.113+0000 7ff56f000700 1 -- 172.21.15.40:0/4161689977 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5681094a0 msgr2=0x7ff568102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.113+0000 7ff56f000700 1 --2- 172.21.15.40:0/4161689977 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff5681094a0 0x7ff568102e70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff56000efc0 tx=0x7ff56000c3a0 comp rx=0 tx=0).stop 2024-04-27T00:00:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.113+0000 7ff56d000700 1 -- 172.21.15.40:0/4161689977 reap_dead start 2024-04-27T00:00:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.113+0000 7ff56f000700 1 -- 172.21.15.40:0/4161689977 shutdown_connections 2024-04-27T00:00:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.113+0000 7ff56f000700 1 -- 172.21.15.40:0/4161689977 >> 172.21.15.40:0/4161689977 conn(0x7ff5680792d0 msgr2=0x7ff568100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.113+0000 7ff56f000700 1 -- 172.21.15.40:0/4161689977 shutdown_connections 2024-04-27T00:00:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:12.113+0000 7ff56f000700 1 -- 172.21.15.40:0/4161689977 wait complete. 2024-04-27T00:00:12.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:12 smithi040 ceph-mon[28095]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-27T00:00:12.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:12 smithi080 ceph-mon[32362]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-27T00:00:12.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:12 smithi103 ceph-mon[32952]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-27T00:00:13.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.955+0000 7f45e4800700 1 -- 172.21.15.40:0/1925204434 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f45dc1094c0 msgr2=0x7f45dc1098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:13.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.955+0000 7f45e4800700 1 --2- 172.21.15.40:0/1925204434 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f45dc1094c0 0x7f45dc1098a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009990 tx=0x7f45d0009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.956+0000 7f45e4800700 1 -- 172.21.15.40:0/1925204434 shutdown_connections 2024-04-27T00:00:13.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.956+0000 7f45e4800700 1 --2- 172.21.15.40:0/1925204434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45dc103e30 0x7f45dc110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.956+0000 7f45e4800700 1 --2- 172.21.15.40:0/1925204434 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f45dc1094c0 0x7f45dc1098a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.956+0000 7f45e4800700 1 --2- 172.21.15.40:0/1925204434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45dc103470 0x7f45dc1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.956+0000 7f45e4800700 1 -- 172.21.15.40:0/1925204434 >> 172.21.15.40:0/1925204434 conn(0x7f45dc07a580 msgr2=0x7f45dc07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:13.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.956+0000 7f45e4800700 1 -- 172.21.15.40:0/1925204434 shutdown_connections 2024-04-27T00:00:13.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.956+0000 7f45e4800700 1 -- 172.21.15.40:0/1925204434 wait complete. 2024-04-27T00:00:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.957+0000 7f45e4800700 1 Processor -- start 2024-04-27T00:00:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.957+0000 7f45e4800700 1 -- start start 2024-04-27T00:00:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.957+0000 7f45e4800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45dc103470 0x7f45dc19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.957+0000 7f45e4800700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f45dc103e30 0x7f45dc19fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.957+0000 7f45e4800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45dc1094c0 0x7f45dc199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.957+0000 7f45e4800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc113640 con 0x7f45dc103470 2024-04-27T00:00:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.957+0000 7f45e4800700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1134c0 con 0x7f45dc103e30 2024-04-27T00:00:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.957+0000 7f45e4800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc1137c0 con 0x7f45dc1094c0 2024-04-27T00:00:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.958+0000 7f45e1e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45dc103470 0x7f45dc19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.958+0000 7f45e1400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f45dc103e30 0x7f45dc19fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.958+0000 7f45e1400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f45dc103e30 0x7f45dc19fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:55956/0 (socket says 172.21.15.40:55956) 2024-04-27T00:00:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.958+0000 7f45e1400700 1 -- 172.21.15.40:0/3348432630 learned_addr learned my addr 172.21.15.40:0/3348432630 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.958+0000 7f45e2800700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45dc1094c0 0x7f45dc199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.958+0000 7f45e1400700 1 -- 172.21.15.40:0/3348432630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45dc1094c0 msgr2=0x7f45dc199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.958+0000 7f45e1400700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45dc1094c0 0x7f45dc199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.958+0000 7f45e1400700 1 -- 172.21.15.40:0/3348432630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45dc103470 msgr2=0x7f45dc19f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.958+0000 7f45e1400700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45dc103470 0x7f45dc19f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.958+0000 7f45e1400700 1 -- 172.21.15.40:0/3348432630 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45dc19a0c0 con 0x7f45dc103e30 2024-04-27T00:00:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.959+0000 7f45e1400700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f45dc103e30 0x7f45dc19fce0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00e9c0 tx=0x7f45cc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.959+0000 7f45d6a00700 1 -- 172.21.15.40:0/3348432630 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc00cbd0 con 0x7f45dc103e30 2024-04-27T00:00:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.959+0000 7f45d6a00700 1 -- 172.21.15.40:0/3348432630 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc00ee60 con 0x7f45dc103e30 2024-04-27T00:00:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.959+0000 7f45e4800700 1 -- 172.21.15.40:0/3348432630 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45dc19a3b0 con 0x7f45dc103e30 2024-04-27T00:00:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.959+0000 7f45d6a00700 1 -- 172.21.15.40:0/3348432630 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc0055c0 con 0x7f45dc103e30 2024-04-27T00:00:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.959+0000 7f45e4800700 1 -- 172.21.15.40:0/3348432630 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45dc1a6670 con 0x7f45dc103e30 2024-04-27T00:00:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.960+0000 7f45e4800700 1 -- 172.21.15.40:0/3348432630 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f45dc063b30 con 0x7f45dc103e30 2024-04-27T00:00:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.963+0000 7f45d6a00700 1 -- 172.21.15.40:0/3348432630 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f45cc004750 con 0x7f45dc103e30 2024-04-27T00:00:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.964+0000 7f45d6a00700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f45c40a3cd0 0x7f45c40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.964+0000 7f45d6a00700 1 -- 172.21.15.40:0/3348432630 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f45cc0d0e90 con 0x7f45dc103e30 2024-04-27T00:00:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.965+0000 7f45e1e00700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f45c40a3cd0 0x7f45c40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.965+0000 7f45d6a00700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598] conn(0x7f45c40ae990 0x7f45c40b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.965+0000 7f45d6a00700 1 -- 172.21.15.40:0/3348432630 --> [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45cc01ee50 con 0x7f45c40ae990 2024-04-27T00:00:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.965+0000 7f45d6a00700 1 -- 172.21.15.40:0/3348432630 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f45cc0d1240 con 0x7f45dc103e30 2024-04-27T00:00:13.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.965+0000 7f45e2800700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598] conn(0x7f45c40ae990 0x7f45c40b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:13.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.966+0000 7f45e1e00700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f45c40a3cd0 0x7f45c40a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f45dc19aaa0 tx=0x7f45d000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:13.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.966+0000 7f45e2800700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598] conn(0x7f45c40ae990 0x7f45c40b0dd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:13.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.967+0000 7f45d6a00700 1 -- 172.21.15.40:0/3348432630 <== osd.0 v2:172.21.15.40:6802/1203247598 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45cc01ee50 con 0x7f45c40ae990 2024-04-27T00:00:13.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.985+0000 7f45e4800700 1 -- 172.21.15.40:0/3348432630 --> [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f45dc063fb0 con 0x7f45c40ae990 2024-04-27T00:00:13.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.986+0000 7f45d6a00700 1 -- 172.21.15.40:0/3348432630 <== osd.0 v2:172.21.15.40:6802/1203247598 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f45dc063fb0 con 0x7f45c40ae990 2024-04-27T00:00:13.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.986+0000 7f45e4800700 1 -- 172.21.15.40:0/3348432630 >> [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598] conn(0x7f45c40ae990 msgr2=0x7f45c40b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:13.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.987+0000 7f45e4800700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598] conn(0x7f45c40ae990 0x7f45c40b0dd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.987+0000 7f45e4800700 1 -- 172.21.15.40:0/3348432630 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f45c40a3cd0 msgr2=0x7f45c40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:13.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.987+0000 7f45e4800700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f45c40a3cd0 0x7f45c40a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f45dc19aaa0 tx=0x7f45d000b260 comp rx=0 tx=0).stop 2024-04-27T00:00:13.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.987+0000 7f45e4800700 1 -- 172.21.15.40:0/3348432630 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f45dc103e30 msgr2=0x7f45dc19fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:13.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.987+0000 7f45e4800700 1 --2- 172.21.15.40:0/3348432630 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f45dc103e30 0x7f45dc19fce0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00e9c0 tx=0x7f45cc00ecd0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.987+0000 7f45e2800700 1 -- 172.21.15.40:0/3348432630 reap_dead start 2024-04-27T00:00:13.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.987+0000 7f45e4800700 1 -- 172.21.15.40:0/3348432630 shutdown_connections 2024-04-27T00:00:13.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.987+0000 7f45e4800700 1 -- 172.21.15.40:0/3348432630 >> 172.21.15.40:0/3348432630 conn(0x7f45dc07a580 msgr2=0x7f45dc102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:13.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.987+0000 7f45e4800700 1 -- 172.21.15.40:0/3348432630 shutdown_connections 2024-04-27T00:00:13.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:11.987+0000 7f45e4800700 1 -- 172.21.15.40:0/3348432630 wait complete. 2024-04-27T00:00:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.474+0000 7fce00a00700 1 -- 172.21.15.40:0/1866179073 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcdfc102100 msgr2=0x7fcdfc1024e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.474+0000 7fce00a00700 1 --2- 172.21.15.40:0/1866179073 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcdfc102100 0x7fcdfc1024e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0098f0 tx=0x7fcdec009c00 comp rx=0 tx=0).stop 2024-04-27T00:00:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.474+0000 7fce00a00700 1 -- 172.21.15.40:0/1866179073 shutdown_connections 2024-04-27T00:00:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.474+0000 7fce00a00700 1 --2- 172.21.15.40:0/1866179073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdfc10fcb0 0x7fcdfc1120a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.474+0000 7fce00a00700 1 --2- 172.21.15.40:0/1866179073 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcdfc102100 0x7fcdfc1024e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.474+0000 7fce00a00700 1 --2- 172.21.15.40:0/1866179073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102ab0 0x7fcdfc10f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.474+0000 7fce00a00700 1 -- 172.21.15.40:0/1866179073 >> 172.21.15.40:0/1866179073 conn(0x7fcdfc0fdd40 msgr2=0x7fcdfc100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.474+0000 7fce00a00700 1 -- 172.21.15.40:0/1866179073 shutdown_connections 2024-04-27T00:00:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.475+0000 7fce00a00700 1 -- 172.21.15.40:0/1866179073 wait complete. 2024-04-27T00:00:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.476+0000 7fce00a00700 1 Processor -- start 2024-04-27T00:00:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.476+0000 7fce00a00700 1 -- start start 2024-04-27T00:00:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.477+0000 7fce00a00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcdfc102100 0x7fcdfc1a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.477+0000 7fce00a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102ab0 0x7fcdfc1a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.477+0000 7fce00a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdfc10fcb0 0x7fcdfc19ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.477+0000 7fce00a00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc115790 con 0x7fcdfc102ab0 2024-04-27T00:00:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.477+0000 7fce00a00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc115610 con 0x7fcdfc102100 2024-04-27T00:00:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.477+0000 7fce00a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc115910 con 0x7fcdfc10fcb0 2024-04-27T00:00:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.477+0000 7fcdfb400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcdfc102100 0x7fcdfc1a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.477+0000 7fcdfaa00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102ab0 0x7fcdfc1a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.477+0000 7fcdfb400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcdfc102100 0x7fcdfc1a18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:55984/0 (socket says 172.21.15.40:55984) 2024-04-27T00:00:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.477+0000 7fcdfb400700 1 -- 172.21.15.40:0/3137425228 learned_addr learned my addr 172.21.15.40:0/3137425228 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.477+0000 7fcdfbe00700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdfc10fcb0 0x7fcdfc19ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.478+0000 7fcdfb400700 1 -- 172.21.15.40:0/3137425228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdfc10fcb0 msgr2=0x7fcdfc19ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:13.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.478+0000 7fcdfb400700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdfc10fcb0 0x7fcdfc19ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.478+0000 7fcdfb400700 1 -- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102ab0 msgr2=0x7fcdfc1a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:13.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.478+0000 7fcdfb400700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102ab0 0x7fcdfc1a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.478+0000 7fcdfb400700 1 -- 172.21.15.40:0/3137425228 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdfc19c1d0 con 0x7fcdfc102100 2024-04-27T00:00:13.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.478+0000 7fcdfaa00700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102ab0 0x7fcdfc1a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:13.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.478+0000 7fcdfb400700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcdfc102100 0x7fcdfc1a18b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009230 tx=0x7fcdec00fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:13.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.478+0000 7fcdf3e00700 1 -- 172.21.15.40:0/3137425228 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec005140 con 0x7fcdfc102100 2024-04-27T00:00:13.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.478+0000 7fcdf3e00700 1 -- 172.21.15.40:0/3137425228 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdec00b560 con 0x7fcdfc102100 2024-04-27T00:00:13.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.478+0000 7fce00a00700 1 -- 172.21.15.40:0/3137425228 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdfc19c460 con 0x7fcdfc102100 2024-04-27T00:00:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.478+0000 7fcdf3e00700 1 -- 172.21.15.40:0/3137425228 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec003970 con 0x7fcdfc102100 2024-04-27T00:00:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.479+0000 7fce00a00700 1 -- 172.21.15.40:0/3137425228 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdfc1a8780 con 0x7fcdfc102100 2024-04-27T00:00:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.480+0000 7fce00a00700 1 -- 172.21.15.40:0/3137425228 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcdfc063bb0 con 0x7fcdfc102100 2024-04-27T00:00:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.482+0000 7fcdf3e00700 1 -- 172.21.15.40:0/3137425228 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcdec00ba50 con 0x7fcdfc102100 2024-04-27T00:00:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.483+0000 7fcdf3e00700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcde00a3d50 0x7fcde00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.484+0000 7fcdf3e00700 1 -- 172.21.15.40:0/3137425228 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fcdec0d1b80 con 0x7fcdfc102100 2024-04-27T00:00:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.484+0000 7fcdfaa00700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcde00a3d50 0x7fcde00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.484+0000 7fcdf3e00700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893] conn(0x7fcde00ae8e0 0x7fcde00b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.484+0000 7fcdf3e00700 1 -- 172.21.15.40:0/3137425228 --> [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcdec001d30 con 0x7fcde00ae8e0 2024-04-27T00:00:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.484+0000 7fcdf3e00700 1 -- 172.21.15.40:0/3137425228 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fcdec0d1f30 con 0x7fcdfc102100 2024-04-27T00:00:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.484+0000 7fcdfbe00700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893] conn(0x7fcde00ae8e0 0x7fcde00b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.484+0000 7fcdfaa00700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcde00a3d50 0x7fcde00a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc103aa0 tx=0x7fcde8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:13.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.485+0000 7fcdfbe00700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893] conn(0x7fcde00ae8e0 0x7fcde00b0d20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:13.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.486+0000 7fcdf3e00700 1 -- 172.21.15.40:0/3137425228 <== osd.3 v2:172.21.15.40:6810/4161271893 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcdec001d30 con 0x7fcde00ae8e0 2024-04-27T00:00:13.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.504+0000 7fce00a00700 1 -- 172.21.15.40:0/3137425228 --> [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcdfc063fb0 con 0x7fcde00ae8e0 2024-04-27T00:00:13.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.505+0000 7fcdf3e00700 1 -- 172.21.15.40:0/3137425228 <== osd.3 v2:172.21.15.40:6810/4161271893 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcdfc063fb0 con 0x7fcde00ae8e0 2024-04-27T00:00:13.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.505+0000 7fce00a00700 1 -- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893] conn(0x7fcde00ae8e0 msgr2=0x7fcde00b0d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:13.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.505+0000 7fce00a00700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893] conn(0x7fcde00ae8e0 0x7fcde00b0d20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:13.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.506+0000 7fce00a00700 1 -- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcde00a3d50 msgr2=0x7fcde00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:13.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.506+0000 7fce00a00700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcde00a3d50 0x7fcde00a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc103aa0 tx=0x7fcde8009310 comp rx=0 tx=0).stop 2024-04-27T00:00:13.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.506+0000 7fce00a00700 1 -- 172.21.15.40:0/3137425228 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcdfc102100 msgr2=0x7fcdfc1a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:13.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.506+0000 7fce00a00700 1 --2- 172.21.15.40:0/3137425228 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcdfc102100 0x7fcdfc1a18b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009230 tx=0x7fcdec00fa70 comp rx=0 tx=0).stop 2024-04-27T00:00:13.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.506+0000 7fcdfbe00700 1 -- 172.21.15.40:0/3137425228 reap_dead start 2024-04-27T00:00:13.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.506+0000 7fce00a00700 1 -- 172.21.15.40:0/3137425228 shutdown_connections 2024-04-27T00:00:13.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.506+0000 7fce00a00700 1 -- 172.21.15.40:0/3137425228 >> 172.21.15.40:0/3137425228 conn(0x7fcdfc0fdd40 msgr2=0x7fcdfc100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:13.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.506+0000 7fce00a00700 1 -- 172.21.15.40:0/3137425228 shutdown_connections 2024-04-27T00:00:13.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:13.506+0000 7fce00a00700 1 -- 172.21.15.40:0/3137425228 wait complete. 2024-04-27T00:00:13.984 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:13.987 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:14.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.030+0000 7fc1df600700 1 -- 172.21.15.40:0/4158463948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8103e30 msgr2=0x7fc1d8110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:14.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.030+0000 7fc1df600700 1 --2- 172.21.15.40:0/4158463948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8103e30 0x7fc1d8110290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc1d000b0a0 tx=0x7fc1d000b3b0 comp rx=0 tx=0).stop 2024-04-27T00:00:14.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.030+0000 7fc1df600700 1 -- 172.21.15.40:0/4158463948 shutdown_connections 2024-04-27T00:00:14.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.030+0000 7fc1df600700 1 --2- 172.21.15.40:0/4158463948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8103e30 0x7fc1d8110290 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:14.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.030+0000 7fc1df600700 1 --2- 172.21.15.40:0/4158463948 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1d81094c0 0x7fc1d81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:14.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.030+0000 7fc1df600700 1 --2- 172.21.15.40:0/4158463948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d8103470 0x7fc1d81038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:14.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.030+0000 7fc1df600700 1 -- 172.21.15.40:0/4158463948 >> 172.21.15.40:0/4158463948 conn(0x7fc1d807a580 msgr2=0x7fc1d807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:14.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.030+0000 7fc1df600700 1 -- 172.21.15.40:0/4158463948 shutdown_connections 2024-04-27T00:00:14.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.031+0000 7fc1df600700 1 -- 172.21.15.40:0/4158463948 wait complete. 2024-04-27T00:00:14.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.032+0000 7fc1df600700 1 Processor -- start 2024-04-27T00:00:14.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.032+0000 7fc1df600700 1 -- start start 2024-04-27T00:00:14.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.032+0000 7fc1df600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8103470 0x7fc1d8075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:14.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.033+0000 7fc1df600700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1d8103e30 0x7fc1d8075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:14.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.033+0000 7fc1df600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d81094c0 0x7fc1d80761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:14.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.033+0000 7fc1df600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d8113610 con 0x7fc1d81094c0 2024-04-27T00:00:14.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.033+0000 7fc1df600700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d8113490 con 0x7fc1d8103e30 2024-04-27T00:00:14.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.033+0000 7fc1df600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d8113790 con 0x7fc1d8103470 2024-04-27T00:00:14.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.033+0000 7fc1d7e00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1d8103e30 0x7fc1d8075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:14.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.033+0000 7fc1d7e00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1d8103e30 0x7fc1d8075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:56002/0 (socket says 172.21.15.40:56002) 2024-04-27T00:00:14.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.033+0000 7fc1d7e00700 1 -- 172.21.15.40:0/4291002907 learned_addr learned my addr 172.21.15.40:0/4291002907 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:14.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.033+0000 7fc1dcc00700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8103470 0x7fc1d8075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:14.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.033+0000 7fc1dd600700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d81094c0 0x7fc1d80761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:14.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.034+0000 7fc1dcc00700 1 -- 172.21.15.40:0/4291002907 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1d8103e30 msgr2=0x7fc1d8075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:14.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.034+0000 7fc1dcc00700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc1d8103e30 0x7fc1d8075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:14.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.034+0000 7fc1dcc00700 1 -- 172.21.15.40:0/4291002907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d81094c0 msgr2=0x7fc1d80761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:14.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.034+0000 7fc1dcc00700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d81094c0 0x7fc1d80761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:14.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.034+0000 7fc1dcc00700 1 -- 172.21.15.40:0/4291002907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d81a7fe0 con 0x7fc1d8103470 2024-04-27T00:00:14.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.034+0000 7fc1dd600700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d81094c0 0x7fc1d80761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:14.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.034+0000 7fc1dcc00700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8103470 0x7fc1d8075720 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc00e9f0 tx=0x7fc1cc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:14.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.034+0000 7fc1d5600700 1 -- 172.21.15.40:0/4291002907 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc00c9c0 con 0x7fc1d8103470 2024-04-27T00:00:14.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.034+0000 7fc1d5600700 1 -- 172.21.15.40:0/4291002907 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1cc00cb60 con 0x7fc1d8103470 2024-04-27T00:00:14.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.034+0000 7fc1df600700 1 -- 172.21.15.40:0/4291002907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d81a82d0 con 0x7fc1d8103470 2024-04-27T00:00:14.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.034+0000 7fc1d5600700 1 -- 172.21.15.40:0/4291002907 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc018680 con 0x7fc1d8103470 2024-04-27T00:00:14.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.034+0000 7fc1df600700 1 -- 172.21.15.40:0/4291002907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d81a8860 con 0x7fc1d8103470 2024-04-27T00:00:14.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.035+0000 7fc1df600700 1 -- 172.21.15.40:0/4291002907 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc1b8001030 con 0x7fc1d8103470 2024-04-27T00:00:14.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.038+0000 7fc1d5600700 1 -- 172.21.15.40:0/4291002907 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc1cc018820 con 0x7fc1d8103470 2024-04-27T00:00:14.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.039+0000 7fc1d5600700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc1c00a3d50 0x7fc1c00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:14.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.040+0000 7fc1d5600700 1 -- 172.21.15.40:0/4291002907 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc1cc0d1290 con 0x7fc1d8103470 2024-04-27T00:00:14.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.040+0000 7fc1d7e00700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc1c00a3d50 0x7fc1c00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:14.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.040+0000 7fc1d5600700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187] conn(0x7fc1c00aea10 0x7fc1c00b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:14.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.040+0000 7fc1d5600700 1 -- 172.21.15.40:0/4291002907 --> [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc1cc018e50 con 0x7fc1c00aea10 2024-04-27T00:00:14.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.040+0000 7fc1d5600700 1 -- 172.21.15.40:0/4291002907 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fc1cc0d1640 con 0x7fc1d8103470 2024-04-27T00:00:14.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.040+0000 7fc1dd600700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187] conn(0x7fc1c00aea10 0x7fc1c00b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:14.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.041+0000 7fc1d7e00700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc1c00a3d50 0x7fc1c00a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc1d807ab80 tx=0x7fc1c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:14.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.041+0000 7fc1dd600700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187] conn(0x7fc1c00aea10 0x7fc1c00b0e50 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:14.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.042+0000 7fc1d5600700 1 -- 172.21.15.40:0/4291002907 <== osd.2 v2:172.21.15.103:6800/3393467187 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc1cc018e50 con 0x7fc1c00aea10 2024-04-27T00:00:14.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.060+0000 7fc1df600700 1 -- 172.21.15.40:0/4291002907 --> [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc1b8002e60 con 0x7fc1c00aea10 2024-04-27T00:00:14.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.061+0000 7fc1d5600700 1 -- 172.21.15.40:0/4291002907 <== osd.2 v2:172.21.15.103:6800/3393467187 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc1b8002e60 con 0x7fc1c00aea10 2024-04-27T00:00:14.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.061+0000 7fc1df600700 1 -- 172.21.15.40:0/4291002907 >> [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187] conn(0x7fc1c00aea10 msgr2=0x7fc1c00b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:14.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.061+0000 7fc1df600700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187] conn(0x7fc1c00aea10 0x7fc1c00b0e50 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:14.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.061+0000 7fc1df600700 1 -- 172.21.15.40:0/4291002907 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc1c00a3d50 msgr2=0x7fc1c00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:14.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.061+0000 7fc1df600700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc1c00a3d50 0x7fc1c00a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc1d807ab80 tx=0x7fc1c4009260 comp rx=0 tx=0).stop 2024-04-27T00:00:14.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.061+0000 7fc1df600700 1 -- 172.21.15.40:0/4291002907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8103470 msgr2=0x7fc1d8075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:14.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.061+0000 7fc1df600700 1 --2- 172.21.15.40:0/4291002907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc1d8103470 0x7fc1d8075720 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc00e9f0 tx=0x7fc1cc00ed00 comp rx=0 tx=0).stop 2024-04-27T00:00:14.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.062+0000 7fc1dd600700 1 -- 172.21.15.40:0/4291002907 reap_dead start 2024-04-27T00:00:14.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.062+0000 7fc1df600700 1 -- 172.21.15.40:0/4291002907 shutdown_connections 2024-04-27T00:00:14.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.062+0000 7fc1df600700 1 -- 172.21.15.40:0/4291002907 >> 172.21.15.40:0/4291002907 conn(0x7fc1d807a580 msgr2=0x7fc1d8102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:14.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.062+0000 7fc1df600700 1 -- 172.21.15.40:0/4291002907 shutdown_connections 2024-04-27T00:00:14.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:14.062+0000 7fc1df600700 1 -- 172.21.15.40:0/4291002907 wait complete. 2024-04-27T00:00:14.411 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:14.439 INFO:teuthology.orchestra.run.smithi040.stdout:34359738460 2024-04-27T00:00:14.439 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-27T00:00:14.499 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:14 smithi040 ceph-mon[28095]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:14.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:14 smithi080 ceph-mon[32362]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:14.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:14 smithi103 ceph-mon[32952]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:15.250 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:15.251 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:15.254 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:15.785 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:16.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:16 smithi040 ceph-mon[28095]: pgmap v365: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-27T00:00:16.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:16 smithi080 ceph-mon[32362]: pgmap v365: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-27T00:00:16.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:16 smithi103 ceph-mon[32952]: pgmap v365: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-27T00:00:18.024 INFO:teuthology.orchestra.run.smithi040.stdout:167503724609 2024-04-27T00:00:18.024 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-27T00:00:18.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:18 smithi040 ceph-mon[28095]: pgmap v366: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:18.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:18 smithi080 ceph-mon[32362]: pgmap v366: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:18.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:18 smithi103 ceph-mon[32952]: pgmap v366: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:19.101 INFO:teuthology.orchestra.run.smithi040.stdout:77309411412 2024-04-27T00:00:19.101 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-27T00:00:20.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:20 smithi080 ceph-mon[32362]: pgmap v367: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-27T00:00:20.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:20 smithi103 ceph-mon[32952]: pgmap v367: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-27T00:00:20.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:20 smithi040 ceph-mon[28095]: pgmap v367: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-27T00:00:21.071 INFO:teuthology.orchestra.run.smithi040.stdout:111669149772 2024-04-27T00:00:21.071 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-27T00:00:21.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.175+0000 7fb2b1c00700 1 -- 172.21.15.40:0/1942231794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac075f70 msgr2=0x7fb2ac1140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:21.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.175+0000 7fb2b1c00700 1 --2- 172.21.15.40:0/1942231794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac075f70 0x7fb2ac1140b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb298009990 tx=0x7fb298009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:21.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.175+0000 7fb2b1c00700 1 -- 172.21.15.40:0/1942231794 shutdown_connections 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.175+0000 7fb2b1c00700 1 --2- 172.21.15.40:0/1942231794 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2ac1145f0 0x7fb2ac1169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.175+0000 7fb2b1c00700 1 --2- 172.21.15.40:0/1942231794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2ac075650 0x7fb2ac075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.175+0000 7fb2b1c00700 1 --2- 172.21.15.40:0/1942231794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac075f70 0x7fb2ac1140b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.175+0000 7fb2b1c00700 1 -- 172.21.15.40:0/1942231794 >> 172.21.15.40:0/1942231794 conn(0x7fb2ac07a580 msgr2=0x7fb2ac07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.175+0000 7fb2b1c00700 1 -- 172.21.15.40:0/1942231794 shutdown_connections 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.175+0000 7fb2b1c00700 1 -- 172.21.15.40:0/1942231794 wait complete. 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.177+0000 7fb2b1c00700 1 Processor -- start 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.177+0000 7fb2b1c00700 1 -- start start 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.177+0000 7fb2b1c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac075650 0x7fb2ac1a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.177+0000 7fb2b1c00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2ac075f70 0x7fb2ac1a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.177+0000 7fb2b1c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2ac1145f0 0x7fb2ac1a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.177+0000 7fb2b1c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac119c30 con 0x7fb2ac075650 2024-04-27T00:00:21.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.177+0000 7fb2b1c00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac119ab0 con 0x7fb2ac075f70 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.177+0000 7fb2b1c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac119db0 con 0x7fb2ac1145f0 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.177+0000 7fb2aaa00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac075650 0x7fb2ac1a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.177+0000 7fb2aa000700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2ac075f70 0x7fb2ac1a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.178+0000 7fb2aa000700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2ac075f70 0x7fb2ac1a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:33122/0 (socket says 172.21.15.40:33122) 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.178+0000 7fb2aa000700 1 -- 172.21.15.40:0/951381930 learned_addr learned my addr 172.21.15.40:0/951381930 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.178+0000 7fb2ab400700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2ac1145f0 0x7fb2ac1a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.178+0000 7fb2aa000700 1 -- 172.21.15.40:0/951381930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2ac1145f0 msgr2=0x7fb2ac1a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.178+0000 7fb2aa000700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2ac1145f0 0x7fb2ac1a4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.178+0000 7fb2aa000700 1 -- 172.21.15.40:0/951381930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac075650 msgr2=0x7fb2ac1a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.178+0000 7fb2aa000700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac075650 0x7fb2ac1a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.178+0000 7fb2aa000700 1 -- 172.21.15.40:0/951381930 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ac1a5190 con 0x7fb2ac075f70 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.178+0000 7fb2aa000700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2ac075f70 0x7fb2ac1a06c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb298000c00 tx=0x7fb2980057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.179+0000 7fb2a3400700 1 -- 172.21.15.40:0/951381930 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb298027070 con 0x7fb2ac075f70 2024-04-27T00:00:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.179+0000 7fb2b1c00700 1 -- 172.21.15.40:0/951381930 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2ac1a5420 con 0x7fb2ac075f70 2024-04-27T00:00:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.179+0000 7fb2b1c00700 1 -- 172.21.15.40:0/951381930 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2ac1acd00 con 0x7fb2ac075f70 2024-04-27T00:00:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.179+0000 7fb2a3400700 1 -- 172.21.15.40:0/951381930 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb298009510 con 0x7fb2ac075f70 2024-04-27T00:00:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.179+0000 7fb2a3400700 1 -- 172.21.15.40:0/951381930 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29802bc10 con 0x7fb2ac075f70 2024-04-27T00:00:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.180+0000 7fb2b1c00700 1 -- 172.21.15.40:0/951381930 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb28c005330 con 0x7fb2ac075f70 2024-04-27T00:00:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.182+0000 7fb2a3400700 1 -- 172.21.15.40:0/951381930 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb29802a070 con 0x7fb2ac075f70 2024-04-27T00:00:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.183+0000 7fb2a3400700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb2900a3d50 0x7fb2900a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.184+0000 7fb2a3400700 1 -- 172.21.15.40:0/951381930 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb2980d1330 con 0x7fb2ac075f70 2024-04-27T00:00:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.184+0000 7fb2aaa00700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb2900a3d50 0x7fb2900a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.184+0000 7fb2aaa00700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb2900a3d50 0x7fb2900a6210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009680 tx=0x7fb29c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.185+0000 7fb2a3400700 1 -- 172.21.15.40:0/951381930 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb298060060 con 0x7fb2ac075f70 2024-04-27T00:00:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.435+0000 7fb2b1c00700 1 -- 172.21.15.40:0/951381930 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fb28c0058f0 con 0x7fb2ac075f70 2024-04-27T00:00:21.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.436+0000 7fb2a3400700 1 -- 172.21.15.40:0/951381930 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb298013070 con 0x7fb2ac075f70 2024-04-27T00:00:21.437 INFO:teuthology.orchestra.run.smithi040.stdout:257698037810 2024-04-27T00:00:21.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.438+0000 7fb2b1c00700 1 -- 172.21.15.40:0/951381930 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb2900a3d50 msgr2=0x7fb2900a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:21.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.438+0000 7fb2b1c00700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb2900a3d50 0x7fb2900a6210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009680 tx=0x7fb29c009260 comp rx=0 tx=0).stop 2024-04-27T00:00:21.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.438+0000 7fb2b1c00700 1 -- 172.21.15.40:0/951381930 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2ac075f70 msgr2=0x7fb2ac1a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:21.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.438+0000 7fb2b1c00700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2ac075f70 0x7fb2ac1a06c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb298000c00 tx=0x7fb2980057c0 comp rx=0 tx=0).stop 2024-04-27T00:00:21.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.438+0000 7fb2b1c00700 1 -- 172.21.15.40:0/951381930 shutdown_connections 2024-04-27T00:00:21.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.438+0000 7fb2b1c00700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb2900a3d50 0x7fb2900a6210 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:21.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.438+0000 7fb2b1c00700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2ac1145f0 0x7fb2ac1a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:21.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.438+0000 7fb2b1c00700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac075650 0x7fb2ac1a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:21.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.438+0000 7fb2b1c00700 1 --2- 172.21.15.40:0/951381930 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2ac075f70 0x7fb2ac1a06c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:21.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.438+0000 7fb2b1c00700 1 -- 172.21.15.40:0/951381930 >> 172.21.15.40:0/951381930 conn(0x7fb2ac07a580 msgr2=0x7fb2ac113490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:21.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.439+0000 7fb2b1c00700 1 -- 172.21.15.40:0/951381930 shutdown_connections 2024-04-27T00:00:21.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:21.439+0000 7fb2b1c00700 1 -- 172.21.15.40:0/951381930 wait complete. 2024-04-27T00:00:22.385 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:22 smithi040 ceph-mon[28095]: pgmap v368: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-27T00:00:22.385 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:22 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/951381930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-27T00:00:22.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:22 smithi080 ceph-mon[32362]: pgmap v368: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-27T00:00:22.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:22 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/951381930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-27T00:00:22.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:22 smithi103 ceph-mon[32952]: pgmap v368: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-27T00:00:22.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:22 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/951381930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-27T00:00:22.960 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:23.658 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037807 got 257698037810 for osd.8 2024-04-27T00:00:23.659 DEBUG:teuthology.parallel:result is None 2024-04-27T00:00:24.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:24 smithi040 ceph-mon[28095]: pgmap v369: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-27T00:00:24.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:24 smithi080 ceph-mon[32362]: pgmap v369: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-27T00:00:24.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:24 smithi103 ceph-mon[32952]: pgmap v369: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-27T00:00:25.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.796+0000 7ff42d600700 1 -- 172.21.15.40:0/2146867635 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff428075f70 msgr2=0x7ff4281140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:25.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.796+0000 7ff42d600700 1 --2- 172.21.15.40:0/2146867635 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff428075f70 0x7ff4281140b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff414009990 tx=0x7ff414009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:25.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.796+0000 7ff42d600700 1 -- 172.21.15.40:0/2146867635 shutdown_connections 2024-04-27T00:00:25.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.796+0000 7ff42d600700 1 --2- 172.21.15.40:0/2146867635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4281145f0 0x7ff4281169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:25.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.796+0000 7ff42d600700 1 --2- 172.21.15.40:0/2146867635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff428075650 0x7ff428075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:25.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.796+0000 7ff42d600700 1 --2- 172.21.15.40:0/2146867635 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff428075f70 0x7ff4281140b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:25.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.796+0000 7ff42d600700 1 -- 172.21.15.40:0/2146867635 >> 172.21.15.40:0/2146867635 conn(0x7ff42807a580 msgr2=0x7ff42807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:25.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.796+0000 7ff42d600700 1 -- 172.21.15.40:0/2146867635 shutdown_connections 2024-04-27T00:00:25.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.796+0000 7ff42d600700 1 -- 172.21.15.40:0/2146867635 wait complete. 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.797+0000 7ff42d600700 1 Processor -- start 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.797+0000 7ff42d600700 1 -- start start 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.798+0000 7ff42d600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff428075650 0x7ff4281a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.798+0000 7ff42d600700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff428075f70 0x7ff4281a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.798+0000 7ff42d600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4281145f0 0x7ff4281a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.798+0000 7ff42d600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff428119c40 con 0x7ff4281145f0 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.798+0000 7ff42d600700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7ff428119ac0 con 0x7ff428075f70 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.798+0000 7ff42d600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff428119dc0 con 0x7ff428075650 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.798+0000 7ff427400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4281145f0 0x7ff4281a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.798+0000 7ff426000700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff428075f70 0x7ff4281a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.798+0000 7ff426a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff428075650 0x7ff4281a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.798+0000 7ff427400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4281145f0 0x7ff4281a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48630/0 (socket says 172.21.15.40:48630) 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.798+0000 7ff427400700 1 -- 172.21.15.40:0/1691031766 learned_addr learned my addr 172.21.15.40:0/1691031766 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:25.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.798+0000 7ff426000700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff428075f70 0x7ff4281a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:33136/0 (socket says 172.21.15.40:33136) 2024-04-27T00:00:25.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.799+0000 7ff426000700 1 -- 172.21.15.40:0/1691031766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff428075650 msgr2=0x7ff4281a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:25.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.799+0000 7ff426000700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff428075650 0x7ff4281a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:25.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.799+0000 7ff426000700 1 -- 172.21.15.40:0/1691031766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4281145f0 msgr2=0x7ff4281a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:25.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.799+0000 7ff426000700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4281145f0 0x7ff4281a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:25.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.799+0000 7ff426000700 1 -- 172.21.15.40:0/1691031766 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4281a51a0 con 0x7ff428075f70 2024-04-27T00:00:25.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.799+0000 7ff426000700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff428075f70 0x7ff4281a06d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff4140098e0 tx=0x7ff4140057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:25.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.800+0000 7ff41f400700 1 -- 172.21.15.40:0/1691031766 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff414027070 con 0x7ff428075f70 2024-04-27T00:00:25.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.800+0000 7ff42d600700 1 -- 172.21.15.40:0/1691031766 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4281a5430 con 0x7ff428075f70 2024-04-27T00:00:25.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.800+0000 7ff41f400700 1 -- 172.21.15.40:0/1691031766 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff414022470 con 0x7ff428075f70 2024-04-27T00:00:25.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.800+0000 7ff41f400700 1 -- 172.21.15.40:0/1691031766 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff414017650 con 0x7ff428075f70 2024-04-27T00:00:25.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.800+0000 7ff42d600700 1 -- 172.21.15.40:0/1691031766 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4281acd10 con 0x7ff428075f70 2024-04-27T00:00:25.273 INFO:teuthology.orchestra.run.smithi040.stdout:223338299446 2024-04-27T00:00:25.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.801+0000 7ff42d600700 1 -- 172.21.15.40:0/1691031766 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff428076d40 con 0x7ff428075f70 2024-04-27T00:00:25.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.803+0000 7ff41f400700 1 -- 172.21.15.40:0/1691031766 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff41402a070 con 0x7ff428075f70 2024-04-27T00:00:25.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.805+0000 7ff41f400700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff40c0a3cd0 0x7ff40c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:25.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.805+0000 7ff41f400700 1 -- 172.21.15.40:0/1691031766 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff4140d16c0 con 0x7ff428075f70 2024-04-27T00:00:25.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.805+0000 7ff426a00700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff40c0a3cd0 0x7ff40c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.806+0000 7ff426a00700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff40c0a3cd0 0x7ff40c0a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff418009c60 tx=0x7ff418009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:24.807+0000 7ff41f400700 1 -- 172.21.15.40:0/1691031766 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff414031080 con 0x7ff428075f70 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.061+0000 7ff42d600700 1 -- 172.21.15.40:0/1691031766 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ff4281acee0 con 0x7ff428075f70 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.061+0000 7ff41f400700 1 -- 172.21.15.40:0/1691031766 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff414013070 con 0x7ff428075f70 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.065+0000 7ff42d600700 1 -- 172.21.15.40:0/1691031766 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff40c0a3cd0 msgr2=0x7ff40c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.065+0000 7ff42d600700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff40c0a3cd0 0x7ff40c0a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff418009c60 tx=0x7ff418009260 comp rx=0 tx=0).stop 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.065+0000 7ff42d600700 1 -- 172.21.15.40:0/1691031766 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff428075f70 msgr2=0x7ff4281a06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.065+0000 7ff42d600700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff428075f70 0x7ff4281a06d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff4140098e0 tx=0x7ff4140057c0 comp rx=0 tx=0).stop 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.065+0000 7ff42d600700 1 -- 172.21.15.40:0/1691031766 shutdown_connections 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.065+0000 7ff42d600700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff40c0a3cd0 0x7ff40c0a6190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.065+0000 7ff42d600700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff4281145f0 0x7ff4281a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.065+0000 7ff42d600700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff428075650 0x7ff4281a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:25.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.065+0000 7ff42d600700 1 --2- 172.21.15.40:0/1691031766 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff428075f70 0x7ff4281a06d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:25.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.065+0000 7ff42d600700 1 -- 172.21.15.40:0/1691031766 >> 172.21.15.40:0/1691031766 conn(0x7ff42807a580 msgr2=0x7ff428110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:25.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.065+0000 7ff42d600700 1 -- 172.21.15.40:0/1691031766 shutdown_connections 2024-04-27T00:00:25.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:25.065+0000 7ff42d600700 1 -- 172.21.15.40:0/1691031766 wait complete. 2024-04-27T00:00:25.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:25 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1691031766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-27T00:00:25.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:25 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1691031766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-27T00:00:25.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:25 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1691031766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-27T00:00:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.368+0000 7fef30400700 1 -- 172.21.15.40:0/3118854154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2806b130 msgr2=0x7fef2806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.368+0000 7fef30400700 1 --2- 172.21.15.40:0/3118854154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2806b130 0x7fef2806b590 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fef14009990 tx=0x7fef14009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.368+0000 7fef30400700 1 -- 172.21.15.40:0/3118854154 shutdown_connections 2024-04-27T00:00:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.368+0000 7fef30400700 1 --2- 172.21.15.40:0/3118854154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef2806bad0 0x7fef281102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.368+0000 7fef30400700 1 --2- 172.21.15.40:0/3118854154 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef281094a0 0x7fef28109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.368+0000 7fef30400700 1 --2- 172.21.15.40:0/3118854154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef2806b130 0x7fef2806b590 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.368+0000 7fef30400700 1 -- 172.21.15.40:0/3118854154 >> 172.21.15.40:0/3118854154 conn(0x7fef280792d0 msgr2=0x7fef280796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.368+0000 7fef30400700 1 -- 172.21.15.40:0/3118854154 shutdown_connections 2024-04-27T00:00:26.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.368+0000 7fef30400700 1 -- 172.21.15.40:0/3118854154 wait complete. 2024-04-27T00:00:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.370+0000 7fef30400700 1 Processor -- start 2024-04-27T00:00:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.370+0000 7fef30400700 1 -- start start 2024-04-27T00:00:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.370+0000 7fef30400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef2806b130 0x7fef2819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:26.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.370+0000 7fef30400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef2806bad0 0x7fef2819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.370+0000 7fef30400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef281094a0 0x7fef281a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.370+0000 7fef30400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef28113600 con 0x7fef2806bad0 2024-04-27T00:00:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.370+0000 7fef30400700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fef28113480 con 0x7fef2806b130 2024-04-27T00:00:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.370+0000 7fef30400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef28113780 con 0x7fef281094a0 2024-04-27T00:00:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.371+0000 7fef2d000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef2806bad0 0x7fef2819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.371+0000 7fef2da00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef2806b130 0x7fef2819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.371+0000 7fef2da00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef2806b130 0x7fef2819bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:33160/0 (socket says 172.21.15.40:33160) 2024-04-27T00:00:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.371+0000 7fef2da00700 1 -- 172.21.15.40:0/1242121611 learned_addr learned my addr 172.21.15.40:0/1242121611 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.371+0000 7fef2d000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef2806bad0 0x7fef2819c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48658/0 (socket says 172.21.15.40:48658) 2024-04-27T00:00:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.371+0000 7fef2e400700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef281094a0 0x7fef281a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.371+0000 7fef2da00700 1 -- 172.21.15.40:0/1242121611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef281094a0 msgr2=0x7fef281a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.371+0000 7fef2da00700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef281094a0 0x7fef281a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.371+0000 7fef2da00700 1 -- 172.21.15.40:0/1242121611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef2806bad0 msgr2=0x7fef2819c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.371+0000 7fef2da00700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef2806bad0 0x7fef2819c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.371+0000 7fef2da00700 1 -- 172.21.15.40:0/1242121611 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef281a0da0 con 0x7fef2806b130 2024-04-27T00:00:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.371+0000 7fef2da00700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef2806b130 0x7fef2819bd90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00e9a0 tx=0x7fef1c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.372+0000 7fef26000700 1 -- 172.21.15.40:0/1242121611 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c00c9c0 con 0x7fef2806b130 2024-04-27T00:00:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.372+0000 7fef26000700 1 -- 172.21.15.40:0/1242121611 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef1c00cb60 con 0x7fef2806b130 2024-04-27T00:00:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.372+0000 7fef30400700 1 -- 172.21.15.40:0/1242121611 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef281a1090 con 0x7fef2806b130 2024-04-27T00:00:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.372+0000 7fef26000700 1 -- 172.21.15.40:0/1242121611 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c018680 con 0x7fef2806b130 2024-04-27T00:00:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.372+0000 7fef30400700 1 -- 172.21.15.40:0/1242121611 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef281a8910 con 0x7fef2806b130 2024-04-27T00:00:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.374+0000 7fef30400700 1 -- 172.21.15.40:0/1242121611 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef2806a110 con 0x7fef2806b130 2024-04-27T00:00:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.375+0000 7fef26000700 1 -- 172.21.15.40:0/1242121611 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fef1c010430 con 0x7fef2806b130 2024-04-27T00:00:26.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.376+0000 7fef26000700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fef100a3d50 0x7fef100a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:26.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.377+0000 7fef26000700 1 -- 172.21.15.40:0/1242121611 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fef1c0d09f0 con 0x7fef2806b130 2024-04-27T00:00:26.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.377+0000 7fef2d000700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fef100a3d50 0x7fef100a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.378+0000 7fef2d000700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fef100a3d50 0x7fef100a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fef28110790 tx=0x7fef1400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:26.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.381+0000 7fef26000700 1 -- 172.21.15.40:0/1242121611 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fef1c09cfd0 con 0x7fef2806b130 2024-04-27T00:00:26.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:26 smithi040 ceph-mon[28095]: pgmap v370: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:26.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:26 smithi080 ceph-mon[32362]: pgmap v370: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:26.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:26 smithi103 ceph-mon[32952]: pgmap v370: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.642+0000 7fef30400700 1 -- 172.21.15.40:0/1242121611 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fef281a8b40 con 0x7fef2806b130 2024-04-27T00:00:26.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.643+0000 7fef26000700 1 -- 172.21.15.40:0/1242121611 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fef1c09c8d0 con 0x7fef2806b130 2024-04-27T00:00:26.645 INFO:teuthology.orchestra.run.smithi040.stdout:292057776172 2024-04-27T00:00:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.646+0000 7fef30400700 1 -- 172.21.15.40:0/1242121611 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fef100a3d50 msgr2=0x7fef100a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.646+0000 7fef30400700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fef100a3d50 0x7fef100a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fef28110790 tx=0x7fef1400b260 comp rx=0 tx=0).stop 2024-04-27T00:00:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.646+0000 7fef30400700 1 -- 172.21.15.40:0/1242121611 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef2806b130 msgr2=0x7fef2819bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.646+0000 7fef30400700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef2806b130 0x7fef2819bd90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00e9a0 tx=0x7fef1c00ecb0 comp rx=0 tx=0).stop 2024-04-27T00:00:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.646+0000 7fef30400700 1 -- 172.21.15.40:0/1242121611 shutdown_connections 2024-04-27T00:00:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.646+0000 7fef30400700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fef100a3d50 0x7fef100a6210 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.646+0000 7fef30400700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef281094a0 0x7fef281a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.646+0000 7fef30400700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fef2806b130 0x7fef2819bd90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:26.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.646+0000 7fef30400700 1 --2- 172.21.15.40:0/1242121611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef2806bad0 0x7fef2819c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:26.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.646+0000 7fef30400700 1 -- 172.21.15.40:0/1242121611 >> 172.21.15.40:0/1242121611 conn(0x7fef280792d0 msgr2=0x7fef28100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:26.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.646+0000 7fef30400700 1 -- 172.21.15.40:0/1242121611 shutdown_connections 2024-04-27T00:00:26.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:26.646+0000 7fef30400700 1 -- 172.21.15.40:0/1242121611 wait complete. 2024-04-27T00:00:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.107+0000 7f529ce00700 1 -- 172.21.15.40:0/2430602139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5298103470 msgr2=0x7f52981038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.107+0000 7f529ce00700 1 --2- 172.21.15.40:0/2430602139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5298103470 0x7f52981038f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5280009990 tx=0x7f5280009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.107+0000 7f529ce00700 1 -- 172.21.15.40:0/2430602139 shutdown_connections 2024-04-27T00:00:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.107+0000 7f529ce00700 1 --2- 172.21.15.40:0/2430602139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5298103e30 0x7f5298110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.107+0000 7f529ce00700 1 --2- 172.21.15.40:0/2430602139 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52981094c0 0x7f52981098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.107+0000 7f529ce00700 1 --2- 172.21.15.40:0/2430602139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5298103470 0x7f52981038f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.107+0000 7f529ce00700 1 -- 172.21.15.40:0/2430602139 >> 172.21.15.40:0/2430602139 conn(0x7f529807a580 msgr2=0x7f529807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.107+0000 7f529ce00700 1 -- 172.21.15.40:0/2430602139 shutdown_connections 2024-04-27T00:00:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.107+0000 7f529ce00700 1 -- 172.21.15.40:0/2430602139 wait complete. 2024-04-27T00:00:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.108+0000 7f529ce00700 1 Processor -- start 2024-04-27T00:00:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.109+0000 7f529ce00700 1 -- start start 2024-04-27T00:00:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.109+0000 7f529ce00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5298103470 0x7f529819bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.109+0000 7f529ce00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5298103e30 0x7f529819c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.109+0000 7f529ce00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52981094c0 0x7f52981a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.109+0000 7f529ce00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5298113610 con 0x7f5298103e30 2024-04-27T00:00:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.109+0000 7f529ce00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f5298113490 con 0x7f52981094c0 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.109+0000 7f529ce00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5298113790 con 0x7f5298103470 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.110+0000 7f528fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5298103e30 0x7f529819c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.110+0000 7f528fe00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5298103e30 0x7f529819c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48664/0 (socket says 172.21.15.40:48664) 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.110+0000 7f528fe00700 1 -- 172.21.15.40:0/1789712887 learned_addr learned my addr 172.21.15.40:0/1789712887 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.110+0000 7f5295e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5298103470 0x7f529819bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.110+0000 7f5296800700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52981094c0 0x7f52981a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.110+0000 7f528fe00700 1 -- 172.21.15.40:0/1789712887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5298103470 msgr2=0x7f529819bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.110+0000 7f528fe00700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5298103470 0x7f529819bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.110+0000 7f528fe00700 1 -- 172.21.15.40:0/1789712887 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52981094c0 msgr2=0x7f52981a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.110+0000 7f528fe00700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52981094c0 0x7f52981a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.110+0000 7f528fe00700 1 -- 172.21.15.40:0/1789712887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52981a0df0 con 0x7f5298103e30 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.111+0000 7f528fe00700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5298103e30 0x7f529819c320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5280000c00 tx=0x7f52800057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.111+0000 7f528ea00700 1 -- 172.21.15.40:0/1789712887 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5280027070 con 0x7f5298103e30 2024-04-27T00:00:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.111+0000 7f528ea00700 1 -- 172.21.15.40:0/1789712887 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5280022470 con 0x7f5298103e30 2024-04-27T00:00:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.111+0000 7f529ce00700 1 -- 172.21.15.40:0/1789712887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52981a1080 con 0x7f5298103e30 2024-04-27T00:00:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.111+0000 7f528ea00700 1 -- 172.21.15.40:0/1789712887 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5280017650 con 0x7f5298103e30 2024-04-27T00:00:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.111+0000 7f529ce00700 1 -- 172.21.15.40:0/1789712887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52981a8960 con 0x7f5298103e30 2024-04-27T00:00:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.112+0000 7f529ce00700 1 -- 172.21.15.40:0/1789712887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f529806a110 con 0x7f5298103e30 2024-04-27T00:00:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.114+0000 7f528ea00700 1 -- 172.21.15.40:0/1789712887 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f528002a070 con 0x7f5298103e30 2024-04-27T00:00:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.117+0000 7f528ea00700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f527c0a3d50 0x7f527c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.117+0000 7f528ea00700 1 -- 172.21.15.40:0/1789712887 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f52800d1470 con 0x7f5298103e30 2024-04-27T00:00:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.117+0000 7f5295e00700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f527c0a3d50 0x7f527c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.118+0000 7f5295e00700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f527c0a3d50 0x7f527c0a6210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5288009680 tx=0x7f5288009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.121+0000 7f528ea00700 1 -- 172.21.15.40:0/1789712887 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5280013070 con 0x7f5298103e30 2024-04-27T00:00:27.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.387+0000 7f529ce00700 1 -- 172.21.15.40:0/1789712887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f52981a8b30 con 0x7f5298103e30 2024-04-27T00:00:27.389 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:27 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1242121611' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-27T00:00:27.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.388+0000 7f528ea00700 1 -- 172.21.15.40:0/1789712887 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f528009dae0 con 0x7f5298103e30 2024-04-27T00:00:27.390 INFO:teuthology.orchestra.run.smithi040.stdout:313532612646 2024-04-27T00:00:27.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.391+0000 7f529ce00700 1 -- 172.21.15.40:0/1789712887 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f527c0a3d50 msgr2=0x7f527c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:27.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.391+0000 7f529ce00700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f527c0a3d50 0x7f527c0a6210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5288009680 tx=0x7f5288009260 comp rx=0 tx=0).stop 2024-04-27T00:00:27.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.391+0000 7f529ce00700 1 -- 172.21.15.40:0/1789712887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5298103e30 msgr2=0x7f529819c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:27.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.391+0000 7f529ce00700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5298103e30 0x7f529819c320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5280000c00 tx=0x7f52800057c0 comp rx=0 tx=0).stop 2024-04-27T00:00:27.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.391+0000 7f529ce00700 1 -- 172.21.15.40:0/1789712887 shutdown_connections 2024-04-27T00:00:27.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.391+0000 7f529ce00700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f527c0a3d50 0x7f527c0a6210 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:27.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.391+0000 7f529ce00700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f52981094c0 0x7f52981a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:27.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.391+0000 7f529ce00700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5298103470 0x7f529819bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:27.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.391+0000 7f529ce00700 1 --2- 172.21.15.40:0/1789712887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5298103e30 0x7f529819c320 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:27.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.391+0000 7f529ce00700 1 -- 172.21.15.40:0/1789712887 >> 172.21.15.40:0/1789712887 conn(0x7f529807a580 msgr2=0x7f529810e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:27.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.392+0000 7f529ce00700 1 -- 172.21.15.40:0/1789712887 shutdown_connections 2024-04-27T00:00:27.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:27.392+0000 7f529ce00700 1 -- 172.21.15.40:0/1789712887 wait complete. 2024-04-27T00:00:27.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:27 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1242121611' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-27T00:00:27.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:27 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1242121611' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-27T00:00:28.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:28 smithi040 ceph-mon[28095]: pgmap v371: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:28.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:28 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1789712887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-27T00:00:28.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:28 smithi080 ceph-mon[32362]: pgmap v371: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:28.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:28 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1789712887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-27T00:00:28.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:28 smithi103 ceph-mon[32952]: pgmap v371: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:28.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:28 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1789712887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-27T00:00:29.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.354+0000 7fd5d8a00700 1 -- 172.21.15.40:0/2481001474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5d41094f0 msgr2=0x7fd5d41098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:29.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.354+0000 7fd5d8a00700 1 --2- 172.21.15.40:0/2481001474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5d41094f0 0x7fd5d41098d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009990 tx=0x7fd5c4009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:29.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.354+0000 7fd5d8a00700 1 -- 172.21.15.40:0/2481001474 shutdown_connections 2024-04-27T00:00:29.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.354+0000 7fd5d8a00700 1 --2- 172.21.15.40:0/2481001474 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5d4103e30 0x7fd5d41102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:29.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.354+0000 7fd5d8a00700 1 --2- 172.21.15.40:0/2481001474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5d41094f0 0x7fd5d41098d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:29.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.354+0000 7fd5d8a00700 1 --2- 172.21.15.40:0/2481001474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d4103470 0x7fd5d41038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:29.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.354+0000 7fd5d8a00700 1 -- 172.21.15.40:0/2481001474 >> 172.21.15.40:0/2481001474 conn(0x7fd5d407a5d0 msgr2=0x7fd5d407a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:29.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.354+0000 7fd5d8a00700 1 -- 172.21.15.40:0/2481001474 shutdown_connections 2024-04-27T00:00:29.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.355+0000 7fd5d8a00700 1 -- 172.21.15.40:0/2481001474 wait complete. 2024-04-27T00:00:29.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.356+0000 7fd5d8a00700 1 Processor -- start 2024-04-27T00:00:29.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.356+0000 7fd5d8a00700 1 -- start start 2024-04-27T00:00:29.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.356+0000 7fd5d8a00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5d4103470 0x7fd5d4075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:29.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.357+0000 7fd5d8a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5d4103e30 0x7fd5d4075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:29.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.357+0000 7fd5d8a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d41094f0 0x7fd5d40761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:29.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.357+0000 7fd5d8a00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4113670 con 0x7fd5d41094f0 2024-04-27T00:00:29.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.357+0000 7fd5d8a00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41134f0 con 0x7fd5d4103470 2024-04-27T00:00:29.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.357+0000 7fd5d8a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41137f0 con 0x7fd5d4103e30 2024-04-27T00:00:29.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.357+0000 7fd5d1400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5d4103e30 0x7fd5d4075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:29.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.357+0000 7fd5d1e00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5d4103470 0x7fd5d4075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.357+0000 7fd5d1400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5d4103e30 0x7fd5d4075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:35694/0 (socket says 172.21.15.40:35694) 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.357+0000 7fd5d1400700 1 -- 172.21.15.40:0/3900826335 learned_addr learned my addr 172.21.15.40:0/3900826335 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.357+0000 7fd5d2800700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d41094f0 0x7fd5d40761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.357+0000 7fd5d1400700 1 -- 172.21.15.40:0/3900826335 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5d4103470 msgr2=0x7fd5d4075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.358+0000 7fd5d1400700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5d4103470 0x7fd5d4075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.358+0000 7fd5d1400700 1 -- 172.21.15.40:0/3900826335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d41094f0 msgr2=0x7fd5d40761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.358+0000 7fd5d1400700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d41094f0 0x7fd5d40761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.358+0000 7fd5d1400700 1 -- 172.21.15.40:0/3900826335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d41a8030 con 0x7fd5d4103e30 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.358+0000 7fd5d1e00700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5d4103470 0x7fd5d4075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.358+0000 7fd5d2800700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d41094f0 0x7fd5d40761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.358+0000 7fd5d1400700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5d4103e30 0x7fd5d4075c60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000c9b0 tx=0x7fd5c000ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.358+0000 7fd5caa00700 1 -- 172.21.15.40:0/3900826335 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0007b00 con 0x7fd5d4103e30 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.358+0000 7fd5caa00700 1 -- 172.21.15.40:0/3900826335 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5c0007ca0 con 0x7fd5d4103e30 2024-04-27T00:00:29.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.358+0000 7fd5caa00700 1 -- 172.21.15.40:0/3900826335 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0018660 con 0x7fd5d4103e30 2024-04-27T00:00:29.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.358+0000 7fd5d8a00700 1 -- 172.21.15.40:0/3900826335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d41a8320 con 0x7fd5d4103e30 2024-04-27T00:00:29.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.359+0000 7fd5d8a00700 1 -- 172.21.15.40:0/3900826335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d41a8810 con 0x7fd5d4103e30 2024-04-27T00:00:29.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.360+0000 7fd5d8a00700 1 -- 172.21.15.40:0/3900826335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5d410b960 con 0x7fd5d4103e30 2024-04-27T00:00:29.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.363+0000 7fd5caa00700 1 -- 172.21.15.40:0/3900826335 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd5c000f450 con 0x7fd5d4103e30 2024-04-27T00:00:29.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.364+0000 7fd5caa00700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd5b80a3d50 0x7fd5b80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:29.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.364+0000 7fd5caa00700 1 -- 172.21.15.40:0/3900826335 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd5c00d0c00 con 0x7fd5d4103e30 2024-04-27T00:00:29.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.365+0000 7fd5d1e00700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd5b80a3d50 0x7fd5b80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:29.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.366+0000 7fd5d1e00700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd5b80a3d50 0x7fd5b80a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4006010 tx=0x7fd5c400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:29.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.366+0000 7fd5caa00700 1 -- 172.21.15.40:0/3900826335 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd5c0014030 con 0x7fd5d4103e30 2024-04-27T00:00:29.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.607+0000 7fd5d8a00700 1 -- 172.21.15.40:0/3900826335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fd5d410b340 con 0x7fd5d4103e30 2024-04-27T00:00:29.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.608+0000 7fd5caa00700 1 -- 172.21.15.40:0/3900826335 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fd5c009d260 con 0x7fd5d4103e30 2024-04-27T00:00:29.610 INFO:teuthology.orchestra.run.smithi040.stdout:347892351010 2024-04-27T00:00:29.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.612+0000 7fd5d8a00700 1 -- 172.21.15.40:0/3900826335 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd5b80a3d50 msgr2=0x7fd5b80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:29.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.612+0000 7fd5d8a00700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd5b80a3d50 0x7fd5b80a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4006010 tx=0x7fd5c400b260 comp rx=0 tx=0).stop 2024-04-27T00:00:29.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.612+0000 7fd5d8a00700 1 -- 172.21.15.40:0/3900826335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5d4103e30 msgr2=0x7fd5d4075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:29.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.612+0000 7fd5d8a00700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5d4103e30 0x7fd5d4075c60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000c9b0 tx=0x7fd5c000ccc0 comp rx=0 tx=0).stop 2024-04-27T00:00:29.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.612+0000 7fd5d8a00700 1 -- 172.21.15.40:0/3900826335 shutdown_connections 2024-04-27T00:00:29.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.612+0000 7fd5d8a00700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd5b80a3d50 0x7fd5b80a6210 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:29.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.612+0000 7fd5d8a00700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d41094f0 0x7fd5d40761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:29.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.612+0000 7fd5d8a00700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd5d4103470 0x7fd5d4075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:29.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.612+0000 7fd5d8a00700 1 --2- 172.21.15.40:0/3900826335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5d4103e30 0x7fd5d4075c60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:29.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.612+0000 7fd5d8a00700 1 -- 172.21.15.40:0/3900826335 >> 172.21.15.40:0/3900826335 conn(0x7fd5d407a5d0 msgr2=0x7fd5d41020c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:29.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.613+0000 7fd5d8a00700 1 -- 172.21.15.40:0/3900826335 shutdown_connections 2024-04-27T00:00:29.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:29.613+0000 7fd5d8a00700 1 -- 172.21.15.40:0/3900826335 wait complete. 2024-04-27T00:00:30.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:30 smithi080 ceph-mon[32362]: pgmap v372: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:30.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:30 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3900826335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-27T00:00:30.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:30 smithi040 ceph-mon[28095]: pgmap v372: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:30.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:30 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3900826335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-27T00:00:30.594 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299441 got 223338299446 for osd.7 2024-04-27T00:00:30.595 DEBUG:teuthology.parallel:result is None 2024-04-27T00:00:30.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:30 smithi103 ceph-mon[32952]: pgmap v372: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:30.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:30 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3900826335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-27T00:00:31.347 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612642 got 313532612646 for osd.10 2024-04-27T00:00:31.347 DEBUG:teuthology.parallel:result is None 2024-04-27T00:00:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.366+0000 7f9523e00700 1 -- 172.21.15.40:0/4147160167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9524102120 msgr2=0x7f9524102500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.366+0000 7f9523e00700 1 --2- 172.21.15.40:0/4147160167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9524102120 0x7f9524102500 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9514009990 tx=0x7f9514009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.366+0000 7f9523e00700 1 -- 172.21.15.40:0/4147160167 shutdown_connections 2024-04-27T00:00:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.366+0000 7f9523e00700 1 --2- 172.21.15.40:0/4147160167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f952410fcd0 0x7f95241120c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.366+0000 7f9523e00700 1 --2- 172.21.15.40:0/4147160167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9524102120 0x7f9524102500 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.366+0000 7f9523e00700 1 --2- 172.21.15.40:0/4147160167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9524102ad0 0x7f952410f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.366+0000 7f9523e00700 1 -- 172.21.15.40:0/4147160167 >> 172.21.15.40:0/4147160167 conn(0x7f95240fdd40 msgr2=0x7f9524100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.366+0000 7f9523e00700 1 -- 172.21.15.40:0/4147160167 shutdown_connections 2024-04-27T00:00:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.366+0000 7f9523e00700 1 -- 172.21.15.40:0/4147160167 wait complete. 2024-04-27T00:00:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.368+0000 7f9523e00700 1 Processor -- start 2024-04-27T00:00:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.368+0000 7f9523e00700 1 -- start start 2024-04-27T00:00:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.368+0000 7f9523e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9524102120 0x7f95241a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.368+0000 7f9523e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9524102ad0 0x7f95241a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.368+0000 7f9523e00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f952410fcd0 0x7f952419baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9523e00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f95241157f0 con 0x7f9524102ad0 2024-04-27T00:00:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9523e00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f9524115670 con 0x7f952410fcd0 2024-04-27T00:00:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9523e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9524115970 con 0x7f9524102120 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9522a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9524102120 0x7f95241a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9523400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f952410fcd0 0x7f952419baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9522a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9524102120 0x7f95241a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49678/0 (socket says 172.21.15.40:49678) 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9522a00700 1 -- 172.21.15.40:0/4061332998 learned_addr learned my addr 172.21.15.40:0/4061332998 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9522000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9524102ad0 0x7f95241a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9523400700 1 -- 172.21.15.40:0/4061332998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9524102120 msgr2=0x7f95241a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9523400700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9524102120 0x7f95241a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9523400700 1 -- 172.21.15.40:0/4061332998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9524102ad0 msgr2=0x7f95241a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9523400700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9524102ad0 0x7f95241a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9523400700 1 -- 172.21.15.40:0/4061332998 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952419c250 con 0x7f952410fcd0 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.369+0000 7f9522a00700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9524102120 0x7f95241a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.370+0000 7f9523400700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f952410fcd0 0x7f952419baf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f952406aa90 tx=0x7f951c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.370+0000 7f951b400700 1 -- 172.21.15.40:0/4061332998 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c0092e0 con 0x7f952410fcd0 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.370+0000 7f951b400700 1 -- 172.21.15.40:0/4061332998 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f951c00c940 con 0x7f952410fcd0 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.370+0000 7f951b400700 1 -- 172.21.15.40:0/4061332998 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c0044c0 con 0x7f952410fcd0 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.370+0000 7f9523e00700 1 -- 172.21.15.40:0/4061332998 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f952419c4e0 con 0x7f952410fcd0 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.370+0000 7f9523e00700 1 -- 172.21.15.40:0/4061332998 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95241a87b0 con 0x7f952410fcd0 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.371+0000 7f9523e00700 1 -- 172.21.15.40:0/4061332998 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f952406a110 con 0x7f952410fcd0 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.373+0000 7f951b400700 1 -- 172.21.15.40:0/4061332998 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f951c01b030 con 0x7f952410fcd0 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.374+0000 7f951b400700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f95080a3d50 0x7f95080a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.375+0000 7f951b400700 1 -- 172.21.15.40:0/4061332998 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f951c0cff70 con 0x7f952410fcd0 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.375+0000 7f9522a00700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f95080a3d50 0x7f95080a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.376+0000 7f9522a00700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f95080a3d50 0x7f95080a6210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9514006010 tx=0x7f951400b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.379+0000 7f951b400700 1 -- 172.21.15.40:0/4061332998 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f951c022020 con 0x7f952410fcd0 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.617+0000 7f9523e00700 1 -- 172.21.15.40:0/4061332998 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f952402e490 con 0x7f952410fcd0 2024-04-27T00:00:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.617+0000 7f951b400700 1 -- 172.21.15.40:0/4061332998 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f951c010040 con 0x7f952410fcd0 2024-04-27T00:00:31.727 INFO:teuthology.orchestra.run.smithi040.stdout:201863462975 2024-04-27T00:00:31.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.622+0000 7f9523e00700 1 -- 172.21.15.40:0/4061332998 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f95080a3d50 msgr2=0x7f95080a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:31.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.622+0000 7f9523e00700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f95080a3d50 0x7f95080a6210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9514006010 tx=0x7f951400b280 comp rx=0 tx=0).stop 2024-04-27T00:00:31.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.622+0000 7f9523e00700 1 -- 172.21.15.40:0/4061332998 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f952410fcd0 msgr2=0x7f952419baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:31.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.622+0000 7f9523e00700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f952410fcd0 0x7f952419baf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f952406aa90 tx=0x7f951c00c3a0 comp rx=0 tx=0).stop 2024-04-27T00:00:31.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.622+0000 7f9523e00700 1 -- 172.21.15.40:0/4061332998 shutdown_connections 2024-04-27T00:00:31.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.622+0000 7f9523e00700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f95080a3d50 0x7f95080a6210 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:31.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.622+0000 7f9523e00700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f952410fcd0 0x7f952419baf0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:31.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.622+0000 7f9523e00700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9524102120 0x7f95241a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:31.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.622+0000 7f9523e00700 1 --2- 172.21.15.40:0/4061332998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9524102ad0 0x7f95241a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:31.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.622+0000 7f9523e00700 1 -- 172.21.15.40:0/4061332998 >> 172.21.15.40:0/4061332998 conn(0x7f95240fdd40 msgr2=0x7f9524100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:31.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.623+0000 7f9523e00700 1 -- 172.21.15.40:0/4061332998 shutdown_connections 2024-04-27T00:00:31.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:31.623+0000 7f9523e00700 1 -- 172.21.15.40:0/4061332998 wait complete. 2024-04-27T00:00:32.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:32 smithi040 ceph-mon[28095]: pgmap v373: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:00:32.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:32 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/4061332998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-27T00:00:32.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:00:32.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:32 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:00:32.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:32 smithi103 ceph-mon[32952]: pgmap v373: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:00:32.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:32 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/4061332998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-27T00:00:32.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:00:32.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:32 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:00:32.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:32 smithi080 ceph-mon[32362]: pgmap v373: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:00:32.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:32 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/4061332998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-27T00:00:32.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:00:32.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:32 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:00:32.844 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776165 got 292057776172 for osd.9 2024-04-27T00:00:32.844 DEBUG:teuthology.parallel:result is None 2024-04-27T00:00:33.375 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351003 got 347892351010 for osd.11 2024-04-27T00:00:33.375 DEBUG:teuthology.parallel:result is None 2024-04-27T00:00:33.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.105+0000 7f6b6c600700 1 -- 172.21.15.40:0/3687888976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b64103e90 msgr2=0x7f6b64110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:33.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.105+0000 7f6b6c600700 1 --2- 172.21.15.40:0/3687888976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b64103e90 0x7f6b64110300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0099c0 tx=0x7f6b5c009cd0 comp rx=0 tx=0).stop 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.105+0000 7f6b6c600700 1 -- 172.21.15.40:0/3687888976 shutdown_connections 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.105+0000 7f6b6c600700 1 --2- 172.21.15.40:0/3687888976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b64103e90 0x7f6b64110300 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.105+0000 7f6b6c600700 1 --2- 172.21.15.40:0/3687888976 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b641083b0 0x7f6b64108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.105+0000 7f6b6c600700 1 --2- 172.21.15.40:0/3687888976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b641034d0 0x7f6b64103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.105+0000 7f6b6c600700 1 -- 172.21.15.40:0/3687888976 >> 172.21.15.40:0/3687888976 conn(0x7f6b6407a580 msgr2=0x7f6b6407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.105+0000 7f6b6c600700 1 -- 172.21.15.40:0/3687888976 shutdown_connections 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.106+0000 7f6b6c600700 1 -- 172.21.15.40:0/3687888976 wait complete. 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.107+0000 7f6b6c600700 1 Processor -- start 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.107+0000 7f6b6c600700 1 -- start start 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.107+0000 7f6b6c600700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b641034d0 0x7f6b641a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.107+0000 7f6b6c600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b64103e90 0x7f6b641a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.107+0000 7f6b6c600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b641083b0 0x7f6b6419baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:33.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.107+0000 7f6b6c600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b64113690 con 0x7f6b64103e90 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.107+0000 7f6b6c600700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f6b64113510 con 0x7f6b641034d0 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.107+0000 7f6b6c600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6b64113810 con 0x7f6b641083b0 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.108+0000 7f6b69c00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b641034d0 0x7f6b641a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.108+0000 7f6b69200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b64103e90 0x7f6b641a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.108+0000 7f6b69c00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b641034d0 0x7f6b641a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:38128/0 (socket says 172.21.15.40:38128) 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.108+0000 7f6b69c00700 1 -- 172.21.15.40:0/1063529880 learned_addr learned my addr 172.21.15.40:0/1063529880 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.108+0000 7f6b6a600700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b641083b0 0x7f6b6419baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.108+0000 7f6b69c00700 1 -- 172.21.15.40:0/1063529880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b641083b0 msgr2=0x7f6b6419baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.108+0000 7f6b69c00700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b641083b0 0x7f6b6419baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.108+0000 7f6b69c00700 1 -- 172.21.15.40:0/1063529880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b64103e90 msgr2=0x7f6b641a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.108+0000 7f6b69c00700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b64103e90 0x7f6b641a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.108+0000 7f6b69c00700 1 -- 172.21.15.40:0/1063529880 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6419c250 con 0x7f6b641034d0 2024-04-27T00:00:33.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.108+0000 7f6b69c00700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b641034d0 0x7f6b641a1930 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800ef10 tx=0x7f6b5800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.109+0000 7f6b62000700 1 -- 172.21.15.40:0/1063529880 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5800c9c0 con 0x7f6b641034d0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.109+0000 7f6b62000700 1 -- 172.21.15.40:0/1063529880 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b58019070 con 0x7f6b641034d0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.109+0000 7f6b6c600700 1 -- 172.21.15.40:0/1063529880 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6419c4e0 con 0x7f6b641034d0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.109+0000 7f6b62000700 1 -- 172.21.15.40:0/1063529880 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b58014480 con 0x7f6b641034d0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.109+0000 7f6b6c600700 1 -- 172.21.15.40:0/1063529880 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b641a8800 con 0x7f6b641034d0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.110+0000 7f6b6c600700 1 -- 172.21.15.40:0/1063529880 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b6406a110 con 0x7f6b641034d0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.114+0000 7f6b62000700 1 -- 172.21.15.40:0/1063529880 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6b58012050 con 0x7f6b641034d0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.115+0000 7f6b62000700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b4c0a3cd0 0x7f6b4c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.115+0000 7f6b62000700 1 -- 172.21.15.40:0/1063529880 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f6b580d0dd0 con 0x7f6b641034d0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.115+0000 7f6b69200700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b4c0a3cd0 0x7f6b4c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.116+0000 7f6b62000700 1 -- 172.21.15.40:0/1063529880 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6b58010040 con 0x7f6b641034d0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.116+0000 7f6b69200700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b4c0a3cd0 0x7f6b4c0a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6b641106b0 tx=0x7f6b50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.350+0000 7f6b6c600700 1 -- 172.21.15.40:0/1063529880 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f6b641a89d0 con 0x7f6b641034d0 2024-04-27T00:00:33.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.351+0000 7f6b62000700 1 -- 172.21.15.40:0/1063529880 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6b5809d3b0 con 0x7f6b641034d0 2024-04-27T00:00:33.590 INFO:teuthology.orchestra.run.smithi040.stdout:146028888140 2024-04-27T00:00:33.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.354+0000 7f6b6c600700 1 -- 172.21.15.40:0/1063529880 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b4c0a3cd0 msgr2=0x7f6b4c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:33.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.354+0000 7f6b6c600700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b4c0a3cd0 0x7f6b4c0a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6b641106b0 tx=0x7f6b50009260 comp rx=0 tx=0).stop 2024-04-27T00:00:33.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.354+0000 7f6b6c600700 1 -- 172.21.15.40:0/1063529880 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b641034d0 msgr2=0x7f6b641a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:33.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.354+0000 7f6b6c600700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b641034d0 0x7f6b641a1930 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800ef10 tx=0x7f6b5800c3a0 comp rx=0 tx=0).stop 2024-04-27T00:00:33.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.354+0000 7f6b6c600700 1 -- 172.21.15.40:0/1063529880 shutdown_connections 2024-04-27T00:00:33.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.354+0000 7f6b6c600700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b4c0a3cd0 0x7f6b4c0a6190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:33.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.354+0000 7f6b6c600700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b641083b0 0x7f6b6419baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:33.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.354+0000 7f6b6c600700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b641034d0 0x7f6b641a1930 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:33.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.354+0000 7f6b6c600700 1 --2- 172.21.15.40:0/1063529880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b64103e90 0x7f6b641a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:33.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.354+0000 7f6b6c600700 1 -- 172.21.15.40:0/1063529880 >> 172.21.15.40:0/1063529880 conn(0x7f6b6407a580 msgr2=0x7f6b64101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:33.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.354+0000 7f6b6c600700 1 -- 172.21.15.40:0/1063529880 shutdown_connections 2024-04-27T00:00:33.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:33.354+0000 7f6b6c600700 1 -- 172.21.15.40:0/1063529880 wait complete. 2024-04-27T00:00:33.827 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:33.829 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:34.586 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:34 smithi040 ceph-mon[28095]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:34.586 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:34 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1063529880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T00:00:34.586 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:00:34.586 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:00:34.586 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:00:34.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:34 smithi103 ceph-mon[32952]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:34.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:34 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1063529880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T00:00:34.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:00:34.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:00:34.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:00:34.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:34 smithi080 ceph-mon[32362]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:34.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:34 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1063529880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T00:00:34.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:00:34.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:00:34.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:00:35.165 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888134 got 146028888140 for osd.4 2024-04-27T00:00:35.165 DEBUG:teuthology.parallel:result is None 2024-04-27T00:00:35.517 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462971 got 201863462975 for osd.6 2024-04-27T00:00:35.517 DEBUG:teuthology.parallel:result is None 2024-04-27T00:00:36.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.065+0000 7f0b0b400700 1 -- 172.21.15.40:0/1026479228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b041094a0 msgr2=0x7f0b04109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:36.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.065+0000 7f0b0b400700 1 --2- 172.21.15.40:0/1026479228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b041094a0 0x7f0b04109880 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009990 tx=0x7f0af8009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:36.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.065+0000 7f0b0b400700 1 -- 172.21.15.40:0/1026479228 shutdown_connections 2024-04-27T00:00:36.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.065+0000 7f0b0b400700 1 --2- 172.21.15.40:0/1026479228 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b0406bad0 0x7f0b041102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:36.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.065+0000 7f0b0b400700 1 --2- 172.21.15.40:0/1026479228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b041094a0 0x7f0b04109880 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:36.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.065+0000 7f0b0b400700 1 --2- 172.21.15.40:0/1026479228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b0406b130 0x7f0b0406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:36.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.065+0000 7f0b0b400700 1 -- 172.21.15.40:0/1026479228 >> 172.21.15.40:0/1026479228 conn(0x7f0b040792d0 msgr2=0x7f0b040796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:36.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.065+0000 7f0b0b400700 1 -- 172.21.15.40:0/1026479228 shutdown_connections 2024-04-27T00:00:36.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.065+0000 7f0b0b400700 1 -- 172.21.15.40:0/1026479228 wait complete. 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.067+0000 7f0b0b400700 1 Processor -- start 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.067+0000 7f0b0b400700 1 -- start start 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.067+0000 7f0b0b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b0406b130 0x7f0b0419bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.067+0000 7f0b0b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b0406bad0 0x7f0b0419c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.067+0000 7f0b0b400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b041094a0 0x7f0b041a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.067+0000 7f0b0b400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b041135e0 con 0x7f0b0406b130 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.067+0000 7f0b0b400700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0b04113460 con 0x7f0b041094a0 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.067+0000 7f0b0b400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0b04113760 con 0x7f0b0406bad0 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.068+0000 7f0b08a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b0406b130 0x7f0b0419bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.068+0000 7f0b08a00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b0406b130 0x7f0b0419bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52708/0 (socket says 172.21.15.40:52708) 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.068+0000 7f0b08a00700 1 -- 172.21.15.40:0/4244672767 learned_addr learned my addr 172.21.15.40:0/4244672767 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.068+0000 7f0b09400700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b041094a0 0x7f0b041a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:36.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.068+0000 7f0b03e00700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b0406bad0 0x7f0b0419c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.068+0000 7f0b09400700 1 -- 172.21.15.40:0/4244672767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b0406bad0 msgr2=0x7f0b0419c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.068+0000 7f0b09400700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b0406bad0 0x7f0b0419c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.068+0000 7f0b09400700 1 -- 172.21.15.40:0/4244672767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b0406b130 msgr2=0x7f0b0419bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.068+0000 7f0b09400700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b0406b130 0x7f0b0419bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.068+0000 7f0b09400700 1 -- 172.21.15.40:0/4244672767 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b041a0d80 con 0x7f0b041094a0 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.068+0000 7f0b08a00700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b0406b130 0x7f0b0419bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.068+0000 7f0b03e00700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b0406bad0 0x7f0b0419c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.069+0000 7f0b09400700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b041094a0 0x7f0b041a0640 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00efc0 tx=0x7f0afc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.069+0000 7f0b01600700 1 -- 172.21.15.40:0/4244672767 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0afc0092e0 con 0x7f0b041094a0 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.069+0000 7f0b01600700 1 -- 172.21.15.40:0/4244672767 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0afc004bb0 con 0x7f0b041094a0 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.069+0000 7f0b0b400700 1 -- 172.21.15.40:0/4244672767 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b041a1010 con 0x7f0b041094a0 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.069+0000 7f0b01600700 1 -- 172.21.15.40:0/4244672767 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0afc004080 con 0x7f0b041094a0 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.069+0000 7f0b0b400700 1 -- 172.21.15.40:0/4244672767 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b041a88f0 con 0x7f0b041094a0 2024-04-27T00:00:36.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.070+0000 7f0b0b400700 1 -- 172.21.15.40:0/4244672767 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b0406a110 con 0x7f0b041094a0 2024-04-27T00:00:36.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.072+0000 7f0b01600700 1 -- 172.21.15.40:0/4244672767 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0afc01b030 con 0x7f0b041094a0 2024-04-27T00:00:36.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.074+0000 7f0b01600700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0aec0a3cd0 0x7f0aec0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:36.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.074+0000 7f0b01600700 1 -- 172.21.15.40:0/4244672767 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f0afc0d0320 con 0x7f0b041094a0 2024-04-27T00:00:36.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.074+0000 7f0b08a00700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0aec0a3cd0 0x7f0aec0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:36.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.075+0000 7f0b08a00700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0aec0a3cd0 0x7f0aec0a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0af8006010 tx=0x7f0af800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:36.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.078+0000 7f0b01600700 1 -- 172.21.15.40:0/4244672767 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0afc022020 con 0x7f0b041094a0 2024-04-27T00:00:36.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.310+0000 7f0b0b400700 1 -- 172.21.15.40:0/4244672767 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0b041a8ac0 con 0x7f0b041094a0 2024-04-27T00:00:36.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.314+0000 7f0b01600700 1 -- 172.21.15.40:0/4244672767 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0afc010040 con 0x7f0b041094a0 2024-04-27T00:00:36.315 INFO:teuthology.orchestra.run.smithi040.stdout:55834574940 2024-04-27T00:00:36.319 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:36 smithi040 ceph-mon[28095]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:36.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.317+0000 7f0b0b400700 1 -- 172.21.15.40:0/4244672767 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0aec0a3cd0 msgr2=0x7f0aec0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:36.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.317+0000 7f0b0b400700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0aec0a3cd0 0x7f0aec0a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0af8006010 tx=0x7f0af800b280 comp rx=0 tx=0).stop 2024-04-27T00:00:36.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.317+0000 7f0b0b400700 1 -- 172.21.15.40:0/4244672767 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b041094a0 msgr2=0x7f0b041a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:36.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.317+0000 7f0b0b400700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b041094a0 0x7f0b041a0640 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00efc0 tx=0x7f0afc00c3a0 comp rx=0 tx=0).stop 2024-04-27T00:00:36.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.317+0000 7f0b0b400700 1 -- 172.21.15.40:0/4244672767 shutdown_connections 2024-04-27T00:00:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.317+0000 7f0b0b400700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0aec0a3cd0 0x7f0aec0a6190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.317+0000 7f0b0b400700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0b041094a0 0x7f0b041a0640 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.317+0000 7f0b0b400700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b0406b130 0x7f0b0419bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.317+0000 7f0b0b400700 1 --2- 172.21.15.40:0/4244672767 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b0406bad0 0x7f0b0419c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.317+0000 7f0b0b400700 1 -- 172.21.15.40:0/4244672767 >> 172.21.15.40:0/4244672767 conn(0x7f0b040792d0 msgr2=0x7f0b04100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.317+0000 7f0b0b400700 1 -- 172.21.15.40:0/4244672767 shutdown_connections 2024-04-27T00:00:36.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:36.318+0000 7f0b0b400700 1 -- 172.21.15.40:0/4244672767 wait complete. 2024-04-27T00:00:36.459 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:36.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:36 smithi103 ceph-mon[32952]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:36.698 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:36 smithi080 ceph-mon[32362]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:37.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:37 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/4244672767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T00:00:37.365 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574934 got 55834574940 for osd.1 2024-04-27T00:00:37.365 DEBUG:teuthology.parallel:result is None 2024-04-27T00:00:37.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:37 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/4244672767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T00:00:37.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:37 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/4244672767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T00:00:38.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:38 smithi040 ceph-mon[28095]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:38.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:38 smithi103 ceph-mon[32952]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:38.700 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:38 smithi080 ceph-mon[32362]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:40.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:40 smithi040 ceph-mon[28095]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:40.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:40 smithi103 ceph-mon[32952]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:40.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:40 smithi080 ceph-mon[32362]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:41.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.749+0000 7f57b5400700 1 -- 172.21.15.40:0/2831227018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b0075f70 msgr2=0x7f57b01140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:41.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.749+0000 7f57b5400700 1 --2- 172.21.15.40:0/2831227018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b0075f70 0x7f57b01140b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5798009990 tx=0x7f5798009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:41.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.749+0000 7f57b5400700 1 -- 172.21.15.40:0/2831227018 shutdown_connections 2024-04-27T00:00:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.749+0000 7f57b5400700 1 --2- 172.21.15.40:0/2831227018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57b01145f0 0x7f57b01169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.749+0000 7f57b5400700 1 --2- 172.21.15.40:0/2831227018 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57b0075650 0x7f57b0075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.749+0000 7f57b5400700 1 --2- 172.21.15.40:0/2831227018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b0075f70 0x7f57b01140b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.749+0000 7f57b5400700 1 -- 172.21.15.40:0/2831227018 >> 172.21.15.40:0/2831227018 conn(0x7f57b007a580 msgr2=0x7f57b007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.749+0000 7f57b5400700 1 -- 172.21.15.40:0/2831227018 shutdown_connections 2024-04-27T00:00:41.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.749+0000 7f57b5400700 1 -- 172.21.15.40:0/2831227018 wait complete. 2024-04-27T00:00:41.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.750+0000 7f57b5400700 1 Processor -- start 2024-04-27T00:00:41.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.750+0000 7f57b5400700 1 -- start start 2024-04-27T00:00:41.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57b5400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57b0075650 0x7f57b01a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:41.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57b5400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57b0075f70 0x7f57b01a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:41.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57b5400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b01145f0 0x7f57b01a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:41.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57b5400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57b0119c30 con 0x7f57b0075f70 2024-04-27T00:00:41.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57b5400700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f57b0119ab0 con 0x7f57b0075650 2024-04-27T00:00:41.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57b5400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57b0119db0 con 0x7f57b01145f0 2024-04-27T00:00:41.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57ae800700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57b0075650 0x7f57b01a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:41.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57a5e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57b0075f70 0x7f57b01a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:41.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57ae800700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57b0075650 0x7f57b01a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:46088/0 (socket says 172.21.15.40:46088) 2024-04-27T00:00:41.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57ae800700 1 -- 172.21.15.40:0/1833813350 learned_addr learned my addr 172.21.15.40:0/1833813350 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:41.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57af200700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b01145f0 0x7f57b01a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:41.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57ae800700 1 -- 172.21.15.40:0/1833813350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b01145f0 msgr2=0x7f57b01a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:41.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57ae800700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b01145f0 0x7f57b01a4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:41.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57ae800700 1 -- 172.21.15.40:0/1833813350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57b0075f70 msgr2=0x7f57b01a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:41.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57ae800700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57b0075f70 0x7f57b01a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:41.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.751+0000 7f57ae800700 1 -- 172.21.15.40:0/1833813350 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b01a5190 con 0x7f57b0075650 2024-04-27T00:00:41.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.752+0000 7f57ae800700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57b0075650 0x7f57b01a0180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f57a000eb30 tx=0x7f57a000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:41.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.752+0000 7f57a7e00700 1 -- 172.21.15.40:0/1833813350 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a000cbd0 con 0x7f57b0075650 2024-04-27T00:00:41.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.752+0000 7f57b5400700 1 -- 172.21.15.40:0/1833813350 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57b01a5480 con 0x7f57b0075650 2024-04-27T00:00:41.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.752+0000 7f57a7e00700 1 -- 172.21.15.40:0/1833813350 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a00044d0 con 0x7f57b0075650 2024-04-27T00:00:41.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.752+0000 7f57a7e00700 1 -- 172.21.15.40:0/1833813350 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0003e30 con 0x7f57b0075650 2024-04-27T00:00:41.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.752+0000 7f57b5400700 1 -- 172.21.15.40:0/1833813350 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b01acd00 con 0x7f57b0075650 2024-04-27T00:00:41.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.753+0000 7f57b5400700 1 -- 172.21.15.40:0/1833813350 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57b006a110 con 0x7f57b0075650 2024-04-27T00:00:41.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.756+0000 7f57a7e00700 1 -- 172.21.15.40:0/1833813350 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f57a0004000 con 0x7f57b0075650 2024-04-27T00:00:41.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.757+0000 7f57a7e00700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f57940a3d50 0x7f57940a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:41.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.757+0000 7f57a7e00700 1 -- 172.21.15.40:0/1833813350 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f57a00d0f50 con 0x7f57b0075650 2024-04-27T00:00:41.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.757+0000 7f57a5e00700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f57940a3d50 0x7f57940a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:41.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.758+0000 7f57a5e00700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f57940a3d50 0x7f57940a6210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f57b0101af0 tx=0x7f579800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:41.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:40.760+0000 7f57a7e00700 1 -- 172.21.15.40:0/1833813350 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f57a00042f0 con 0x7f57b0075650 2024-04-27T00:00:41.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.011+0000 7f57b5400700 1 -- 172.21.15.40:0/1833813350 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f57b01a14e0 con 0x7f57b0075650 2024-04-27T00:00:41.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.012+0000 7f57a7e00700 1 -- 172.21.15.40:0/1833813350 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f57a009d530 con 0x7f57b0075650 2024-04-27T00:00:41.112 INFO:teuthology.orchestra.run.smithi040.stdout:77309411417 2024-04-27T00:00:41.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.015+0000 7f57b5400700 1 -- 172.21.15.40:0/1833813350 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f57940a3d50 msgr2=0x7f57940a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:41.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.015+0000 7f57b5400700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f57940a3d50 0x7f57940a6210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f57b0101af0 tx=0x7f579800b260 comp rx=0 tx=0).stop 2024-04-27T00:00:41.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.015+0000 7f57b5400700 1 -- 172.21.15.40:0/1833813350 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57b0075650 msgr2=0x7f57b01a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:41.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.015+0000 7f57b5400700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57b0075650 0x7f57b01a0180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f57a000eb30 tx=0x7f57a000ee40 comp rx=0 tx=0).stop 2024-04-27T00:00:41.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.015+0000 7f57b5400700 1 -- 172.21.15.40:0/1833813350 shutdown_connections 2024-04-27T00:00:41.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.015+0000 7f57b5400700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f57940a3d50 0x7f57940a6210 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:41.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.015+0000 7f57b5400700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57b01145f0 0x7f57b01a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:41.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.015+0000 7f57b5400700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57b0075650 0x7f57b01a0180 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:41.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.015+0000 7f57b5400700 1 --2- 172.21.15.40:0/1833813350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57b0075f70 0x7f57b01a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:41.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.015+0000 7f57b5400700 1 -- 172.21.15.40:0/1833813350 >> 172.21.15.40:0/1833813350 conn(0x7f57b007a580 msgr2=0x7f57b0110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:41.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.015+0000 7f57b5400700 1 -- 172.21.15.40:0/1833813350 shutdown_connections 2024-04-27T00:00:41.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:41.015+0000 7f57b5400700 1 -- 172.21.15.40:0/1833813350 wait complete. 2024-04-27T00:00:41.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:41 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1833813350' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T00:00:41.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:41 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1833813350' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T00:00:41.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:41 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1833813350' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T00:00:41.991 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411412 got 77309411417 for osd.2 2024-04-27T00:00:41.991 DEBUG:teuthology.parallel:result is None 2024-04-27T00:00:42.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.445+0000 7fd203c00700 1 -- 172.21.15.40:0/3354565694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1fc1094c0 msgr2=0x7fd1fc1098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:42.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.445+0000 7fd203c00700 1 --2- 172.21.15.40:0/3354565694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1fc1094c0 0x7fd1fc1098a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00b0a0 tx=0x7fd1ec00b3b0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.445+0000 7fd203c00700 1 -- 172.21.15.40:0/3354565694 shutdown_connections 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.445+0000 7fd203c00700 1 --2- 172.21.15.40:0/3354565694 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd1fc103e30 0x7fd1fc110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.445+0000 7fd203c00700 1 --2- 172.21.15.40:0/3354565694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1fc1094c0 0x7fd1fc1098a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.445+0000 7fd203c00700 1 --2- 172.21.15.40:0/3354565694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1fc103470 0x7fd1fc1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.445+0000 7fd203c00700 1 -- 172.21.15.40:0/3354565694 >> 172.21.15.40:0/3354565694 conn(0x7fd1fc07a580 msgr2=0x7fd1fc07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.445+0000 7fd203c00700 1 -- 172.21.15.40:0/3354565694 shutdown_connections 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.446+0000 7fd203c00700 1 -- 172.21.15.40:0/3354565694 wait complete. 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.447+0000 7fd203c00700 1 Processor -- start 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.447+0000 7fd203c00700 1 -- start start 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.448+0000 7fd203c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1fc103470 0x7fd1fc19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.448+0000 7fd203c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1fc103e30 0x7fd1fc19fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.448+0000 7fd203c00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd1fc1094c0 0x7fd1fc199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.448+0000 7fd203c00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc113610 con 0x7fd1fc103e30 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.448+0000 7fd203c00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc113490 con 0x7fd1fc1094c0 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.448+0000 7fd203c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc113790 con 0x7fd1fc103470 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.448+0000 7fd1f3e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1fc103e30 0x7fd1fc19fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.448+0000 7fd201200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1fc103470 0x7fd1fc19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.448+0000 7fd1f3e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1fc103e30 0x7fd1fc19fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32874/0 (socket says 172.21.15.40:32874) 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.448+0000 7fd1f3e00700 1 -- 172.21.15.40:0/989562046 learned_addr learned my addr 172.21.15.40:0/989562046 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.448+0000 7fd201200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1fc103470 0x7fd1fc19f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46524/0 (socket says 172.21.15.40:46524) 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.449+0000 7fd201200700 1 -- 172.21.15.40:0/989562046 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd1fc1094c0 msgr2=0x7fd1fc199990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.449+0000 7fd201200700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd1fc1094c0 0x7fd1fc199990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.449+0000 7fd201200700 1 -- 172.21.15.40:0/989562046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1fc103e30 msgr2=0x7fd1fc19fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.449+0000 7fd201200700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1fc103e30 0x7fd1fc19fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.449+0000 7fd201200700 1 -- 172.21.15.40:0/989562046 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1fc19a0f0 con 0x7fd1fc103470 2024-04-27T00:00:42.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.449+0000 7fd1f3e00700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1fc103e30 0x7fd1fc19fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.449+0000 7fd201200700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1fc103470 0x7fd1fc19f7d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00b070 tx=0x7fd1ec004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.450+0000 7fd1fa000700 1 -- 172.21.15.40:0/989562046 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec021070 con 0x7fd1fc103470 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.450+0000 7fd203c00700 1 -- 172.21.15.40:0/989562046 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1fc19a380 con 0x7fd1fc103470 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.450+0000 7fd203c00700 1 -- 172.21.15.40:0/989562046 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1fc1a66a0 con 0x7fd1fc103470 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.450+0000 7fd1fa000700 1 -- 172.21.15.40:0/989562046 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1ec007e30 con 0x7fd1fc103470 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.450+0000 7fd1fa000700 1 -- 172.21.15.40:0/989562046 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec004710 con 0x7fd1fc103470 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.451+0000 7fd203c00700 1 -- 172.21.15.40:0/989562046 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1fc06a110 con 0x7fd1fc103470 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.453+0000 7fd1fa000700 1 -- 172.21.15.40:0/989562046 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd1ec007980 con 0x7fd1fc103470 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.455+0000 7fd1fa000700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd1e00a3d50 0x7fd1e00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.455+0000 7fd1fa000700 1 -- 172.21.15.40:0/989562046 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd1ec00f020 con 0x7fd1fc103470 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.455+0000 7fd1f3e00700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd1e00a3d50 0x7fd1e00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.456+0000 7fd1f3e00700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd1e00a3d50 0x7fd1e00a6210 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc07acc0 tx=0x7fd1e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:42.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.459+0000 7fd1fa000700 1 -- 172.21.15.40:0/989562046 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd1ec02a020 con 0x7fd1fc103470 2024-04-27T00:00:42.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:42 smithi103 ceph-mon[32952]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:00:42.610 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:42 smithi040 ceph-mon[28095]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:00:42.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.697+0000 7fd203c00700 1 -- 172.21.15.40:0/989562046 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd1fc02e490 con 0x7fd1fc103470 2024-04-27T00:00:42.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.698+0000 7fd1fa000700 1 -- 172.21.15.40:0/989562046 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd1ec09e1b0 con 0x7fd1fc103470 2024-04-27T00:00:42.699 INFO:teuthology.orchestra.run.smithi040.stdout:167503724615 2024-04-27T00:00:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.701+0000 7fd203c00700 1 -- 172.21.15.40:0/989562046 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd1e00a3d50 msgr2=0x7fd1e00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.701+0000 7fd203c00700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd1e00a3d50 0x7fd1e00a6210 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc07acc0 tx=0x7fd1e4009260 comp rx=0 tx=0).stop 2024-04-27T00:00:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.701+0000 7fd203c00700 1 -- 172.21.15.40:0/989562046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1fc103470 msgr2=0x7fd1fc19f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.701+0000 7fd203c00700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1fc103470 0x7fd1fc19f7d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00b070 tx=0x7fd1ec004d00 comp rx=0 tx=0).stop 2024-04-27T00:00:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.701+0000 7fd203c00700 1 -- 172.21.15.40:0/989562046 shutdown_connections 2024-04-27T00:00:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.701+0000 7fd203c00700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd1e00a3d50 0x7fd1e00a6210 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.701+0000 7fd203c00700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd1fc1094c0 0x7fd1fc199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.701+0000 7fd203c00700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1fc103470 0x7fd1fc19f7d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.701+0000 7fd203c00700 1 --2- 172.21.15.40:0/989562046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1fc103e30 0x7fd1fc19fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.701+0000 7fd203c00700 1 -- 172.21.15.40:0/989562046 >> 172.21.15.40:0/989562046 conn(0x7fd1fc07a580 msgr2=0x7fd1fc102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.701+0000 7fd203c00700 1 -- 172.21.15.40:0/989562046 shutdown_connections 2024-04-27T00:00:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.701+0000 7fd203c00700 1 -- 172.21.15.40:0/989562046 wait complete. 2024-04-27T00:00:42.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.776+0000 7fc431e00700 1 -- 172.21.15.40:0/2119889979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc42c075f70 msgr2=0x7fc42c10f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:42.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.776+0000 7fc431e00700 1 --2- 172.21.15.40:0/2119889979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc42c075f70 0x7fc42c10f8e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc418009990 tx=0x7fc418009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.776+0000 7fc431e00700 1 -- 172.21.15.40:0/2119889979 shutdown_connections 2024-04-27T00:00:42.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.776+0000 7fc431e00700 1 --2- 172.21.15.40:0/2119889979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc42c10fe20 0x7fc42c112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.776+0000 7fc431e00700 1 --2- 172.21.15.40:0/2119889979 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc42c075650 0x7fc42c075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.776+0000 7fc431e00700 1 --2- 172.21.15.40:0/2119889979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc42c075f70 0x7fc42c10f8e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.776+0000 7fc431e00700 1 -- 172.21.15.40:0/2119889979 >> 172.21.15.40:0/2119889979 conn(0x7fc42c07a580 msgr2=0x7fc42c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:42.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.776+0000 7fc431e00700 1 -- 172.21.15.40:0/2119889979 shutdown_connections 2024-04-27T00:00:42.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.776+0000 7fc431e00700 1 -- 172.21.15.40:0/2119889979 wait complete. 2024-04-27T00:00:42.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.777+0000 7fc431e00700 1 Processor -- start 2024-04-27T00:00:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.777+0000 7fc431e00700 1 -- start start 2024-04-27T00:00:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.778+0000 7fc431e00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc42c075650 0x7fc42c1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.778+0000 7fc431e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc42c075f70 0x7fc42c1a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.778+0000 7fc431e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc42c10fe20 0x7fc42c19bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.778+0000 7fc431e00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c1157e0 con 0x7fc42c10fe20 2024-04-27T00:00:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.778+0000 7fc431e00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c115660 con 0x7fc42c075650 2024-04-27T00:00:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.778+0000 7fc431e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c115960 con 0x7fc42c075f70 2024-04-27T00:00:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.778+0000 7fc42aa00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc42c075650 0x7fc42c1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.778+0000 7fc42b400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc42c10fe20 0x7fc42c19bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.778+0000 7fc42aa00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc42c075650 0x7fc42c1a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:46124/0 (socket says 172.21.15.40:46124) 2024-04-27T00:00:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.778+0000 7fc42aa00700 1 -- 172.21.15.40:0/95056221 learned_addr learned my addr 172.21.15.40:0/95056221 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.778+0000 7fc42a000700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc42c075f70 0x7fc42c1a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.779+0000 7fc42aa00700 1 -- 172.21.15.40:0/95056221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc42c075f70 msgr2=0x7fc42c1a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.779+0000 7fc42aa00700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc42c075f70 0x7fc42c1a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.779+0000 7fc42aa00700 1 -- 172.21.15.40:0/95056221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc42c10fe20 msgr2=0x7fc42c19bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.779+0000 7fc42aa00700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc42c10fe20 0x7fc42c19bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.779+0000 7fc42aa00700 1 -- 172.21.15.40:0/95056221 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc42c19c240 con 0x7fc42c075650 2024-04-27T00:00:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.779+0000 7fc42b400700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc42c10fe20 0x7fc42c19bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.779+0000 7fc42aa00700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc42c075650 0x7fc42c1a1920 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00e9a0 tx=0x7fc41c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:42.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.779+0000 7fc430a00700 1 -- 172.21.15.40:0/95056221 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c00ca10 con 0x7fc42c075650 2024-04-27T00:00:42.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.779+0000 7fc431e00700 1 -- 172.21.15.40:0/95056221 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc42c19c530 con 0x7fc42c075650 2024-04-27T00:00:42.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.779+0000 7fc431e00700 1 -- 172.21.15.40:0/95056221 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc42c1a87f0 con 0x7fc42c075650 2024-04-27T00:00:42.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.779+0000 7fc430a00700 1 -- 172.21.15.40:0/95056221 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc41c00cbb0 con 0x7fc42c075650 2024-04-27T00:00:42.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.779+0000 7fc430a00700 1 -- 172.21.15.40:0/95056221 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c018420 con 0x7fc42c075650 2024-04-27T00:00:42.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.780+0000 7fc431e00700 1 -- 172.21.15.40:0/95056221 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc40c005330 con 0x7fc42c075650 2024-04-27T00:00:42.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.783+0000 7fc430a00700 1 -- 172.21.15.40:0/95056221 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc41c010430 con 0x7fc42c075650 2024-04-27T00:00:42.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.784+0000 7fc430a00700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4100a3cd0 0x7fc4100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:42.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.784+0000 7fc430a00700 1 -- 172.21.15.40:0/95056221 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc41c0d09b0 con 0x7fc42c075650 2024-04-27T00:00:42.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.784+0000 7fc42a000700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4100a3cd0 0x7fc4100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:42.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.785+0000 7fc42a000700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4100a3cd0 0x7fc4100a6190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc418006010 tx=0x7fc418005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:42.787+0000 7fc430a00700 1 -- 172.21.15.40:0/95056221 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc41c09cf90 con 0x7fc42c075650 2024-04-27T00:00:42.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:42 smithi080 ceph-mon[32362]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:00:43.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.023+0000 7fc431e00700 1 -- 172.21.15.40:0/95056221 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fc40c0058f0 con 0x7fc42c075650 2024-04-27T00:00:43.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.024+0000 7fc430a00700 1 -- 172.21.15.40:0/95056221 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc41c085e20 con 0x7fc42c075650 2024-04-27T00:00:43.026 INFO:teuthology.orchestra.run.smithi040.stdout:34359738466 2024-04-27T00:00:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.027+0000 7fc431e00700 1 -- 172.21.15.40:0/95056221 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4100a3cd0 msgr2=0x7fc4100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.027+0000 7fc431e00700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4100a3cd0 0x7fc4100a6190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc418006010 tx=0x7fc418005960 comp rx=0 tx=0).stop 2024-04-27T00:00:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.027+0000 7fc431e00700 1 -- 172.21.15.40:0/95056221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc42c075650 msgr2=0x7fc42c1a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.027+0000 7fc431e00700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc42c075650 0x7fc42c1a1920 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00e9a0 tx=0x7fc41c00ecb0 comp rx=0 tx=0).stop 2024-04-27T00:00:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.027+0000 7fc431e00700 1 -- 172.21.15.40:0/95056221 shutdown_connections 2024-04-27T00:00:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.027+0000 7fc431e00700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4100a3cd0 0x7fc4100a6190 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.027+0000 7fc431e00700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc42c10fe20 0x7fc42c19bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.027+0000 7fc431e00700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc42c075650 0x7fc42c1a1920 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:43.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.027+0000 7fc431e00700 1 --2- 172.21.15.40:0/95056221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc42c075f70 0x7fc42c1a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:43.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.027+0000 7fc431e00700 1 -- 172.21.15.40:0/95056221 >> 172.21.15.40:0/95056221 conn(0x7fc42c07a580 msgr2=0x7fc42c1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:43.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.027+0000 7fc431e00700 1 -- 172.21.15.40:0/95056221 shutdown_connections 2024-04-27T00:00:43.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:43.028+0000 7fc431e00700 1 -- 172.21.15.40:0/95056221 wait complete. 2024-04-27T00:00:43.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:43 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/989562046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T00:00:43.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:43 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/95056221' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T00:00:43.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:43 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/989562046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T00:00:43.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:43 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/95056221' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T00:00:43.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:43 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/989562046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T00:00:43.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:43 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/95056221' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T00:00:43.989 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738460 got 34359738466 for osd.0 2024-04-27T00:00:43.990 DEBUG:teuthology.parallel:result is None 2024-04-27T00:00:44.405 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724609 got 167503724615 for osd.5 2024-04-27T00:00:44.405 DEBUG:teuthology.parallel:result is None 2024-04-27T00:00:44.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:44 smithi040 ceph-mon[28095]: pgmap v379: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:44.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:44 smithi080 ceph-mon[32362]: pgmap v379: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:44.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:44 smithi103 ceph-mon[32952]: pgmap v379: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:45.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-27T00:00:45.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-04-27T00:00:45.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-04-27T00:00:45.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9, 5, 8]}]: dispatch 2024-04-27T00:00:45.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-04-27T00:00:45.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-04-27T00:00:45.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-04-27T00:00:45.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-27T00:00:45.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-04-27T00:00:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.637+0000 7f27f5800700 1 -- 172.21.15.40:0/3464547871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27f006b130 msgr2=0x7f27f006b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.637+0000 7f27f5800700 1 --2- 172.21.15.40:0/3464547871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27f006b130 0x7f27f006b590 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009990 tx=0x7f27d8009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.637+0000 7f27f5800700 1 -- 172.21.15.40:0/3464547871 shutdown_connections 2024-04-27T00:00:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.637+0000 7f27f5800700 1 --2- 172.21.15.40:0/3464547871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27f006bad0 0x7f27f01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.637+0000 7f27f5800700 1 --2- 172.21.15.40:0/3464547871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f01094a0 0x7f27f0109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.637+0000 7f27f5800700 1 --2- 172.21.15.40:0/3464547871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27f006b130 0x7f27f006b590 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.637+0000 7f27f5800700 1 -- 172.21.15.40:0/3464547871 >> 172.21.15.40:0/3464547871 conn(0x7f27f00792d0 msgr2=0x7f27f00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.637+0000 7f27f5800700 1 -- 172.21.15.40:0/3464547871 shutdown_connections 2024-04-27T00:00:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.637+0000 7f27f5800700 1 -- 172.21.15.40:0/3464547871 wait complete. 2024-04-27T00:00:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.638+0000 7f27f5800700 1 Processor -- start 2024-04-27T00:00:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.638+0000 7f27f5800700 1 -- start start 2024-04-27T00:00:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.639+0000 7f27f5800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27f006b130 0x7f27f019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.639+0000 7f27f5800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f006bad0 0x7f27f019fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.639+0000 7f27f5800700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27f01094a0 0x7f27f0199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.639+0000 7f27f5800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27f01135b0 con 0x7f27f006b130 2024-04-27T00:00:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.639+0000 7f27f5800700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0113430 con 0x7f27f01094a0 2024-04-27T00:00:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.639+0000 7f27f5800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0113730 con 0x7f27f006bad0 2024-04-27T00:00:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.639+0000 7f27e6000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f006bad0 0x7f27f019fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.639+0000 7f27eea00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27f006b130 0x7f27f019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.639+0000 7f27e6000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f006bad0 0x7f27f019fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46572/0 (socket says 172.21.15.40:46572) 2024-04-27T00:00:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.639+0000 7f27e6000700 1 -- 172.21.15.40:0/33098895 learned_addr learned my addr 172.21.15.40:0/33098895 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.639+0000 7f27ef400700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27f01094a0 0x7f27f0199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.640+0000 7f27e6000700 1 -- 172.21.15.40:0/33098895 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27f01094a0 msgr2=0x7f27f0199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.640+0000 7f27e6000700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27f01094a0 0x7f27f0199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.640+0000 7f27e6000700 1 -- 172.21.15.40:0/33098895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27f006b130 msgr2=0x7f27f019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.640+0000 7f27e6000700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27f006b130 0x7f27f019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.640+0000 7f27e6000700 1 -- 172.21.15.40:0/33098895 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f019a090 con 0x7f27f006bad0 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.640+0000 7f27ef400700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27f01094a0 0x7f27f0199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.640+0000 7f27eea00700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27f006b130 0x7f27f019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.640+0000 7f27e6000700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f006bad0 0x7f27f019fcb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f27d80057f0 tx=0x7f27d800bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.641+0000 7f27e7e00700 1 -- 172.21.15.40:0/33098895 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d8026070 con 0x7f27f006bad0 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.641+0000 7f27f5800700 1 -- 172.21.15.40:0/33098895 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f019a320 con 0x7f27f006bad0 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.641+0000 7f27e7e00700 1 -- 172.21.15.40:0/33098895 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27d8021410 con 0x7f27f006bad0 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.641+0000 7f27e7e00700 1 -- 172.21.15.40:0/33098895 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d8017650 con 0x7f27f006bad0 2024-04-27T00:00:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.641+0000 7f27f5800700 1 -- 172.21.15.40:0/33098895 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f01a6640 con 0x7f27f006bad0 2024-04-27T00:00:45.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.642+0000 7f27f5800700 1 -- 172.21.15.40:0/33098895 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f006a110 con 0x7f27f006bad0 2024-04-27T00:00:45.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.645+0000 7f27e7e00700 1 -- 172.21.15.40:0/33098895 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f27d8029070 con 0x7f27f006bad0 2024-04-27T00:00:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.646+0000 7f27e7e00700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f27d40a3cd0 0x7f27d40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.647+0000 7f27e7e00700 1 -- 172.21.15.40:0/33098895 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f27d80d17b0 con 0x7f27f006bad0 2024-04-27T00:00:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.647+0000 7f27eea00700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f27d40a3cd0 0x7f27d40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.648+0000 7f27eea00700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f27d40a3cd0 0x7f27d40a6190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009680 tx=0x7f27e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.648+0000 7f27e7e00700 1 -- 172.21.15.40:0/33098895 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f27d8030080 con 0x7f27f006bad0 2024-04-27T00:00:45.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-27T00:00:45.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-04-27T00:00:45.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-04-27T00:00:45.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9, 5, 8]}]: dispatch 2024-04-27T00:00:45.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-04-27T00:00:45.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-04-27T00:00:45.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-04-27T00:00:45.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-27T00:00:45.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-04-27T00:00:45.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-27T00:00:45.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-04-27T00:00:45.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-04-27T00:00:45.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9, 5, 8]}]: dispatch 2024-04-27T00:00:45.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-04-27T00:00:45.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-04-27T00:00:45.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-04-27T00:00:45.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-27T00:00:45.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-04-27T00:00:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.879+0000 7f27f5800700 1 -- 172.21.15.40:0/33098895 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f27f019ae60 con 0x7f27f006bad0 2024-04-27T00:00:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.880+0000 7f27e7e00700 1 -- 172.21.15.40:0/33098895 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f27d8013070 con 0x7f27f006bad0 2024-04-27T00:00:45.882 INFO:teuthology.orchestra.run.smithi040.stdout:111669149778 2024-04-27T00:00:45.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.883+0000 7f27f5800700 1 -- 172.21.15.40:0/33098895 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f27d40a3cd0 msgr2=0x7f27d40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.883+0000 7f27f5800700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f27d40a3cd0 0x7f27d40a6190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009680 tx=0x7f27e0009260 comp rx=0 tx=0).stop 2024-04-27T00:00:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.883+0000 7f27f5800700 1 -- 172.21.15.40:0/33098895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f006bad0 msgr2=0x7f27f019fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.883+0000 7f27f5800700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f006bad0 0x7f27f019fcb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f27d80057f0 tx=0x7f27d800bfd0 comp rx=0 tx=0).stop 2024-04-27T00:00:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.883+0000 7f27f5800700 1 -- 172.21.15.40:0/33098895 shutdown_connections 2024-04-27T00:00:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.883+0000 7f27f5800700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f27d40a3cd0 0x7f27d40a6190 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.883+0000 7f27f5800700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27f01094a0 0x7f27f0199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.883+0000 7f27f5800700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27f006b130 0x7f27f019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.883+0000 7f27f5800700 1 --2- 172.21.15.40:0/33098895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27f006bad0 0x7f27f019fcb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.883+0000 7f27f5800700 1 -- 172.21.15.40:0/33098895 >> 172.21.15.40:0/33098895 conn(0x7f27f00792d0 msgr2=0x7f27f0100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.883+0000 7f27f5800700 1 -- 172.21.15.40:0/33098895 shutdown_connections 2024-04-27T00:00:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:45.883+0000 7f27f5800700 1 -- 172.21.15.40:0/33098895 wait complete. 2024-04-27T00:00:46.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:46 smithi040 ceph-mon[28095]: pgmap v380: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:46.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-27T00:00:46.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-04-27T00:00:46.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-04-27T00:00:46.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9, 5, 8]}]': finished 2024-04-27T00:00:46.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-04-27T00:00:46.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-04-27T00:00:46.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-04-27T00:00:46.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-27T00:00:46.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-04-27T00:00:46.640 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:46 smithi040 ceph-mon[28095]: osdmap e91: 12 total, 12 up, 12 in 2024-04-27T00:00:46.641 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:46 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/33098895' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T00:00:46.697 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149772 got 111669149778 for osd.3 2024-04-27T00:00:46.697 DEBUG:teuthology.parallel:result is None 2024-04-27T00:00:46.697 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-27T00:00:46.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-27T00:00:46.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:46 smithi080 ceph-mon[32362]: pgmap v380: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-27T00:00:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-04-27T00:00:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-04-27T00:00:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9, 5, 8]}]': finished 2024-04-27T00:00:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-04-27T00:00:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-04-27T00:00:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-04-27T00:00:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-27T00:00:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-04-27T00:00:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:46 smithi080 ceph-mon[32362]: osdmap e91: 12 total, 12 up, 12 in 2024-04-27T00:00:46.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:46 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/33098895' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T00:00:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:46 smithi103 ceph-mon[32952]: pgmap v380: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-27T00:00:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-04-27T00:00:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-04-27T00:00:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9, 5, 8]}]': finished 2024-04-27T00:00:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-04-27T00:00:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-04-27T00:00:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-04-27T00:00:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-27T00:00:46.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-04-27T00:00:46.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:46 smithi103 ceph-mon[32952]: osdmap e91: 12 total, 12 up, 12 in 2024-04-27T00:00:46.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:46 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/33098895' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T00:00:47.169 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:47.677 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:47 smithi040 ceph-mon[28095]: osdmap e92: 12 total, 12 up, 12 in 2024-04-27T00:00:47.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:47 smithi080 ceph-mon[32362]: osdmap e92: 12 total, 12 up, 12 in 2024-04-27T00:00:47.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:47 smithi103 ceph-mon[32952]: osdmap e92: 12 total, 12 up, 12 in 2024-04-27T00:00:48.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:48 smithi040 ceph-mon[28095]: pgmap v383: 129 pgs: 2 activating, 1 peering, 126 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:48.588 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:48 smithi040 ceph-mon[28095]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-27T00:00:48.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:48 smithi080 ceph-mon[32362]: pgmap v383: 129 pgs: 2 activating, 1 peering, 126 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:48.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:48 smithi080 ceph-mon[32362]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-27T00:00:48.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:48 smithi103 ceph-mon[32952]: pgmap v383: 129 pgs: 2 activating, 1 peering, 126 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:48.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:48 smithi103 ceph-mon[32952]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-27T00:00:49.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.493+0000 7f9f84200700 1 -- 172.21.15.40:0/786587380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f7c06b130 msgr2=0x7f9f7c06b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.493+0000 7f9f84200700 1 --2- 172.21.15.40:0/786587380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f7c06b130 0x7f9f7c06b590 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009990 tx=0x7f9f68009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.493+0000 7f9f84200700 1 -- 172.21.15.40:0/786587380 shutdown_connections 2024-04-27T00:00:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.493+0000 7f9f84200700 1 --2- 172.21.15.40:0/786587380 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9f7c06bad0 0x7f9f7c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.493+0000 7f9f84200700 1 --2- 172.21.15.40:0/786587380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f7c1094a0 0x7f9f7c109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.493+0000 7f9f84200700 1 --2- 172.21.15.40:0/786587380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f7c06b130 0x7f9f7c06b590 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.493+0000 7f9f84200700 1 -- 172.21.15.40:0/786587380 >> 172.21.15.40:0/786587380 conn(0x7f9f7c0792d0 msgr2=0x7f9f7c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.494+0000 7f9f84200700 1 -- 172.21.15.40:0/786587380 shutdown_connections 2024-04-27T00:00:49.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.494+0000 7f9f84200700 1 -- 172.21.15.40:0/786587380 wait complete. 2024-04-27T00:00:49.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.495+0000 7f9f84200700 1 Processor -- start 2024-04-27T00:00:49.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.495+0000 7f9f84200700 1 -- start start 2024-04-27T00:00:49.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.495+0000 7f9f84200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f7c06b130 0x7f9f7c19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:49.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.495+0000 7f9f84200700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f7c06bad0 0x7f9f7c19c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.495+0000 7f9f84200700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9f7c1094a0 0x7f9f7c1a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.495+0000 7f9f84200700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7c113600 con 0x7f9f7c06bad0 2024-04-27T00:00:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.495+0000 7f9f84200700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7c113480 con 0x7f9f7c1094a0 2024-04-27T00:00:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.495+0000 7f9f84200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7c113780 con 0x7f9f7c06b130 2024-04-27T00:00:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.496+0000 7f9f81800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f7c06b130 0x7f9f7c19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.496+0000 7f9f81800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f7c06b130 0x7f9f7c19bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46590/0 (socket says 172.21.15.40:46590) 2024-04-27T00:00:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.496+0000 7f9f81800700 1 -- 172.21.15.40:0/4126218944 learned_addr learned my addr 172.21.15.40:0/4126218944 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.496+0000 7f9f82200700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9f7c1094a0 0x7f9f7c1a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:49.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.496+0000 7f9f80e00700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f7c06bad0 0x7f9f7c19c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:49.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.496+0000 7f9f81800700 1 -- 172.21.15.40:0/4126218944 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9f7c1094a0 msgr2=0x7f9f7c1a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:49.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.496+0000 7f9f81800700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9f7c1094a0 0x7f9f7c1a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:49.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.496+0000 7f9f81800700 1 -- 172.21.15.40:0/4126218944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f7c06bad0 msgr2=0x7f9f7c19c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:49.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.496+0000 7f9f81800700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f7c06bad0 0x7f9f7c19c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:49.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.496+0000 7f9f81800700 1 -- 172.21.15.40:0/4126218944 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7c1a0d40 con 0x7f9f7c06b130 2024-04-27T00:00:49.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.496+0000 7f9f82200700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9f7c1094a0 0x7f9f7c1a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:49.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.497+0000 7f9f81800700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f7c06b130 0x7f9f7c19bd30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000e9a0 tx=0x7f9f7000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:49.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.497+0000 7f9f7a000700 1 -- 172.21.15.40:0/4126218944 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7000ca10 con 0x7f9f7c06b130 2024-04-27T00:00:49.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.497+0000 7f9f7a000700 1 -- 172.21.15.40:0/4126218944 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f7000cbb0 con 0x7f9f7c06b130 2024-04-27T00:00:49.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.497+0000 7f9f7a000700 1 -- 172.21.15.40:0/4126218944 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f70018680 con 0x7f9f7c06b130 2024-04-27T00:00:49.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.497+0000 7f9f84200700 1 -- 172.21.15.40:0/4126218944 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f7c1a1030 con 0x7f9f7c06b130 2024-04-27T00:00:49.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.497+0000 7f9f84200700 1 -- 172.21.15.40:0/4126218944 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f7c1a88b0 con 0x7f9f7c06b130 2024-04-27T00:00:49.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.498+0000 7f9f84200700 1 -- 172.21.15.40:0/4126218944 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f7c06a110 con 0x7f9f7c06b130 2024-04-27T00:00:49.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.500+0000 7f9f7a000700 1 -- 172.21.15.40:0/4126218944 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9f70010430 con 0x7f9f7c06b130 2024-04-27T00:00:49.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.502+0000 7f9f7a000700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9f640a3d50 0x7f9f640a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:49.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.502+0000 7f9f7a000700 1 -- 172.21.15.40:0/4126218944 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f9f700d09c0 con 0x7f9f7c06b130 2024-04-27T00:00:49.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.502+0000 7f9f80e00700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9f640a3d50 0x7f9f640a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:49.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.503+0000 7f9f80e00700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9f640a3d50 0x7f9f640a6210 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c110700 tx=0x7f9f6800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:49.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.506+0000 7f9f7a000700 1 -- 172.21.15.40:0/4126218944 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9f7009cea0 con 0x7f9f7c06b130 2024-04-27T00:00:49.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.734+0000 7f9f84200700 1 -- 172.21.15.40:0/4126218944 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f7c103080 con 0x7f9f640a3d50 2024-04-27T00:00:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.755+0000 7f9f7a000700 1 -- 172.21.15.40:0/4126218944 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340340 (secure 0 0 0) 0x7f9f6807b190 con 0x7f9f640a3d50 2024-04-27T00:00:49.759 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:00:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.764+0000 7f9f84200700 1 -- 172.21.15.40:0/4126218944 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9f640a3d50 msgr2=0x7f9f640a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.764+0000 7f9f84200700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9f640a3d50 0x7f9f640a6210 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c110700 tx=0x7f9f6800b260 comp rx=0 tx=0).stop 2024-04-27T00:00:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.764+0000 7f9f84200700 1 -- 172.21.15.40:0/4126218944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f7c06b130 msgr2=0x7f9f7c19bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.764+0000 7f9f84200700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f7c06b130 0x7f9f7c19bd30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000e9a0 tx=0x7f9f7000ecb0 comp rx=0 tx=0).stop 2024-04-27T00:00:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.764+0000 7f9f84200700 1 -- 172.21.15.40:0/4126218944 shutdown_connections 2024-04-27T00:00:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.764+0000 7f9f84200700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9f640a3d50 0x7f9f640a6210 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.764+0000 7f9f84200700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9f7c1094a0 0x7f9f7c1a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.764+0000 7f9f84200700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f7c06b130 0x7f9f7c19bd30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.764+0000 7f9f84200700 1 --2- 172.21.15.40:0/4126218944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f7c06bad0 0x7f9f7c19c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.764+0000 7f9f84200700 1 -- 172.21.15.40:0/4126218944 >> 172.21.15.40:0/4126218944 conn(0x7f9f7c0792d0 msgr2=0x7f9f7c100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.765+0000 7f9f84200700 1 -- 172.21.15.40:0/4126218944 shutdown_connections 2024-04-27T00:00:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:49.765+0000 7f9f84200700 1 -- 172.21.15.40:0/4126218944 wait complete. 2024-04-27T00:00:49.769 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-27T00:00:50.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:50 smithi040 ceph-mon[28095]: pgmap v384: 129 pgs: 2 activating, 1 peering, 126 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-27T00:00:50.646 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-04-27T00:00:48.751961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":650670,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":640,"num_read_kb":430,"num_write":346,"num_write_kb":3050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":420,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371528,"kb_used_data":14008,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570296,"statfs":{"total":1151940427776,"available":1151559983104,"internally_reserved":0,"allocated":14344192,"data_stored":7879683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003293"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390823+0000","last_change":"2024-04-26T23:58:51.162452+0000","last_active":"2024-04-27T00:00:46.390823+0000","last_peered":"2024-04-27T00:00:46.390823+0000","last_clean":"2024-04-27T00:00:46.390823+0000","last_became_active":"2024-04-26T23:58:51.162065+0000","last_became_peered":"2024-04-26T23:58:51.162065+0000","last_unstale":"2024-04-27T00:00:46.390823+0000","last_undegraded":"2024-04-27T00:00:46.390823+0000","last_fullsized":"2024-04-27T00:00:46.390823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:58:05.949099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.397895+0000","last_change":"2024-04-26T23:58:51.161144+0000","last_active":"2024-04-27T00:00:46.397895+0000","last_peered":"2024-04-27T00:00:46.397895+0000","last_clean":"2024-04-27T00:00:46.397895+0000","last_became_active":"2024-04-26T23:58:51.154033+0000","last_became_peered":"2024-04-26T23:58:51.154033+0000","last_unstale":"2024-04-27T00:00:46.397895+0000","last_undegraded":"2024-04-27T00:00:46.397895+0000","last_fullsized":"2024-04-27T00:00:46.397895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:37:39.757654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.398862+0000","last_change":"2024-04-26T23:58:51.159715+0000","last_active":"2024-04-27T00:00:46.398862+0000","last_peered":"2024-04-27T00:00:46.398862+0000","last_clean":"2024-04-27T00:00:46.398862+0000","last_became_active":"2024-04-26T23:58:51.152657+0000","last_became_peered":"2024-04-26T23:58:51.152657+0000","last_unstale":"2024-04-27T00:00:46.398862+0000","last_undegraded":"2024-04-27T00:00:46.398862+0000","last_fullsized":"2024-04-27T00:00:46.398862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:29.210295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.388048+0000","last_change":"2024-04-26T23:58:51.141814+0000","last_active":"2024-04-27T00:00:46.388048+0000","last_peered":"2024-04-27T00:00:46.388048+0000","last_clean":"2024-04-27T00:00:46.388048+0000","last_became_active":"2024-04-26T23:58:51.141344+0000","last_became_peered":"2024-04-26T23:58:51.141344+0000","last_unstale":"2024-04-27T00:00:46.388048+0000","last_undegraded":"2024-04-27T00:00:46.388048+0000","last_fullsized":"2024-04-27T00:00:46.388048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:05:49.532178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382099+0000","last_change":"2024-04-26T23:58:51.170537+0000","last_active":"2024-04-27T00:00:46.382099+0000","last_peered":"2024-04-27T00:00:46.382099+0000","last_clean":"2024-04-27T00:00:46.382099+0000","last_became_active":"2024-04-26T23:58:51.170047+0000","last_became_peered":"2024-04-26T23:58:51.170047+0000","last_unstale":"2024-04-27T00:00:46.382099+0000","last_undegraded":"2024-04-27T00:00:46.382099+0000","last_fullsized":"2024-04-27T00:00:46.382099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:46.463214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.369619+0000","last_change":"2024-04-26T23:58:51.141041+0000","last_active":"2024-04-27T00:00:45.369619+0000","last_peered":"2024-04-27T00:00:45.369619+0000","last_clean":"2024-04-27T00:00:45.369619+0000","last_became_active":"2024-04-26T23:58:51.140709+0000","last_became_peered":"2024-04-26T23:58:51.140709+0000","last_unstale":"2024-04-27T00:00:45.369619+0000","last_undegraded":"2024-04-27T00:00:45.369619+0000","last_fullsized":"2024-04-27T00:00:45.369619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:12:28.613723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451699+0000","last_change":"2024-04-26T23:58:51.145975+0000","last_active":"2024-04-27T00:00:46.451699+0000","last_peered":"2024-04-27T00:00:46.451699+0000","last_clean":"2024-04-27T00:00:46.451699+0000","last_became_active":"2024-04-26T23:58:51.145594+0000","last_became_peered":"2024-04-26T23:58:51.145594+0000","last_unstale":"2024-04-27T00:00:46.451699+0000","last_undegraded":"2024-04-27T00:00:46.451699+0000","last_fullsized":"2024-04-27T00:00:46.451699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:50:59.548009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211401+0000","last_change":"2024-04-26T23:58:51.160363+0000","last_active":"2024-04-26T23:58:56.211401+0000","last_peered":"2024-04-26T23:58:56.211401+0000","last_clean":"2024-04-26T23:58:56.211401+0000","last_became_active":"2024-04-26T23:58:51.152004+0000","last_became_peered":"2024-04-26T23:58:51.152004+0000","last_unstale":"2024-04-26T23:58:56.211401+0000","last_undegraded":"2024-04-26T23:58:56.211401+0000","last_fullsized":"2024-04-26T23:58:56.211401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:01:03.203404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.450588+0000","last_change":"2024-04-26T23:58:51.134247+0000","last_active":"2024-04-27T00:00:46.450588+0000","last_peered":"2024-04-27T00:00:46.450588+0000","last_clean":"2024-04-27T00:00:46.450588+0000","last_became_active":"2024-04-26T23:58:51.133889+0000","last_became_peered":"2024-04-26T23:58:51.133889+0000","last_unstale":"2024-04-27T00:00:46.450588+0000","last_undegraded":"2024-04-27T00:00:46.450588+0000","last_fullsized":"2024-04-27T00:00:46.450588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:51:04.245217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.370106+0000","last_change":"2024-04-26T23:58:51.135255+0000","last_active":"2024-04-27T00:00:45.370106+0000","last_peered":"2024-04-27T00:00:45.370106+0000","last_clean":"2024-04-27T00:00:45.370106+0000","last_became_active":"2024-04-26T23:58:51.134905+0000","last_became_peered":"2024-04-26T23:58:51.134905+0000","last_unstale":"2024-04-27T00:00:45.370106+0000","last_undegraded":"2024-04-27T00:00:45.370106+0000","last_fullsized":"2024-04-27T00:00:45.370106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:51.435008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452305+0000","last_change":"2024-04-26T23:58:51.134962+0000","last_active":"2024-04-27T00:00:46.452305+0000","last_peered":"2024-04-27T00:00:46.452305+0000","last_clean":"2024-04-27T00:00:46.452305+0000","last_became_active":"2024-04-26T23:58:51.134559+0000","last_became_peered":"2024-04-26T23:58:51.134559+0000","last_unstale":"2024-04-27T00:00:46.452305+0000","last_undegraded":"2024-04-27T00:00:46.452305+0000","last_fullsized":"2024-04-27T00:00:46.452305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:31:15.833585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387994+0000","last_change":"2024-04-26T23:58:51.141872+0000","last_active":"2024-04-27T00:00:46.387994+0000","last_peered":"2024-04-27T00:00:46.387994+0000","last_clean":"2024-04-27T00:00:46.387994+0000","last_became_active":"2024-04-26T23:58:51.141464+0000","last_became_peered":"2024-04-26T23:58:51.141464+0000","last_unstale":"2024-04-27T00:00:46.387994+0000","last_undegraded":"2024-04-27T00:00:46.387994+0000","last_fullsized":"2024-04-27T00:00:46.387994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:37.663379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.390032+0000","last_change":"2024-04-26T23:58:51.157828+0000","last_active":"2024-04-27T00:00:45.390032+0000","last_peered":"2024-04-27T00:00:45.390032+0000","last_clean":"2024-04-27T00:00:45.390032+0000","last_became_active":"2024-04-26T23:58:51.156820+0000","last_became_peered":"2024-04-26T23:58:51.156820+0000","last_unstale":"2024-04-27T00:00:45.390032+0000","last_undegraded":"2024-04-27T00:00:45.390032+0000","last_fullsized":"2024-04-27T00:00:45.390032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:22:43.201795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.397129+0000","last_change":"2024-04-26T23:58:51.154108+0000","last_active":"2024-04-27T00:00:46.397129+0000","last_peered":"2024-04-27T00:00:46.397129+0000","last_clean":"2024-04-27T00:00:46.397129+0000","last_became_active":"2024-04-26T23:58:51.149740+0000","last_became_peered":"2024-04-26T23:58:51.149740+0000","last_unstale":"2024-04-27T00:00:46.397129+0000","last_undegraded":"2024-04-27T00:00:46.397129+0000","last_fullsized":"2024-04-27T00:00:46.397129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:23:58.519297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.453312+0000","last_change":"2024-04-27T00:00:46.453312+0000","last_active":"2024-04-27T00:00:46.453312+0000","last_peered":"2024-04-27T00:00:46.453312+0000","last_clean":"2024-04-27T00:00:46.453312+0000","last_became_active":"2024-04-27T00:00:46.451293+0000","last_became_peered":"2024-04-27T00:00:46.451293+0000","last_unstale":"2024-04-27T00:00:46.453312+0000","last_undegraded":"2024-04-27T00:00:46.453312+0000","last_fullsized":"2024-04-27T00:00:46.453312+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:51.188145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449540+0000","last_change":"2024-04-26T23:58:51.162811+0000","last_active":"2024-04-27T00:00:46.449540+0000","last_peered":"2024-04-27T00:00:46.449540+0000","last_clean":"2024-04-27T00:00:46.449540+0000","last_became_active":"2024-04-26T23:58:51.162464+0000","last_became_peered":"2024-04-26T23:58:51.162464+0000","last_unstale":"2024-04-27T00:00:46.449540+0000","last_undegraded":"2024-04-27T00:00:46.449540+0000","last_fullsized":"2024-04-27T00:00:46.449540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:50:26.127894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390369+0000","last_change":"2024-04-26T23:58:51.162363+0000","last_active":"2024-04-27T00:00:46.390369+0000","last_peered":"2024-04-27T00:00:46.390369+0000","last_clean":"2024-04-27T00:00:46.390369+0000","last_became_active":"2024-04-26T23:58:51.162033+0000","last_became_peered":"2024-04-26T23:58:51.162033+0000","last_unstale":"2024-04-27T00:00:46.390369+0000","last_undegraded":"2024-04-27T00:00:46.390369+0000","last_fullsized":"2024-04-27T00:00:46.390369+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:53.755338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250657+0000","last_change":"2024-04-26T23:58:51.147373+0000","last_active":"2024-04-26T23:58:56.250657+0000","last_peered":"2024-04-26T23:58:56.250657+0000","last_clean":"2024-04-26T23:58:56.250657+0000","last_became_active":"2024-04-26T23:58:51.146993+0000","last_became_peered":"2024-04-26T23:58:51.146993+0000","last_unstale":"2024-04-26T23:58:56.250657+0000","last_undegraded":"2024-04-26T23:58:56.250657+0000","last_fullsized":"2024-04-26T23:58:56.250657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:14.887719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.371802+0000","last_change":"2024-04-26T23:58:51.159553+0000","last_active":"2024-04-27T00:00:45.371802+0000","last_peered":"2024-04-27T00:00:45.371802+0000","last_clean":"2024-04-27T00:00:45.371802+0000","last_became_active":"2024-04-26T23:58:51.159252+0000","last_became_peered":"2024-04-26T23:58:51.159252+0000","last_unstale":"2024-04-27T00:00:45.371802+0000","last_undegraded":"2024-04-27T00:00:45.371802+0000","last_fullsized":"2024-04-27T00:00:45.371802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:17.408273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.371166+0000","last_change":"2024-04-26T23:58:55.203263+0000","last_active":"2024-04-27T00:00:45.371166+0000","last_peered":"2024-04-27T00:00:45.371166+0000","last_clean":"2024-04-27T00:00:45.371166+0000","last_became_active":"2024-04-26T23:58:55.202832+0000","last_became_peered":"2024-04-26T23:58:55.202832+0000","last_unstale":"2024-04-27T00:00:45.371166+0000","last_undegraded":"2024-04-27T00:00:45.371166+0000","last_fullsized":"2024-04-27T00:00:45.371166+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:13:38.825916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:24.372129+0000","last_change":"2024-04-26T23:58:50.124003+0000","last_active":"2024-04-26T23:59:24.372129+0000","last_peered":"2024-04-26T23:59:24.372129+0000","last_clean":"2024-04-26T23:59:24.372129+0000","last_became_active":"2024-04-26T23:58:50.123653+0000","last_became_peered":"2024-04-26T23:58:50.123653+0000","last_unstale":"2024-04-26T23:59:24.372129+0000","last_undegraded":"2024-04-26T23:59:24.372129+0000","last_fullsized":"2024-04-26T23:59:24.372129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:16:57.688594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.212780+0000","last_change":"2024-04-26T23:58:51.175946+0000","last_active":"2024-04-26T23:58:56.212780+0000","last_peered":"2024-04-26T23:58:56.212780+0000","last_clean":"2024-04-26T23:58:56.212780+0000","last_became_active":"2024-04-26T23:58:51.174204+0000","last_became_peered":"2024-04-26T23:58:51.174204+0000","last_unstale":"2024-04-26T23:58:56.212780+0000","last_undegraded":"2024-04-26T23:58:56.212780+0000","last_fullsized":"2024-04-26T23:58:56.212780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:30:03.975717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":92,"state":"activating","last_fresh":"2024-04-27T00:00:46.389315+0000","last_change":"2024-04-27T00:00:46.387753+0000","last_active":"2024-04-27T00:00:45.375438+0000","last_peered":"2024-04-26T23:58:56.394217+0000","last_clean":"2024-04-26T23:58:56.394217+0000","last_became_active":"2024-04-26T23:58:55.204248+0000","last_became_peered":"2024-04-26T23:58:55.204248+0000","last_unstale":"2024-04-27T00:00:46.389315+0000","last_undegraded":"2024-04-27T00:00:46.389315+0000","last_fullsized":"2024-04-27T00:00:46.389315+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":["4(0)","3(3)","6(1)","8(2)","9(2)"],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.213176+0000","last_change":"2024-04-26T23:58:50.128773+0000","last_active":"2024-04-26T23:58:56.213176+0000","last_peered":"2024-04-26T23:58:56.213176+0000","last_clean":"2024-04-26T23:58:56.213176+0000","last_became_active":"2024-04-26T23:58:50.128445+0000","last_became_peered":"2024-04-26T23:58:50.128445+0000","last_unstale":"2024-04-26T23:58:56.213176+0000","last_undegraded":"2024-04-26T23:58:56.213176+0000","last_fullsized":"2024-04-26T23:58:56.213176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:27:57.577361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.399246+0000","last_change":"2024-04-26T23:58:51.164079+0000","last_active":"2024-04-27T00:00:46.399246+0000","last_peered":"2024-04-27T00:00:46.399246+0000","last_clean":"2024-04-27T00:00:46.399246+0000","last_became_active":"2024-04-26T23:58:51.162514+0000","last_became_peered":"2024-04-26T23:58:51.162514+0000","last_unstale":"2024-04-27T00:00:46.399246+0000","last_undegraded":"2024-04-27T00:00:46.399246+0000","last_fullsized":"2024-04-27T00:00:46.399246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:48:10.904737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387541+0000","last_change":"2024-04-26T23:58:55.211879+0000","last_active":"2024-04-27T00:00:46.387541+0000","last_peered":"2024-04-27T00:00:46.387541+0000","last_clean":"2024-04-27T00:00:46.387541+0000","last_became_active":"2024-04-26T23:58:55.211421+0000","last_became_peered":"2024-04-26T23:58:55.211421+0000","last_unstale":"2024-04-27T00:00:46.387541+0000","last_undegraded":"2024-04-27T00:00:46.387541+0000","last_fullsized":"2024-04-27T00:00:46.387541+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:45:12.754349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382584+0000","last_change":"2024-04-26T23:58:50.101893+0000","last_active":"2024-04-27T00:00:46.382584+0000","last_peered":"2024-04-27T00:00:46.382584+0000","last_clean":"2024-04-27T00:00:46.382584+0000","last_became_active":"2024-04-26T23:58:50.101625+0000","last_became_peered":"2024-04-26T23:58:50.101625+0000","last_unstale":"2024-04-27T00:00:46.382584+0000","last_undegraded":"2024-04-27T00:00:46.382584+0000","last_fullsized":"2024-04-27T00:00:46.382584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:05:24.365710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.450046+0000","last_change":"2024-04-26T23:58:51.146848+0000","last_active":"2024-04-27T00:00:46.450046+0000","last_peered":"2024-04-27T00:00:46.450046+0000","last_clean":"2024-04-27T00:00:46.450046+0000","last_became_active":"2024-04-26T23:58:51.146261+0000","last_became_peered":"2024-04-26T23:58:51.146261+0000","last_unstale":"2024-04-27T00:00:46.450046+0000","last_undegraded":"2024-04-27T00:00:46.450046+0000","last_fullsized":"2024-04-27T00:00:46.450046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:22.200191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387480+0000","last_change":"2024-04-26T23:58:55.196026+0000","last_active":"2024-04-27T00:00:46.387480+0000","last_peered":"2024-04-27T00:00:46.387480+0000","last_clean":"2024-04-27T00:00:46.387480+0000","last_became_active":"2024-04-26T23:58:55.195591+0000","last_became_peered":"2024-04-26T23:58:55.195591+0000","last_unstale":"2024-04-27T00:00:46.387480+0000","last_undegraded":"2024-04-27T00:00:46.387480+0000","last_fullsized":"2024-04-27T00:00:46.387480+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:06.980985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382090+0000","last_change":"2024-04-26T23:58:50.100655+0000","last_active":"2024-04-27T00:00:46.382090+0000","last_peered":"2024-04-27T00:00:46.382090+0000","last_clean":"2024-04-27T00:00:46.382090+0000","last_became_active":"2024-04-26T23:58:50.100314+0000","last_became_peered":"2024-04-26T23:58:50.100314+0000","last_unstale":"2024-04-27T00:00:46.382090+0000","last_undegraded":"2024-04-27T00:00:46.382090+0000","last_fullsized":"2024-04-27T00:00:46.382090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:10:06.230717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.373041+0000","last_change":"2024-04-26T23:58:51.142669+0000","last_active":"2024-04-27T00:00:45.373041+0000","last_peered":"2024-04-27T00:00:45.373041+0000","last_clean":"2024-04-27T00:00:45.373041+0000","last_became_active":"2024-04-26T23:58:51.142337+0000","last_became_peered":"2024-04-26T23:58:51.142337+0000","last_unstale":"2024-04-27T00:00:45.373041+0000","last_undegraded":"2024-04-27T00:00:45.373041+0000","last_fullsized":"2024-04-27T00:00:45.373041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:33:57.351429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.371592+0000","last_change":"2024-04-26T23:58:55.184331+0000","last_active":"2024-04-27T00:00:45.371592+0000","last_peered":"2024-04-27T00:00:45.371592+0000","last_clean":"2024-04-27T00:00:45.371592+0000","last_became_active":"2024-04-26T23:58:55.183928+0000","last_became_peered":"2024-04-26T23:58:55.183928+0000","last_unstale":"2024-04-27T00:00:45.371592+0000","last_undegraded":"2024-04-27T00:00:45.371592+0000","last_fullsized":"2024-04-27T00:00:45.371592+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:47.303104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":114,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.372806+0000","last_change":"2024-04-26T23:58:50.101084+0000","last_active":"2024-04-27T00:00:45.372806+0000","last_peered":"2024-04-27T00:00:45.372806+0000","last_clean":"2024-04-27T00:00:45.372806+0000","last_became_active":"2024-04-26T23:58:50.100736+0000","last_became_peered":"2024-04-26T23:58:50.100736+0000","last_unstale":"2024-04-27T00:00:45.372806+0000","last_undegraded":"2024-04-27T00:00:45.372806+0000","last_fullsized":"2024-04-27T00:00:45.372806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:43.157122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382318+0000","last_change":"2024-04-26T23:58:51.143834+0000","last_active":"2024-04-27T00:00:46.382318+0000","last_peered":"2024-04-27T00:00:46.382318+0000","last_clean":"2024-04-27T00:00:46.382318+0000","last_became_active":"2024-04-26T23:58:51.143480+0000","last_became_peered":"2024-04-26T23:58:51.143480+0000","last_unstale":"2024-04-27T00:00:46.382318+0000","last_undegraded":"2024-04-27T00:00:46.382318+0000","last_fullsized":"2024-04-27T00:00:46.382318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:34:25.474562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.371526+0000","last_change":"2024-04-26T23:58:55.215809+0000","last_active":"2024-04-27T00:00:45.371526+0000","last_peered":"2024-04-27T00:00:45.371526+0000","last_clean":"2024-04-27T00:00:45.371526+0000","last_became_active":"2024-04-26T23:58:55.215369+0000","last_became_peered":"2024-04-26T23:58:55.215369+0000","last_unstale":"2024-04-27T00:00:45.371526+0000","last_undegraded":"2024-04-27T00:00:45.371526+0000","last_fullsized":"2024-04-27T00:00:45.371526+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:04.053963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.369534+0000","last_change":"2024-04-26T23:58:50.129626+0000","last_active":"2024-04-27T00:00:45.369534+0000","last_peered":"2024-04-27T00:00:45.369534+0000","last_clean":"2024-04-27T00:00:45.369534+0000","last_became_active":"2024-04-26T23:58:50.129386+0000","last_became_peered":"2024-04-26T23:58:50.129386+0000","last_unstale":"2024-04-27T00:00:45.369534+0000","last_undegraded":"2024-04-27T00:00:45.369534+0000","last_fullsized":"2024-04-27T00:00:45.369534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:31:54.432289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.372944+0000","last_change":"2024-04-26T23:58:51.166312+0000","last_active":"2024-04-27T00:00:45.372944+0000","last_peered":"2024-04-27T00:00:45.372944+0000","last_clean":"2024-04-27T00:00:45.372944+0000","last_became_active":"2024-04-26T23:58:51.165986+0000","last_became_peered":"2024-04-26T23:58:51.165986+0000","last_unstale":"2024-04-27T00:00:45.372944+0000","last_undegraded":"2024-04-27T00:00:45.372944+0000","last_fullsized":"2024-04-27T00:00:45.372944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:51:49.549912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390622+0000","last_change":"2024-04-26T23:58:55.204142+0000","last_active":"2024-04-27T00:00:46.390622+0000","last_peered":"2024-04-27T00:00:46.390622+0000","last_clean":"2024-04-27T00:00:46.390622+0000","last_became_active":"2024-04-26T23:58:55.203646+0000","last_became_peered":"2024-04-26T23:58:55.203646+0000","last_unstale":"2024-04-27T00:00:46.390622+0000","last_undegraded":"2024-04-27T00:00:46.390622+0000","last_fullsized":"2024-04-27T00:00:46.390622+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:40.614034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.381910+0000","last_change":"2024-04-26T23:58:50.132080+0000","last_active":"2024-04-27T00:00:45.381910+0000","last_peered":"2024-04-27T00:00:45.381910+0000","last_clean":"2024-04-27T00:00:45.381910+0000","last_became_active":"2024-04-26T23:58:50.131618+0000","last_became_peered":"2024-04-26T23:58:50.131618+0000","last_unstale":"2024-04-27T00:00:45.381910+0000","last_undegraded":"2024-04-27T00:00:45.381910+0000","last_fullsized":"2024-04-27T00:00:45.381910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:42:07.897166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211042+0000","last_change":"2024-04-26T23:58:51.175083+0000","last_active":"2024-04-26T23:58:56.211042+0000","last_peered":"2024-04-26T23:58:56.211042+0000","last_clean":"2024-04-26T23:58:56.211042+0000","last_became_active":"2024-04-26T23:58:51.174159+0000","last_became_peered":"2024-04-26T23:58:51.174159+0000","last_unstale":"2024-04-26T23:58:56.211042+0000","last_undegraded":"2024-04-26T23:58:56.211042+0000","last_fullsized":"2024-04-26T23:58:56.211042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:51:06.583217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.397791+0000","last_change":"2024-04-26T23:58:55.200019+0000","last_active":"2024-04-27T00:00:46.397791+0000","last_peered":"2024-04-27T00:00:46.397791+0000","last_clean":"2024-04-27T00:00:46.397791+0000","last_became_active":"2024-04-26T23:58:55.199593+0000","last_became_peered":"2024-04-26T23:58:55.199593+0000","last_unstale":"2024-04-27T00:00:46.397791+0000","last_undegraded":"2024-04-27T00:00:46.397791+0000","last_fullsized":"2024-04-27T00:00:46.397791+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:41.465294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.372638+0000","last_change":"2024-04-26T23:58:50.100114+0000","last_active":"2024-04-27T00:00:45.372638+0000","last_peered":"2024-04-27T00:00:45.372638+0000","last_clean":"2024-04-27T00:00:45.372638+0000","last_became_active":"2024-04-26T23:58:50.099762+0000","last_became_peered":"2024-04-26T23:58:50.099762+0000","last_unstale":"2024-04-27T00:00:45.372638+0000","last_undegraded":"2024-04-27T00:00:45.372638+0000","last_fullsized":"2024-04-27T00:00:45.372638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:51:16.855014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249902+0000","last_change":"2024-04-26T23:58:51.171236+0000","last_active":"2024-04-26T23:58:56.249902+0000","last_peered":"2024-04-26T23:58:56.249902+0000","last_clean":"2024-04-26T23:58:56.249902+0000","last_became_active":"2024-04-26T23:58:51.170709+0000","last_became_peered":"2024-04-26T23:58:51.170709+0000","last_unstale":"2024-04-26T23:58:56.249902+0000","last_undegraded":"2024-04-26T23:58:56.249902+0000","last_fullsized":"2024-04-26T23:58:56.249902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:11.404827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.212920+0000","last_change":"2024-04-26T23:58:55.201931+0000","last_active":"2024-04-26T23:58:56.212920+0000","last_peered":"2024-04-26T23:58:56.212920+0000","last_clean":"2024-04-26T23:58:56.212920+0000","last_became_active":"2024-04-26T23:58:55.196810+0000","last_became_peered":"2024-04-26T23:58:55.196810+0000","last_unstale":"2024-04-26T23:58:56.212920+0000","last_undegraded":"2024-04-26T23:58:56.212920+0000","last_fullsized":"2024-04-26T23:58:56.212920+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:04:39.775990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.370048+0000","last_change":"2024-04-26T23:58:50.126477+0000","last_active":"2024-04-27T00:00:45.370048+0000","last_peered":"2024-04-27T00:00:45.370048+0000","last_clean":"2024-04-27T00:00:45.370048+0000","last_became_active":"2024-04-26T23:58:50.126205+0000","last_became_peered":"2024-04-26T23:58:50.126205+0000","last_unstale":"2024-04-27T00:00:45.370048+0000","last_undegraded":"2024-04-27T00:00:45.370048+0000","last_fullsized":"2024-04-27T00:00:45.370048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:03:18.513036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.450467+0000","last_change":"2024-04-26T23:58:51.145782+0000","last_active":"2024-04-27T00:00:46.450467+0000","last_peered":"2024-04-27T00:00:46.450467+0000","last_clean":"2024-04-27T00:00:46.450467+0000","last_became_active":"2024-04-26T23:58:51.145344+0000","last_became_peered":"2024-04-26T23:58:51.145344+0000","last_unstale":"2024-04-27T00:00:46.450467+0000","last_undegraded":"2024-04-27T00:00:46.450467+0000","last_fullsized":"2024-04-27T00:00:46.450467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:32.062690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.249720+0000","last_change":"2024-04-26T23:58:55.198953+0000","last_active":"2024-04-26T23:58:56.249720+0000","last_peered":"2024-04-26T23:58:56.249720+0000","last_clean":"2024-04-26T23:58:56.249720+0000","last_became_active":"2024-04-26T23:58:55.198560+0000","last_became_peered":"2024-04-26T23:58:55.198560+0000","last_unstale":"2024-04-26T23:58:56.249720+0000","last_undegraded":"2024-04-26T23:58:56.249720+0000","last_fullsized":"2024-04-26T23:58:56.249720+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:57:07.109698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452448+0000","last_change":"2024-04-26T23:58:50.103489+0000","last_active":"2024-04-27T00:00:46.452448+0000","last_peered":"2024-04-27T00:00:46.452448+0000","last_clean":"2024-04-27T00:00:46.452448+0000","last_became_active":"2024-04-26T23:58:50.103068+0000","last_became_peered":"2024-04-26T23:58:50.103068+0000","last_unstale":"2024-04-27T00:00:46.452448+0000","last_undegraded":"2024-04-27T00:00:46.452448+0000","last_fullsized":"2024-04-27T00:00:46.452448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:22:26.490816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.399359+0000","last_change":"2024-04-26T23:58:51.163680+0000","last_active":"2024-04-27T00:00:46.399359+0000","last_peered":"2024-04-27T00:00:46.399359+0000","last_clean":"2024-04-27T00:00:46.399359+0000","last_became_active":"2024-04-26T23:58:51.158135+0000","last_became_peered":"2024-04-26T23:58:51.158135+0000","last_unstale":"2024-04-27T00:00:46.399359+0000","last_undegraded":"2024-04-27T00:00:46.399359+0000","last_fullsized":"2024-04-27T00:00:46.399359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:56.562260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"activating","last_fresh":"2024-04-27T00:00:46.388237+0000","last_change":"2024-04-27T00:00:46.388237+0000","last_active":"2024-04-27T00:00:45.374349+0000","last_peered":"2024-04-26T23:58:56.394260+0000","last_clean":"2024-04-26T23:58:56.394260+0000","last_became_active":"2024-04-26T23:58:55.211430+0000","last_became_peered":"2024-04-26T23:58:55.211430+0000","last_unstale":"2024-04-27T00:00:46.388237+0000","last_undegraded":"2024-04-27T00:00:46.388237+0000","last_fullsized":"2024-04-27T00:00:46.388237+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":["4(0)","1(1)","2(2)","8(3)","9(3)"],"object_location_counts":[],"blocked_by":[1,2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.372881+0000","last_change":"2024-04-26T23:58:50.116030+0000","last_active":"2024-04-27T00:00:45.372881+0000","last_peered":"2024-04-27T00:00:45.372881+0000","last_clean":"2024-04-27T00:00:45.372881+0000","last_became_active":"2024-04-26T23:58:50.115744+0000","last_became_peered":"2024-04-26T23:58:50.115744+0000","last_unstale":"2024-04-27T00:00:45.372881+0000","last_undegraded":"2024-04-27T00:00:45.372881+0000","last_fullsized":"2024-04-27T00:00:45.372881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:05:21.823842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.450301+0000","last_change":"2024-04-26T23:58:51.140652+0000","last_active":"2024-04-27T00:00:46.450301+0000","last_peered":"2024-04-27T00:00:46.450301+0000","last_clean":"2024-04-27T00:00:46.450301+0000","last_became_active":"2024-04-26T23:58:51.140330+0000","last_became_peered":"2024-04-26T23:58:51.140330+0000","last_unstale":"2024-04-27T00:00:46.450301+0000","last_undegraded":"2024-04-27T00:00:46.450301+0000","last_fullsized":"2024-04-27T00:00:46.450301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:59:56.037856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250695+0000","last_change":"2024-04-26T23:58:55.200964+0000","last_active":"2024-04-26T23:58:56.250695+0000","last_peered":"2024-04-26T23:58:56.250695+0000","last_clean":"2024-04-26T23:58:56.250695+0000","last_became_active":"2024-04-26T23:58:55.200404+0000","last_became_peered":"2024-04-26T23:58:55.200404+0000","last_unstale":"2024-04-26T23:58:56.250695+0000","last_undegraded":"2024-04-26T23:58:56.250695+0000","last_fullsized":"2024-04-26T23:58:56.250695+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:43:51.159653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'32","reported_seq":226,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.517681+0000","last_change":"2024-04-26T23:58:50.101850+0000","last_active":"2024-04-27T00:00:45.517681+0000","last_peered":"2024-04-27T00:00:45.517681+0000","last_clean":"2024-04-27T00:00:45.517681+0000","last_became_active":"2024-04-26T23:58:50.101521+0000","last_became_peered":"2024-04-26T23:58:50.101521+0000","last_unstale":"2024-04-27T00:00:45.517681+0000","last_undegraded":"2024-04-27T00:00:45.517681+0000","last_fullsized":"2024-04-27T00:00:45.517681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:06.810644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55676,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":170,"num_read_kb":139,"num_write":33,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382478+0000","last_change":"2024-04-26T23:58:51.143441+0000","last_active":"2024-04-27T00:00:46.382478+0000","last_peered":"2024-04-27T00:00:46.382478+0000","last_clean":"2024-04-27T00:00:46.382478+0000","last_became_active":"2024-04-26T23:58:51.141779+0000","last_became_peered":"2024-04-26T23:58:51.141779+0000","last_unstale":"2024-04-27T00:00:46.382478+0000","last_undegraded":"2024-04-27T00:00:46.382478+0000","last_fullsized":"2024-04-27T00:00:46.382478+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:56:28.067685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.373586+0000","last_change":"2024-04-26T23:58:55.207277+0000","last_active":"2024-04-27T00:00:45.373586+0000","last_peered":"2024-04-27T00:00:45.373586+0000","last_clean":"2024-04-27T00:00:45.373586+0000","last_became_active":"2024-04-26T23:58:55.206905+0000","last_became_peered":"2024-04-26T23:58:55.206905+0000","last_unstale":"2024-04-27T00:00:45.373586+0000","last_undegraded":"2024-04-27T00:00:45.373586+0000","last_fullsized":"2024-04-27T00:00:45.373586+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:03:34.254752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.371400+0000","last_change":"2024-04-26T23:58:50.115642+0000","last_active":"2024-04-27T00:00:45.371400+0000","last_peered":"2024-04-27T00:00:45.371400+0000","last_clean":"2024-04-27T00:00:45.371400+0000","last_became_active":"2024-04-26T23:58:50.115310+0000","last_became_peered":"2024-04-26T23:58:50.115310+0000","last_unstale":"2024-04-27T00:00:45.371400+0000","last_undegraded":"2024-04-27T00:00:45.371400+0000","last_fullsized":"2024-04-27T00:00:45.371400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:33.189230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.387026+0000","last_change":"2024-04-26T23:58:51.159397+0000","last_active":"2024-04-27T00:00:45.387026+0000","last_peered":"2024-04-27T00:00:45.387026+0000","last_clean":"2024-04-27T00:00:45.387026+0000","last_became_active":"2024-04-26T23:58:51.159054+0000","last_became_peered":"2024-04-26T23:58:51.159054+0000","last_unstale":"2024-04-27T00:00:45.387026+0000","last_undegraded":"2024-04-27T00:00:45.387026+0000","last_fullsized":"2024-04-27T00:00:45.387026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:54:13.306198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390398+0000","last_change":"2024-04-26T23:58:55.215794+0000","last_active":"2024-04-27T00:00:46.390398+0000","last_peered":"2024-04-27T00:00:46.390398+0000","last_clean":"2024-04-27T00:00:46.390398+0000","last_became_active":"2024-04-26T23:58:55.215366+0000","last_became_peered":"2024-04-26T23:58:55.215366+0000","last_unstale":"2024-04-27T00:00:46.390398+0000","last_undegraded":"2024-04-27T00:00:46.390398+0000","last_fullsized":"2024-04-27T00:00:46.390398+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:54:20.347552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.373347+0000","last_change":"2024-04-26T23:58:50.112207+0000","last_active":"2024-04-27T00:00:45.373347+0000","last_peered":"2024-04-27T00:00:45.373347+0000","last_clean":"2024-04-27T00:00:45.373347+0000","last_became_active":"2024-04-26T23:58:50.111975+0000","last_became_peered":"2024-04-26T23:58:50.111975+0000","last_unstale":"2024-04-27T00:00:45.373347+0000","last_undegraded":"2024-04-27T00:00:45.373347+0000","last_fullsized":"2024-04-27T00:00:45.373347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:29.091698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452067+0000","last_change":"2024-04-26T23:58:51.147154+0000","last_active":"2024-04-27T00:00:46.452067+0000","last_peered":"2024-04-27T00:00:46.452067+0000","last_clean":"2024-04-27T00:00:46.452067+0000","last_became_active":"2024-04-26T23:58:51.146809+0000","last_became_peered":"2024-04-26T23:58:51.146809+0000","last_unstale":"2024-04-27T00:00:46.452067+0000","last_undegraded":"2024-04-27T00:00:46.452067+0000","last_fullsized":"2024-04-27T00:00:46.452067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:22.308365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.252855+0000","last_change":"2024-04-26T23:58:55.214461+0000","last_active":"2024-04-26T23:58:56.252855+0000","last_peered":"2024-04-26T23:58:56.252855+0000","last_clean":"2024-04-26T23:58:56.252855+0000","last_became_active":"2024-04-26T23:58:55.213442+0000","last_became_peered":"2024-04-26T23:58:55.213442+0000","last_unstale":"2024-04-26T23:58:56.252855+0000","last_undegraded":"2024-04-26T23:58:56.252855+0000","last_fullsized":"2024-04-26T23:58:56.252855+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:07:03.114274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":114,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387601+0000","last_change":"2024-04-26T23:58:50.100554+0000","last_active":"2024-04-27T00:00:46.387601+0000","last_peered":"2024-04-27T00:00:46.387601+0000","last_clean":"2024-04-27T00:00:46.387601+0000","last_became_active":"2024-04-26T23:58:50.100200+0000","last_became_peered":"2024-04-26T23:58:50.100200+0000","last_unstale":"2024-04-27T00:00:46.387601+0000","last_undegraded":"2024-04-27T00:00:46.387601+0000","last_fullsized":"2024-04-27T00:00:46.387601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:59.913424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451594+0000","last_change":"2024-04-26T23:58:51.141030+0000","last_active":"2024-04-27T00:00:46.451594+0000","last_peered":"2024-04-27T00:00:46.451594+0000","last_clean":"2024-04-27T00:00:46.451594+0000","last_became_active":"2024-04-26T23:58:51.140694+0000","last_became_peered":"2024-04-26T23:58:51.140694+0000","last_unstale":"2024-04-27T00:00:46.451594+0000","last_undegraded":"2024-04-27T00:00:46.451594+0000","last_fullsized":"2024-04-27T00:00:46.451594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:35:29.827020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.387147+0000","last_change":"2024-04-26T23:58:55.216556+0000","last_active":"2024-04-27T00:00:45.387147+0000","last_peered":"2024-04-27T00:00:45.387147+0000","last_clean":"2024-04-27T00:00:45.387147+0000","last_became_active":"2024-04-26T23:58:55.215798+0000","last_became_peered":"2024-04-26T23:58:55.215798+0000","last_unstale":"2024-04-27T00:00:45.387147+0000","last_undegraded":"2024-04-27T00:00:45.387147+0000","last_fullsized":"2024-04-27T00:00:45.387147+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:32:08.733655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.381919+0000","last_change":"2024-04-26T23:58:50.119495+0000","last_active":"2024-04-27T00:00:46.381919+0000","last_peered":"2024-04-27T00:00:46.381919+0000","last_clean":"2024-04-27T00:00:46.381919+0000","last_became_active":"2024-04-26T23:58:50.119174+0000","last_became_peered":"2024-04-26T23:58:50.119174+0000","last_unstale":"2024-04-27T00:00:46.381919+0000","last_undegraded":"2024-04-27T00:00:46.381919+0000","last_fullsized":"2024-04-27T00:00:46.381919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:53:23.408578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.372763+0000","last_change":"2024-04-26T23:58:51.141943+0000","last_active":"2024-04-27T00:00:45.372763+0000","last_peered":"2024-04-27T00:00:45.372763+0000","last_clean":"2024-04-27T00:00:45.372763+0000","last_became_active":"2024-04-26T23:58:51.141624+0000","last_became_peered":"2024-04-26T23:58:51.141624+0000","last_unstale":"2024-04-27T00:00:45.372763+0000","last_undegraded":"2024-04-27T00:00:45.372763+0000","last_fullsized":"2024-04-27T00:00:45.372763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:46:25.114059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.372558+0000","last_change":"2024-04-26T23:58:55.200247+0000","last_active":"2024-04-27T00:00:45.372558+0000","last_peered":"2024-04-27T00:00:45.372558+0000","last_clean":"2024-04-27T00:00:45.372558+0000","last_became_active":"2024-04-26T23:58:55.199854+0000","last_became_peered":"2024-04-26T23:58:55.199854+0000","last_unstale":"2024-04-27T00:00:45.372558+0000","last_undegraded":"2024-04-27T00:00:45.372558+0000","last_fullsized":"2024-04-27T00:00:45.372558+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:22:00.066549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382481+0000","last_change":"2024-04-26T23:58:50.102901+0000","last_active":"2024-04-27T00:00:46.382481+0000","last_peered":"2024-04-27T00:00:46.382481+0000","last_clean":"2024-04-27T00:00:46.382481+0000","last_became_active":"2024-04-26T23:58:50.102537+0000","last_became_peered":"2024-04-26T23:58:50.102537+0000","last_unstale":"2024-04-27T00:00:46.382481+0000","last_undegraded":"2024-04-27T00:00:46.382481+0000","last_fullsized":"2024-04-27T00:00:46.382481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:21:15.043327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.549175+0000","last_change":"2024-04-27T00:00:46.549175+0000","last_active":"2024-04-27T00:00:46.549175+0000","last_peered":"2024-04-27T00:00:46.549175+0000","last_clean":"2024-04-27T00:00:46.549175+0000","last_became_active":"2024-04-27T00:00:46.548691+0000","last_became_peered":"2024-04-27T00:00:46.548691+0000","last_unstale":"2024-04-27T00:00:46.549175+0000","last_undegraded":"2024-04-27T00:00:46.549175+0000","last_fullsized":"2024-04-27T00:00:46.549175+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:58:22.860610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382182+0000","last_change":"2024-04-26T23:58:55.203555+0000","last_active":"2024-04-27T00:00:46.382182+0000","last_peered":"2024-04-27T00:00:46.382182+0000","last_clean":"2024-04-27T00:00:46.382182+0000","last_became_active":"2024-04-26T23:58:55.202968+0000","last_became_peered":"2024-04-26T23:58:55.202968+0000","last_unstale":"2024-04-27T00:00:46.382182+0000","last_undegraded":"2024-04-27T00:00:46.382182+0000","last_fullsized":"2024-04-27T00:00:46.382182+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:01.149966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.250158+0000","last_change":"2024-04-26T23:58:50.129000+0000","last_active":"2024-04-26T23:58:56.250158+0000","last_peered":"2024-04-26T23:58:56.250158+0000","last_clean":"2024-04-26T23:58:56.250158+0000","last_became_active":"2024-04-26T23:58:50.128646+0000","last_became_peered":"2024-04-26T23:58:50.128646+0000","last_unstale":"2024-04-26T23:58:56.250158+0000","last_undegraded":"2024-04-26T23:58:56.250158+0000","last_fullsized":"2024-04-26T23:58:56.250158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:34.200578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.369828+0000","last_change":"2024-04-26T23:58:51.159557+0000","last_active":"2024-04-27T00:00:45.369828+0000","last_peered":"2024-04-27T00:00:45.369828+0000","last_clean":"2024-04-27T00:00:45.369828+0000","last_became_active":"2024-04-26T23:58:51.159211+0000","last_became_peered":"2024-04-26T23:58:51.159211+0000","last_unstale":"2024-04-27T00:00:45.369828+0000","last_undegraded":"2024-04-27T00:00:45.369828+0000","last_fullsized":"2024-04-27T00:00:45.369828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:48.740612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.390333+0000","last_change":"2024-04-26T23:58:55.193585+0000","last_active":"2024-04-27T00:00:45.390333+0000","last_peered":"2024-04-27T00:00:45.390333+0000","last_clean":"2024-04-27T00:00:45.390333+0000","last_became_active":"2024-04-26T23:58:55.193031+0000","last_became_peered":"2024-04-26T23:58:55.193031+0000","last_unstale":"2024-04-27T00:00:45.390333+0000","last_undegraded":"2024-04-27T00:00:45.390333+0000","last_fullsized":"2024-04-27T00:00:45.390333+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:20.689117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451772+0000","last_change":"2024-04-26T23:58:50.124301+0000","last_active":"2024-04-27T00:00:46.451772+0000","last_peered":"2024-04-27T00:00:46.451772+0000","last_clean":"2024-04-27T00:00:46.451772+0000","last_became_active":"2024-04-26T23:58:50.124016+0000","last_became_peered":"2024-04-26T23:58:50.124016+0000","last_unstale":"2024-04-27T00:00:46.451772+0000","last_undegraded":"2024-04-27T00:00:46.451772+0000","last_fullsized":"2024-04-27T00:00:46.451772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:00.835698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.369675+0000","last_change":"2024-04-26T23:58:51.164703+0000","last_active":"2024-04-27T00:00:45.369675+0000","last_peered":"2024-04-27T00:00:45.369675+0000","last_clean":"2024-04-27T00:00:45.369675+0000","last_became_active":"2024-04-26T23:58:51.164346+0000","last_became_peered":"2024-04-26T23:58:51.164346+0000","last_unstale":"2024-04-27T00:00:45.369675+0000","last_undegraded":"2024-04-27T00:00:45.369675+0000","last_fullsized":"2024-04-27T00:00:45.369675+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:45:39.667034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.391774+0000","last_change":"2024-04-26T23:58:55.193795+0000","last_active":"2024-04-27T00:00:45.391774+0000","last_peered":"2024-04-27T00:00:45.391774+0000","last_clean":"2024-04-27T00:00:45.391774+0000","last_became_active":"2024-04-26T23:58:55.193323+0000","last_became_peered":"2024-04-26T23:58:55.193323+0000","last_unstale":"2024-04-27T00:00:45.391774+0000","last_undegraded":"2024-04-27T00:00:45.391774+0000","last_fullsized":"2024-04-27T00:00:45.391774+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:44.301311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.208737+0000","last_change":"2024-04-26T23:58:51.140198+0000","last_active":"2024-04-26T23:58:56.208737+0000","last_peered":"2024-04-26T23:58:56.208737+0000","last_clean":"2024-04-26T23:58:56.208737+0000","last_became_active":"2024-04-26T23:58:51.139851+0000","last_became_peered":"2024-04-26T23:58:51.139851+0000","last_unstale":"2024-04-26T23:58:56.208737+0000","last_undegraded":"2024-04-26T23:58:56.208737+0000","last_fullsized":"2024-04-26T23:58:56.208737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:50.356312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.381813+0000","last_change":"2024-04-26T23:58:50.132099+0000","last_active":"2024-04-27T00:00:45.381813+0000","last_peered":"2024-04-27T00:00:45.381813+0000","last_clean":"2024-04-27T00:00:45.381813+0000","last_became_active":"2024-04-26T23:58:50.131712+0000","last_became_peered":"2024-04-26T23:58:50.131712+0000","last_unstale":"2024-04-27T00:00:45.381813+0000","last_undegraded":"2024-04-27T00:00:45.381813+0000","last_fullsized":"2024-04-27T00:00:45.381813+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:40.849259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452179+0000","last_change":"2024-04-26T23:58:55.205614+0000","last_active":"2024-04-27T00:00:46.452179+0000","last_peered":"2024-04-27T00:00:46.452179+0000","last_clean":"2024-04-27T00:00:46.452179+0000","last_became_active":"2024-04-26T23:58:55.205206+0000","last_became_peered":"2024-04-26T23:58:55.205206+0000","last_unstale":"2024-04-27T00:00:46.452179+0000","last_undegraded":"2024-04-27T00:00:46.452179+0000","last_fullsized":"2024-04-27T00:00:46.452179+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:42:24.324563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211463+0000","last_change":"2024-04-26T23:58:51.159808+0000","last_active":"2024-04-26T23:58:56.211463+0000","last_peered":"2024-04-26T23:58:56.211463+0000","last_clean":"2024-04-26T23:58:56.211463+0000","last_became_active":"2024-04-26T23:58:51.159444+0000","last_became_peered":"2024-04-26T23:58:51.159444+0000","last_unstale":"2024-04-26T23:58:56.211463+0000","last_undegraded":"2024-04-26T23:58:56.211463+0000","last_fullsized":"2024-04-26T23:58:56.211463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:57.401134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387605+0000","last_change":"2024-04-26T23:58:50.120100+0000","last_active":"2024-04-27T00:00:46.387605+0000","last_peered":"2024-04-27T00:00:46.387605+0000","last_clean":"2024-04-27T00:00:46.387605+0000","last_became_active":"2024-04-26T23:58:50.119613+0000","last_became_peered":"2024-04-26T23:58:50.119613+0000","last_unstale":"2024-04-27T00:00:46.387605+0000","last_undegraded":"2024-04-27T00:00:46.387605+0000","last_fullsized":"2024-04-27T00:00:46.387605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:32:18.116454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":348,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451438+0000","last_change":"2024-04-26T23:57:30.173979+0000","last_active":"2024-04-27T00:00:46.451438+0000","last_peered":"2024-04-27T00:00:46.451438+0000","last_clean":"2024-04-27T00:00:46.451438+0000","last_became_active":"2024-04-26T23:57:30.173549+0000","last_became_peered":"2024-04-26T23:57:30.173549+0000","last_unstale":"2024-04-27T00:00:46.451438+0000","last_undegraded":"2024-04-27T00:00:46.451438+0000","last_fullsized":"2024-04-27T00:00:46.451438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:53:33.634094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:53:33.634094+0000","last_clean_scrub_stamp":"2024-04-26T23:53:33.634094+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:48.026677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"peering","last_fresh":"2024-04-26T23:58:56.209548+0000","last_change":"2024-04-27T00:00:45.372875+0000","last_active":"2024-04-27T00:00:45.372875+0000","last_peered":"2024-04-26T23:58:56.209548+0000","last_clean":"2024-04-26T23:58:56.209548+0000","last_became_active":"2024-04-26T23:58:55.213586+0000","last_became_peered":"2024-04-26T23:58:55.213586+0000","last_unstale":"2024-04-26T23:58:56.209548+0000","last_undegraded":"2024-04-26T23:58:56.209548+0000","last_fullsized":"2024-04-26T23:58:56.209548+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,9,10,11],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.381456+0000","last_change":"2024-04-26T23:58:51.152687+0000","last_active":"2024-04-27T00:00:45.381456+0000","last_peered":"2024-04-27T00:00:45.381456+0000","last_clean":"2024-04-27T00:00:45.381456+0000","last_became_active":"2024-04-26T23:58:51.149082+0000","last_became_peered":"2024-04-26T23:58:51.149082+0000","last_unstale":"2024-04-27T00:00:45.381456+0000","last_undegraded":"2024-04-27T00:00:45.381456+0000","last_fullsized":"2024-04-27T00:00:45.381456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:38:42.630214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451590+0000","last_change":"2024-04-26T23:58:50.104485+0000","last_active":"2024-04-27T00:00:46.451590+0000","last_peered":"2024-04-27T00:00:46.451590+0000","last_clean":"2024-04-27T00:00:46.451590+0000","last_became_active":"2024-04-26T23:58:50.104260+0000","last_became_peered":"2024-04-26T23:58:50.104260+0000","last_unstale":"2024-04-27T00:00:46.451590+0000","last_undegraded":"2024-04-27T00:00:46.451590+0000","last_fullsized":"2024-04-27T00:00:46.451590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:04:06.268993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390116+0000","last_change":"2024-04-26T23:58:55.190219+0000","last_active":"2024-04-27T00:00:46.390116+0000","last_peered":"2024-04-27T00:00:46.390116+0000","last_clean":"2024-04-27T00:00:46.390116+0000","last_became_active":"2024-04-26T23:58:55.189801+0000","last_became_peered":"2024-04-26T23:58:55.189801+0000","last_unstale":"2024-04-27T00:00:46.390116+0000","last_undegraded":"2024-04-27T00:00:46.390116+0000","last_fullsized":"2024-04-27T00:00:46.390116+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:53:42.170710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382415+0000","last_change":"2024-04-26T23:58:50.117442+0000","last_active":"2024-04-27T00:00:46.382415+0000","last_peered":"2024-04-27T00:00:46.382415+0000","last_clean":"2024-04-27T00:00:46.382415+0000","last_became_active":"2024-04-26T23:58:50.117112+0000","last_became_peered":"2024-04-26T23:58:50.117112+0000","last_unstale":"2024-04-27T00:00:46.382415+0000","last_undegraded":"2024-04-27T00:00:46.382415+0000","last_fullsized":"2024-04-27T00:00:46.382415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:18.352145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.390512+0000","last_change":"2024-04-26T23:58:51.156670+0000","last_active":"2024-04-27T00:00:45.390512+0000","last_peered":"2024-04-27T00:00:45.390512+0000","last_clean":"2024-04-27T00:00:45.390512+0000","last_became_active":"2024-04-26T23:58:51.155596+0000","last_became_peered":"2024-04-26T23:58:51.155596+0000","last_unstale":"2024-04-27T00:00:45.390512+0000","last_undegraded":"2024-04-27T00:00:45.390512+0000","last_fullsized":"2024-04-27T00:00:45.390512+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:38.053814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211366+0000","last_change":"2024-04-26T23:58:55.201644+0000","last_active":"2024-04-26T23:58:56.211366+0000","last_peered":"2024-04-26T23:58:56.211366+0000","last_clean":"2024-04-26T23:58:56.211366+0000","last_became_active":"2024-04-26T23:58:55.196421+0000","last_became_peered":"2024-04-26T23:58:55.196421+0000","last_unstale":"2024-04-26T23:58:56.211366+0000","last_undegraded":"2024-04-26T23:58:56.211366+0000","last_fullsized":"2024-04-26T23:58:56.211366+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:26:30.309773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:32.129285+0000","last_change":"2024-04-26T23:58:50.120805+0000","last_active":"2024-04-26T23:59:32.129285+0000","last_peered":"2024-04-26T23:59:32.129285+0000","last_clean":"2024-04-26T23:59:32.129285+0000","last_became_active":"2024-04-26T23:58:50.120403+0000","last_became_peered":"2024-04-26T23:58:50.120403+0000","last_unstale":"2024-04-26T23:59:32.129285+0000","last_undegraded":"2024-04-26T23:59:32.129285+0000","last_fullsized":"2024-04-26T23:59:32.129285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:53:28.950313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.212866+0000","last_change":"2024-04-26T23:58:51.167109+0000","last_active":"2024-04-26T23:58:56.212866+0000","last_peered":"2024-04-26T23:58:56.212866+0000","last_clean":"2024-04-26T23:58:56.212866+0000","last_became_active":"2024-04-26T23:58:51.164584+0000","last_became_peered":"2024-04-26T23:58:51.164584+0000","last_unstale":"2024-04-26T23:58:56.212866+0000","last_undegraded":"2024-04-26T23:58:56.212866+0000","last_fullsized":"2024-04-26T23:58:56.212866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:12:12.160553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.370935+0000","last_change":"2024-04-26T23:58:55.183138+0000","last_active":"2024-04-27T00:00:45.370935+0000","last_peered":"2024-04-27T00:00:45.370935+0000","last_clean":"2024-04-27T00:00:45.370935+0000","last_became_active":"2024-04-26T23:58:55.182779+0000","last_became_peered":"2024-04-26T23:58:55.182779+0000","last_unstale":"2024-04-27T00:00:45.370935+0000","last_undegraded":"2024-04-27T00:00:45.370935+0000","last_fullsized":"2024-04-27T00:00:45.370935+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:16.866145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.388141+0000","last_change":"2024-04-26T23:58:50.120401+0000","last_active":"2024-04-27T00:00:46.388141+0000","last_peered":"2024-04-27T00:00:46.388141+0000","last_clean":"2024-04-27T00:00:46.388141+0000","last_became_active":"2024-04-26T23:58:50.120036+0000","last_became_peered":"2024-04-26T23:58:50.120036+0000","last_unstale":"2024-04-27T00:00:46.388141+0000","last_undegraded":"2024-04-27T00:00:46.388141+0000","last_fullsized":"2024-04-27T00:00:46.388141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:49.852298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.455636+0000","last_change":"2024-04-27T00:00:46.455636+0000","last_active":"2024-04-27T00:00:46.455636+0000","last_peered":"2024-04-27T00:00:46.455636+0000","last_clean":"2024-04-27T00:00:46.455636+0000","last_became_active":"2024-04-27T00:00:46.454493+0000","last_became_peered":"2024-04-27T00:00:46.454493+0000","last_unstale":"2024-04-27T00:00:46.455636+0000","last_undegraded":"2024-04-27T00:00:46.455636+0000","last_fullsized":"2024-04-27T00:00:46.455636+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:25:57.077168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387997+0000","last_change":"2024-04-26T23:58:55.216273+0000","last_active":"2024-04-27T00:00:46.387997+0000","last_peered":"2024-04-27T00:00:46.387997+0000","last_clean":"2024-04-27T00:00:46.387997+0000","last_became_active":"2024-04-26T23:58:55.215630+0000","last_became_peered":"2024-04-26T23:58:55.215630+0000","last_unstale":"2024-04-27T00:00:46.387997+0000","last_undegraded":"2024-04-27T00:00:46.387997+0000","last_fullsized":"2024-04-27T00:00:46.387997+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:15:41.195389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449630+0000","last_change":"2024-04-26T23:58:50.111216+0000","last_active":"2024-04-27T00:00:46.449630+0000","last_peered":"2024-04-27T00:00:46.449630+0000","last_clean":"2024-04-27T00:00:46.449630+0000","last_became_active":"2024-04-26T23:58:50.110969+0000","last_became_peered":"2024-04-26T23:58:50.110969+0000","last_unstale":"2024-04-27T00:00:46.449630+0000","last_undegraded":"2024-04-27T00:00:46.449630+0000","last_fullsized":"2024-04-27T00:00:46.449630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:01.319448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.381941+0000","last_change":"2024-04-26T23:58:51.132643+0000","last_active":"2024-04-27T00:00:46.381941+0000","last_peered":"2024-04-27T00:00:46.381941+0000","last_clean":"2024-04-27T00:00:46.381941+0000","last_became_active":"2024-04-26T23:58:51.132250+0000","last_became_peered":"2024-04-26T23:58:51.132250+0000","last_unstale":"2024-04-27T00:00:46.381941+0000","last_undegraded":"2024-04-27T00:00:46.381941+0000","last_fullsized":"2024-04-27T00:00:46.381941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:00:11.180178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.253164+0000","last_change":"2024-04-26T23:58:55.204718+0000","last_active":"2024-04-26T23:58:56.253164+0000","last_peered":"2024-04-26T23:58:56.253164+0000","last_clean":"2024-04-26T23:58:56.253164+0000","last_became_active":"2024-04-26T23:58:55.204159+0000","last_became_peered":"2024-04-26T23:58:55.204159+0000","last_unstale":"2024-04-26T23:58:56.253164+0000","last_undegraded":"2024-04-26T23:58:56.253164+0000","last_fullsized":"2024-04-26T23:58:56.253164+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:42:52.199200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:59:23.355283+0000","last_change":"2024-04-26T23:58:50.126926+0000","last_active":"2024-04-26T23:59:23.355283+0000","last_peered":"2024-04-26T23:59:23.355283+0000","last_clean":"2024-04-26T23:59:23.355283+0000","last_became_active":"2024-04-26T23:58:50.126552+0000","last_became_peered":"2024-04-26T23:58:50.126552+0000","last_unstale":"2024-04-26T23:59:23.355283+0000","last_undegraded":"2024-04-26T23:59:23.355283+0000","last_fullsized":"2024-04-26T23:59:23.355283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:18:25.024805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.389942+0000","last_change":"2024-04-26T23:58:51.162722+0000","last_active":"2024-04-27T00:00:46.389942+0000","last_peered":"2024-04-27T00:00:46.389942+0000","last_clean":"2024-04-27T00:00:46.389942+0000","last_became_active":"2024-04-26T23:58:51.162387+0000","last_became_peered":"2024-04-26T23:58:51.162387+0000","last_unstale":"2024-04-27T00:00:46.389942+0000","last_undegraded":"2024-04-27T00:00:46.389942+0000","last_fullsized":"2024-04-27T00:00:46.389942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:07.143478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387842+0000","last_change":"2024-04-26T23:58:55.216201+0000","last_active":"2024-04-27T00:00:46.387842+0000","last_peered":"2024-04-27T00:00:46.387842+0000","last_clean":"2024-04-27T00:00:46.387842+0000","last_became_active":"2024-04-26T23:58:55.215730+0000","last_became_peered":"2024-04-26T23:58:55.215730+0000","last_unstale":"2024-04-27T00:00:46.387842+0000","last_undegraded":"2024-04-27T00:00:46.387842+0000","last_fullsized":"2024-04-27T00:00:46.387842+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:21.484879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.371461+0000","last_change":"2024-04-26T23:58:50.111313+0000","last_active":"2024-04-27T00:00:45.371461+0000","last_peered":"2024-04-27T00:00:45.371461+0000","last_clean":"2024-04-27T00:00:45.371461+0000","last_became_active":"2024-04-26T23:58:50.110937+0000","last_became_peered":"2024-04-26T23:58:50.110937+0000","last_unstale":"2024-04-27T00:00:45.371461+0000","last_undegraded":"2024-04-27T00:00:45.371461+0000","last_fullsized":"2024-04-27T00:00:45.371461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:52:24.615527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.212337+0000","last_change":"2024-04-26T23:58:51.174152+0000","last_active":"2024-04-26T23:58:56.212337+0000","last_peered":"2024-04-26T23:58:56.212337+0000","last_clean":"2024-04-26T23:58:56.212337+0000","last_became_active":"2024-04-26T23:58:51.170956+0000","last_became_peered":"2024-04-26T23:58:51.170956+0000","last_unstale":"2024-04-26T23:58:56.212337+0000","last_undegraded":"2024-04-26T23:58:56.212337+0000","last_fullsized":"2024-04-26T23:58:56.212337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:09:09.253242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387993+0000","last_change":"2024-04-26T23:58:55.206248+0000","last_active":"2024-04-27T00:00:46.387993+0000","last_peered":"2024-04-27T00:00:46.387993+0000","last_clean":"2024-04-27T00:00:46.387993+0000","last_became_active":"2024-04-26T23:58:55.205820+0000","last_became_peered":"2024-04-26T23:58:55.205820+0000","last_unstale":"2024-04-27T00:00:46.387993+0000","last_undegraded":"2024-04-27T00:00:46.387993+0000","last_fullsized":"2024-04-27T00:00:46.387993+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:52:24.789706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382089+0000","last_change":"2024-04-26T23:58:50.103874+0000","last_active":"2024-04-27T00:00:46.382089+0000","last_peered":"2024-04-27T00:00:46.382089+0000","last_clean":"2024-04-27T00:00:46.382089+0000","last_became_active":"2024-04-26T23:58:50.103590+0000","last_became_peered":"2024-04-26T23:58:50.103590+0000","last_unstale":"2024-04-27T00:00:46.382089+0000","last_undegraded":"2024-04-27T00:00:46.382089+0000","last_fullsized":"2024-04-27T00:00:46.382089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:34:59.817736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451886+0000","last_change":"2024-04-26T23:58:51.146396+0000","last_active":"2024-04-27T00:00:46.451886+0000","last_peered":"2024-04-27T00:00:46.451886+0000","last_clean":"2024-04-27T00:00:46.451886+0000","last_became_active":"2024-04-26T23:58:51.146046+0000","last_became_peered":"2024-04-26T23:58:51.146046+0000","last_unstale":"2024-04-27T00:00:46.451886+0000","last_undegraded":"2024-04-27T00:00:46.451886+0000","last_fullsized":"2024-04-27T00:00:46.451886+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:14.230111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449790+0000","last_change":"2024-04-26T23:58:55.199320+0000","last_active":"2024-04-27T00:00:46.449790+0000","last_peered":"2024-04-27T00:00:46.449790+0000","last_clean":"2024-04-27T00:00:46.449790+0000","last_became_active":"2024-04-26T23:58:55.198843+0000","last_became_peered":"2024-04-26T23:58:55.198843+0000","last_unstale":"2024-04-27T00:00:46.449790+0000","last_undegraded":"2024-04-27T00:00:46.449790+0000","last_fullsized":"2024-04-27T00:00:46.449790+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:56:54.643998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.371802+0000","last_change":"2024-04-26T23:58:50.120892+0000","last_active":"2024-04-27T00:00:45.371802+0000","last_peered":"2024-04-27T00:00:45.371802+0000","last_clean":"2024-04-27T00:00:45.371802+0000","last_became_active":"2024-04-26T23:58:50.120396+0000","last_became_peered":"2024-04-26T23:58:50.120396+0000","last_unstale":"2024-04-27T00:00:45.371802+0000","last_undegraded":"2024-04-27T00:00:45.371802+0000","last_fullsized":"2024-04-27T00:00:45.371802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:16.670482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452329+0000","last_change":"2024-04-26T23:58:51.134917+0000","last_active":"2024-04-27T00:00:46.452329+0000","last_peered":"2024-04-27T00:00:46.452329+0000","last_clean":"2024-04-27T00:00:46.452329+0000","last_became_active":"2024-04-26T23:58:51.134470+0000","last_became_peered":"2024-04-26T23:58:51.134470+0000","last_unstale":"2024-04-27T00:00:46.452329+0000","last_undegraded":"2024-04-27T00:00:46.452329+0000","last_fullsized":"2024-04-27T00:00:46.452329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:03.696427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.398029+0000","last_change":"2024-04-26T23:58:55.213966+0000","last_active":"2024-04-27T00:00:46.398029+0000","last_peered":"2024-04-27T00:00:46.398029+0000","last_clean":"2024-04-27T00:00:46.398029+0000","last_became_active":"2024-04-26T23:58:55.213521+0000","last_became_peered":"2024-04-26T23:58:55.213521+0000","last_unstale":"2024-04-27T00:00:46.398029+0000","last_undegraded":"2024-04-27T00:00:46.398029+0000","last_fullsized":"2024-04-27T00:00:46.398029+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:10:02.998037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.373182+0000","last_change":"2024-04-26T23:58:50.118136+0000","last_active":"2024-04-27T00:00:45.373182+0000","last_peered":"2024-04-27T00:00:45.373182+0000","last_clean":"2024-04-27T00:00:45.373182+0000","last_became_active":"2024-04-26T23:58:50.117787+0000","last_became_peered":"2024-04-26T23:58:50.117787+0000","last_unstale":"2024-04-27T00:00:45.373182+0000","last_undegraded":"2024-04-27T00:00:45.373182+0000","last_fullsized":"2024-04-27T00:00:45.373182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:14.145327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451956+0000","last_change":"2024-04-26T23:58:51.166491+0000","last_active":"2024-04-27T00:00:46.451956+0000","last_peered":"2024-04-27T00:00:46.451956+0000","last_clean":"2024-04-27T00:00:46.451956+0000","last_became_active":"2024-04-26T23:58:51.166106+0000","last_became_peered":"2024-04-26T23:58:51.166106+0000","last_unstale":"2024-04-27T00:00:46.451956+0000","last_undegraded":"2024-04-27T00:00:46.451956+0000","last_fullsized":"2024-04-27T00:00:46.451956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:18:58.690731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.372412+0000","last_change":"2024-04-26T23:58:55.184623+0000","last_active":"2024-04-27T00:00:45.372412+0000","last_peered":"2024-04-27T00:00:45.372412+0000","last_clean":"2024-04-27T00:00:45.372412+0000","last_became_active":"2024-04-26T23:58:55.184029+0000","last_became_peered":"2024-04-26T23:58:55.184029+0000","last_unstale":"2024-04-27T00:00:45.372412+0000","last_undegraded":"2024-04-27T00:00:45.372412+0000","last_fullsized":"2024-04-27T00:00:45.372412+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:39:36.267352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.387010+0000","last_change":"2024-04-26T23:58:50.127630+0000","last_active":"2024-04-27T00:00:45.387010+0000","last_peered":"2024-04-27T00:00:45.387010+0000","last_clean":"2024-04-27T00:00:45.387010+0000","last_became_active":"2024-04-26T23:58:50.127134+0000","last_became_peered":"2024-04-26T23:58:50.127134+0000","last_unstale":"2024-04-27T00:00:45.387010+0000","last_undegraded":"2024-04-27T00:00:45.387010+0000","last_fullsized":"2024-04-27T00:00:45.387010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:15.061919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452430+0000","last_change":"2024-04-26T23:58:51.134241+0000","last_active":"2024-04-27T00:00:46.452430+0000","last_peered":"2024-04-27T00:00:46.452430+0000","last_clean":"2024-04-27T00:00:46.452430+0000","last_became_active":"2024-04-26T23:58:51.133867+0000","last_became_peered":"2024-04-26T23:58:51.133867+0000","last_unstale":"2024-04-27T00:00:46.452430+0000","last_undegraded":"2024-04-27T00:00:46.452430+0000","last_fullsized":"2024-04-27T00:00:46.452430+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:12:20.171560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390031+0000","last_change":"2024-04-26T23:58:51.132681+0000","last_active":"2024-04-27T00:00:46.390031+0000","last_peered":"2024-04-27T00:00:46.390031+0000","last_clean":"2024-04-27T00:00:46.390031+0000","last_became_active":"2024-04-26T23:58:51.132213+0000","last_became_peered":"2024-04-26T23:58:51.132213+0000","last_unstale":"2024-04-27T00:00:46.390031+0000","last_undegraded":"2024-04-27T00:00:46.390031+0000","last_fullsized":"2024-04-27T00:00:46.390031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:26:52.080670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.372121+0000","last_change":"2024-04-26T23:58:51.135731+0000","last_active":"2024-04-27T00:00:45.372121+0000","last_peered":"2024-04-27T00:00:45.372121+0000","last_clean":"2024-04-27T00:00:45.372121+0000","last_became_active":"2024-04-26T23:58:51.135352+0000","last_became_peered":"2024-04-26T23:58:51.135352+0000","last_unstale":"2024-04-27T00:00:45.372121+0000","last_undegraded":"2024-04-27T00:00:45.372121+0000","last_fullsized":"2024-04-27T00:00:45.372121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:45:37.903396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.371254+0000","last_change":"2024-04-26T23:58:51.160014+0000","last_active":"2024-04-27T00:00:45.371254+0000","last_peered":"2024-04-27T00:00:45.371254+0000","last_clean":"2024-04-27T00:00:45.371254+0000","last_became_active":"2024-04-26T23:58:51.159459+0000","last_became_peered":"2024-04-26T23:58:51.159459+0000","last_unstale":"2024-04-27T00:00:45.371254+0000","last_undegraded":"2024-04-27T00:00:45.371254+0000","last_fullsized":"2024-04-27T00:00:45.371254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:12.757558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.398357+0000","last_change":"2024-04-26T23:58:51.166942+0000","last_active":"2024-04-27T00:00:46.398357+0000","last_peered":"2024-04-27T00:00:46.398357+0000","last_clean":"2024-04-27T00:00:46.398357+0000","last_became_active":"2024-04-26T23:58:51.162605+0000","last_became_peered":"2024-04-26T23:58:51.162605+0000","last_unstale":"2024-04-27T00:00:46.398357+0000","last_undegraded":"2024-04-27T00:00:46.398357+0000","last_fullsized":"2024-04-27T00:00:46.398357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:38:46.346098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.390621+0000","last_change":"2024-04-26T23:58:51.162638+0000","last_active":"2024-04-27T00:00:45.390621+0000","last_peered":"2024-04-27T00:00:45.390621+0000","last_clean":"2024-04-27T00:00:45.390621+0000","last_became_active":"2024-04-26T23:58:51.162165+0000","last_became_peered":"2024-04-26T23:58:51.162165+0000","last_unstale":"2024-04-27T00:00:45.390621+0000","last_undegraded":"2024-04-27T00:00:45.390621+0000","last_fullsized":"2024-04-27T00:00:45.390621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:26:32.277183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390359+0000","last_change":"2024-04-26T23:58:51.132667+0000","last_active":"2024-04-27T00:00:46.390359+0000","last_peered":"2024-04-27T00:00:46.390359+0000","last_clean":"2024-04-27T00:00:46.390359+0000","last_became_active":"2024-04-26T23:58:51.132316+0000","last_became_peered":"2024-04-26T23:58:51.132316+0000","last_unstale":"2024-04-27T00:00:46.390359+0000","last_undegraded":"2024-04-27T00:00:46.390359+0000","last_fullsized":"2024-04-27T00:00:46.390359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:41.427650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.371857+0000","last_change":"2024-04-26T23:58:51.132614+0000","last_active":"2024-04-27T00:00:45.371857+0000","last_peered":"2024-04-27T00:00:45.371857+0000","last_clean":"2024-04-27T00:00:45.371857+0000","last_became_active":"2024-04-26T23:58:51.132274+0000","last_became_peered":"2024-04-26T23:58:51.132274+0000","last_unstale":"2024-04-27T00:00:45.371857+0000","last_undegraded":"2024-04-27T00:00:45.371857+0000","last_fullsized":"2024-04-27T00:00:45.371857+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:52.810401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.213133+0000","last_change":"2024-04-26T23:58:51.165470+0000","last_active":"2024-04-26T23:58:56.213133+0000","last_peered":"2024-04-26T23:58:56.213133+0000","last_clean":"2024-04-26T23:58:56.213133+0000","last_became_active":"2024-04-26T23:58:51.164998+0000","last_became_peered":"2024-04-26T23:58:51.164998+0000","last_unstale":"2024-04-26T23:58:56.213133+0000","last_undegraded":"2024-04-26T23:58:56.213133+0000","last_fullsized":"2024-04-26T23:58:56.213133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:02.703215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.373116+0000","last_change":"2024-04-26T23:58:51.166390+0000","last_active":"2024-04-27T00:00:45.373116+0000","last_peered":"2024-04-27T00:00:45.373116+0000","last_clean":"2024-04-27T00:00:45.373116+0000","last_became_active":"2024-04-26T23:58:51.166113+0000","last_became_peered":"2024-04-26T23:58:51.166113+0000","last_unstale":"2024-04-27T00:00:45.373116+0000","last_undegraded":"2024-04-27T00:00:45.373116+0000","last_fullsized":"2024-04-27T00:00:45.373116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:24:49.676207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T23:58:56.211288+0000","last_change":"2024-04-26T23:58:51.165272+0000","last_active":"2024-04-26T23:58:56.211288+0000","last_peered":"2024-04-26T23:58:56.211288+0000","last_clean":"2024-04-26T23:58:56.211288+0000","last_became_active":"2024-04-26T23:58:51.160355+0000","last_became_peered":"2024-04-26T23:58:51.160355+0000","last_unstale":"2024-04-26T23:58:56.211288+0000","last_undegraded":"2024-04-26T23:58:56.211288+0000","last_fullsized":"2024-04-26T23:58:56.211288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:32.910129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449946+0000","last_change":"2024-04-26T23:58:51.159564+0000","last_active":"2024-04-27T00:00:46.449946+0000","last_peered":"2024-04-27T00:00:46.449946+0000","last_clean":"2024-04-27T00:00:46.449946+0000","last_became_active":"2024-04-26T23:58:51.159229+0000","last_became_peered":"2024-04-26T23:58:51.159229+0000","last_unstale":"2024-04-27T00:00:46.449946+0000","last_undegraded":"2024-04-27T00:00:46.449946+0000","last_fullsized":"2024-04-27T00:00:46.449946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:39.032595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-27T00:00:45.389835+0000","last_change":"2024-04-26T23:58:51.163868+0000","last_active":"2024-04-27T00:00:45.389835+0000","last_peered":"2024-04-27T00:00:45.389835+0000","last_clean":"2024-04-27T00:00:45.389835+0000","last_became_active":"2024-04-26T23:58:51.163450+0000","last_became_peered":"2024-04-26T23:58:51.163450+0000","last_unstale":"2024-04-27T00:00:45.389835+0000","last_undegraded":"2024-04-27T00:00:45.389835+0000","last_fullsized":"2024-04-27T00:00:45.389835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:42:06.945605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390356+0000","last_change":"2024-04-26T23:58:51.136047+0000","last_active":"2024-04-27T00:00:46.390356+0000","last_peered":"2024-04-27T00:00:46.390356+0000","last_clean":"2024-04-27T00:00:46.390356+0000","last_became_active":"2024-04-26T23:58:51.135690+0000","last_became_peered":"2024-04-26T23:58:51.135690+0000","last_unstale":"2024-04-27T00:00:46.390356+0000","last_undegraded":"2024-04-27T00:00:46.390356+0000","last_fullsized":"2024-04-27T00:00:46.390356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:55.721858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":60068,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":514,"num_read_kb":321,"num_write":163,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":180204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351014,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":503795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28680,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716472,"statfs":{"total":95995035648,"available":95965667328,"internally_reserved":0,"allocated":1699840,"data_stored":1137819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776176,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":489276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037816,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":503951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299452,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1642496,"data_stored":1100251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1011712,"data_stored":489896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1089536,"data_stored":554793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1052672,"data_stored":500595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":547223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":480550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888143,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":494937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724616,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28596,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716556,"statfs":{"total":95995035648,"available":95965753344,"internally_reserved":0,"allocated":1617920,"data_stored":1076597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":55676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":57417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":57645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T00:00:50.652 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-27T00:00:50.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:50 smithi080 ceph-mon[32362]: pgmap v384: 129 pgs: 2 activating, 1 peering, 126 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-27T00:00:50.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:50 smithi103 ceph-mon[32952]: pgmap v384: 129 pgs: 2 activating, 1 peering, 126 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-27T00:00:51.174 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:51.626 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:51 smithi040 ceph-mon[28095]: from='client.25063 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T00:00:51.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:51 smithi080 ceph-mon[32362]: from='client.25063 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T00:00:51.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:51 smithi103 ceph-mon[32952]: from='client.25063 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T00:00:52.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:52 smithi040 ceph-mon[28095]: pgmap v385: 129 pgs: 2 activating, 127 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-27T00:00:52.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:52 smithi040 ceph-mon[28095]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-27T00:00:52.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:52 smithi040 ceph-mon[28095]: Cluster is now healthy 2024-04-27T00:00:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:52 smithi080 ceph-mon[32362]: pgmap v385: 129 pgs: 2 activating, 127 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-27T00:00:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:52 smithi080 ceph-mon[32362]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-27T00:00:52.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:52 smithi080 ceph-mon[32362]: Cluster is now healthy 2024-04-27T00:00:52.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:52 smithi103 ceph-mon[32952]: pgmap v385: 129 pgs: 2 activating, 127 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-27T00:00:52.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:52 smithi103 ceph-mon[32952]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-27T00:00:52.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:52 smithi103 ceph-mon[32952]: Cluster is now healthy 2024-04-27T00:00:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.390+0000 7fd279600700 1 -- 172.21.15.40:0/1418484341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd274109a50 msgr2=0x7fd274109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.390+0000 7fd279600700 1 --2- 172.21.15.40:0/1418484341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd274109a50 0x7fd274109ed0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd260009990 tx=0x7fd260009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.390+0000 7fd279600700 1 -- 172.21.15.40:0/1418484341 shutdown_connections 2024-04-27T00:00:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.390+0000 7fd279600700 1 --2- 172.21.15.40:0/1418484341 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd27410a410 0x7fd274116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.390+0000 7fd279600700 1 --2- 172.21.15.40:0/1418484341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd27410fa50 0x7fd27410fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.390+0000 7fd279600700 1 --2- 172.21.15.40:0/1418484341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd274109a50 0x7fd274109ed0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.390+0000 7fd279600700 1 -- 172.21.15.40:0/1418484341 >> 172.21.15.40:0/1418484341 conn(0x7fd274076bf0 msgr2=0x7fd274108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.390+0000 7fd279600700 1 -- 172.21.15.40:0/1418484341 shutdown_connections 2024-04-27T00:00:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.390+0000 7fd279600700 1 -- 172.21.15.40:0/1418484341 wait complete. 2024-04-27T00:00:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.391+0000 7fd279600700 1 Processor -- start 2024-04-27T00:00:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.391+0000 7fd279600700 1 -- start start 2024-04-27T00:00:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.392+0000 7fd279600700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd274109a50 0x7fd2741a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.392+0000 7fd279600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd27410a410 0x7fd2741a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.392+0000 7fd279600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd27410fa50 0x7fd2741a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.392+0000 7fd279600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd274119ce0 con 0x7fd27410a410 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.392+0000 7fd279600700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fd274119b60 con 0x7fd274109a50 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.392+0000 7fd279600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd274119e60 con 0x7fd27410fa50 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.392+0000 7fd272000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd27410a410 0x7fd2741a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.392+0000 7fd273400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd27410fa50 0x7fd2741a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.392+0000 7fd272000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd27410a410 0x7fd2741a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50024/0 (socket says 172.21.15.40:50024) 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.392+0000 7fd272000700 1 -- 172.21.15.40:0/4195884891 learned_addr learned my addr 172.21.15.40:0/4195884891 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.392+0000 7fd272a00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd274109a50 0x7fd2741a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.393+0000 7fd273400700 1 -- 172.21.15.40:0/4195884891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd274109a50 msgr2=0x7fd2741a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.393+0000 7fd273400700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd274109a50 0x7fd2741a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.393+0000 7fd273400700 1 -- 172.21.15.40:0/4195884891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd27410a410 msgr2=0x7fd2741a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.393+0000 7fd273400700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd27410a410 0x7fd2741a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.393+0000 7fd273400700 1 -- 172.21.15.40:0/4195884891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2741a51e0 con 0x7fd27410fa50 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.393+0000 7fd272a00700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd274109a50 0x7fd2741a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.393+0000 7fd273400700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd27410fa50 0x7fd2741a4aa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00efc0 tx=0x7fd26c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:53.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.393+0000 7fd26b400700 1 -- 172.21.15.40:0/4195884891 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0092e0 con 0x7fd27410fa50 2024-04-27T00:00:53.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.393+0000 7fd279600700 1 -- 172.21.15.40:0/4195884891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2741a5470 con 0x7fd27410fa50 2024-04-27T00:00:53.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.393+0000 7fd26b400700 1 -- 172.21.15.40:0/4195884891 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd26c004bb0 con 0x7fd27410fa50 2024-04-27T00:00:53.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.393+0000 7fd26b400700 1 -- 172.21.15.40:0/4195884891 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c004020 con 0x7fd27410fa50 2024-04-27T00:00:53.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.394+0000 7fd279600700 1 -- 172.21.15.40:0/4195884891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2741acd50 con 0x7fd27410fa50 2024-04-27T00:00:53.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.395+0000 7fd279600700 1 -- 172.21.15.40:0/4195884891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd27406a110 con 0x7fd27410fa50 2024-04-27T00:00:53.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.397+0000 7fd26b400700 1 -- 172.21.15.40:0/4195884891 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd26c01b030 con 0x7fd27410fa50 2024-04-27T00:00:53.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.398+0000 7fd26b400700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd2580a3cd0 0x7fd2580a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:53.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.398+0000 7fd26b400700 1 -- 172.21.15.40:0/4195884891 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fd26c0d0390 con 0x7fd27410fa50 2024-04-27T00:00:53.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.398+0000 7fd272a00700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd2580a3cd0 0x7fd2580a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:53.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.399+0000 7fd272a00700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd2580a3cd0 0x7fd2580a6190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd264009fd0 tx=0x7fd264009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.403+0000 7fd26b400700 1 -- 172.21.15.40:0/4195884891 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd26c022020 con 0x7fd27410fa50 2024-04-27T00:00:53.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.627+0000 7fd279600700 1 -- 172.21.15.40:0/4195884891 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd2741a1530 con 0x7fd2580a3cd0 2024-04-27T00:00:53.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.640+0000 7fd26b400700 1 -- 172.21.15.40:0/4195884891 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340399 (secure 0 0 0) 0x7fd2741a1530 con 0x7fd2580a3cd0 2024-04-27T00:00:53.647 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:00:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.647+0000 7fd279600700 1 -- 172.21.15.40:0/4195884891 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd2580a3cd0 msgr2=0x7fd2580a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.647+0000 7fd279600700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd2580a3cd0 0x7fd2580a6190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd264009fd0 tx=0x7fd264009260 comp rx=0 tx=0).stop 2024-04-27T00:00:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.647+0000 7fd279600700 1 -- 172.21.15.40:0/4195884891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd27410fa50 msgr2=0x7fd2741a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.647+0000 7fd279600700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd27410fa50 0x7fd2741a4aa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00efc0 tx=0x7fd26c00c3a0 comp rx=0 tx=0).stop 2024-04-27T00:00:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.647+0000 7fd279600700 1 -- 172.21.15.40:0/4195884891 shutdown_connections 2024-04-27T00:00:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.647+0000 7fd279600700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd2580a3cd0 0x7fd2580a6190 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.647+0000 7fd279600700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd27410fa50 0x7fd2741a4aa0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.647+0000 7fd279600700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd274109a50 0x7fd2741a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.647+0000 7fd279600700 1 --2- 172.21.15.40:0/4195884891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd27410a410 0x7fd2741a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:53.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.647+0000 7fd279600700 1 -- 172.21.15.40:0/4195884891 >> 172.21.15.40:0/4195884891 conn(0x7fd274076bf0 msgr2=0x7fd274077310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:53.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.647+0000 7fd279600700 1 -- 172.21.15.40:0/4195884891 shutdown_connections 2024-04-27T00:00:53.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:53.648+0000 7fd279600700 1 -- 172.21.15.40:0/4195884891 wait complete. 2024-04-27T00:00:53.651 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-04-27T00:00:54.555 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-04-27T00:00:52.753653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":650670,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":670,"num_read_kb":445,"num_write":349,"num_write_kb":3051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":281,"ondisk_log_size":281,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":415,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371704,"kb_used_data":14168,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570120,"statfs":{"total":1151940427776,"available":1151559802880,"internally_reserved":0,"allocated":14508032,"data_stored":7973480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003540"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390823+0000","last_change":"2024-04-26T23:58:51.162452+0000","last_active":"2024-04-27T00:00:46.390823+0000","last_peered":"2024-04-27T00:00:46.390823+0000","last_clean":"2024-04-27T00:00:46.390823+0000","last_became_active":"2024-04-26T23:58:51.162065+0000","last_became_peered":"2024-04-26T23:58:51.162065+0000","last_unstale":"2024-04-27T00:00:46.390823+0000","last_undegraded":"2024-04-27T00:00:46.390823+0000","last_fullsized":"2024-04-27T00:00:46.390823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:58:05.949099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.397895+0000","last_change":"2024-04-26T23:58:51.161144+0000","last_active":"2024-04-27T00:00:46.397895+0000","last_peered":"2024-04-27T00:00:46.397895+0000","last_clean":"2024-04-27T00:00:46.397895+0000","last_became_active":"2024-04-26T23:58:51.154033+0000","last_became_peered":"2024-04-26T23:58:51.154033+0000","last_unstale":"2024-04-27T00:00:46.397895+0000","last_undegraded":"2024-04-27T00:00:46.397895+0000","last_fullsized":"2024-04-27T00:00:46.397895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:37:39.757654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.398862+0000","last_change":"2024-04-26T23:58:51.159715+0000","last_active":"2024-04-27T00:00:46.398862+0000","last_peered":"2024-04-27T00:00:46.398862+0000","last_clean":"2024-04-27T00:00:46.398862+0000","last_became_active":"2024-04-26T23:58:51.152657+0000","last_became_peered":"2024-04-26T23:58:51.152657+0000","last_unstale":"2024-04-27T00:00:46.398862+0000","last_undegraded":"2024-04-27T00:00:46.398862+0000","last_fullsized":"2024-04-27T00:00:46.398862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:29.210295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.388128+0000","last_change":"2024-04-26T23:58:51.141814+0000","last_active":"2024-04-27T00:00:46.388128+0000","last_peered":"2024-04-27T00:00:46.388128+0000","last_clean":"2024-04-27T00:00:46.388128+0000","last_became_active":"2024-04-26T23:58:51.141344+0000","last_became_peered":"2024-04-26T23:58:51.141344+0000","last_unstale":"2024-04-27T00:00:46.388128+0000","last_undegraded":"2024-04-27T00:00:46.388128+0000","last_fullsized":"2024-04-27T00:00:46.388128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:05:49.532178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382099+0000","last_change":"2024-04-26T23:58:51.170537+0000","last_active":"2024-04-27T00:00:46.382099+0000","last_peered":"2024-04-27T00:00:46.382099+0000","last_clean":"2024-04-27T00:00:46.382099+0000","last_became_active":"2024-04-26T23:58:51.170047+0000","last_became_peered":"2024-04-26T23:58:51.170047+0000","last_unstale":"2024-04-27T00:00:46.382099+0000","last_undegraded":"2024-04-27T00:00:46.382099+0000","last_fullsized":"2024-04-27T00:00:46.382099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:46.463214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.450782+0000","last_change":"2024-04-26T23:58:51.141041+0000","last_active":"2024-04-27T00:00:46.450782+0000","last_peered":"2024-04-27T00:00:46.450782+0000","last_clean":"2024-04-27T00:00:46.450782+0000","last_became_active":"2024-04-26T23:58:51.140709+0000","last_became_peered":"2024-04-26T23:58:51.140709+0000","last_unstale":"2024-04-27T00:00:46.450782+0000","last_undegraded":"2024-04-27T00:00:46.450782+0000","last_fullsized":"2024-04-27T00:00:46.450782+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:12:28.613723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451699+0000","last_change":"2024-04-26T23:58:51.145975+0000","last_active":"2024-04-27T00:00:46.451699+0000","last_peered":"2024-04-27T00:00:46.451699+0000","last_clean":"2024-04-27T00:00:46.451699+0000","last_became_active":"2024-04-26T23:58:51.145594+0000","last_became_peered":"2024-04-26T23:58:51.145594+0000","last_unstale":"2024-04-27T00:00:46.451699+0000","last_undegraded":"2024-04-27T00:00:46.451699+0000","last_fullsized":"2024-04-27T00:00:46.451699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:50:59.548009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.385057+0000","last_change":"2024-04-26T23:58:51.160363+0000","last_active":"2024-04-27T00:00:46.385057+0000","last_peered":"2024-04-27T00:00:46.385057+0000","last_clean":"2024-04-27T00:00:46.385057+0000","last_became_active":"2024-04-26T23:58:51.152004+0000","last_became_peered":"2024-04-26T23:58:51.152004+0000","last_unstale":"2024-04-27T00:00:46.385057+0000","last_undegraded":"2024-04-27T00:00:46.385057+0000","last_fullsized":"2024-04-27T00:00:46.385057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:01:03.203404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.450588+0000","last_change":"2024-04-26T23:58:51.134247+0000","last_active":"2024-04-27T00:00:46.450588+0000","last_peered":"2024-04-27T00:00:46.450588+0000","last_clean":"2024-04-27T00:00:46.450588+0000","last_became_active":"2024-04-26T23:58:51.133889+0000","last_became_peered":"2024-04-26T23:58:51.133889+0000","last_unstale":"2024-04-27T00:00:46.450588+0000","last_undegraded":"2024-04-27T00:00:46.450588+0000","last_fullsized":"2024-04-27T00:00:46.450588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:51:04.245217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451349+0000","last_change":"2024-04-26T23:58:51.135255+0000","last_active":"2024-04-27T00:00:46.451349+0000","last_peered":"2024-04-27T00:00:46.451349+0000","last_clean":"2024-04-27T00:00:46.451349+0000","last_became_active":"2024-04-26T23:58:51.134905+0000","last_became_peered":"2024-04-26T23:58:51.134905+0000","last_unstale":"2024-04-27T00:00:46.451349+0000","last_undegraded":"2024-04-27T00:00:46.451349+0000","last_fullsized":"2024-04-27T00:00:46.451349+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:51.435008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452305+0000","last_change":"2024-04-26T23:58:51.134962+0000","last_active":"2024-04-27T00:00:46.452305+0000","last_peered":"2024-04-27T00:00:46.452305+0000","last_clean":"2024-04-27T00:00:46.452305+0000","last_became_active":"2024-04-26T23:58:51.134559+0000","last_became_peered":"2024-04-26T23:58:51.134559+0000","last_unstale":"2024-04-27T00:00:46.452305+0000","last_undegraded":"2024-04-27T00:00:46.452305+0000","last_fullsized":"2024-04-27T00:00:46.452305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:31:15.833585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387994+0000","last_change":"2024-04-26T23:58:51.141872+0000","last_active":"2024-04-27T00:00:46.387994+0000","last_peered":"2024-04-27T00:00:46.387994+0000","last_clean":"2024-04-27T00:00:46.387994+0000","last_became_active":"2024-04-26T23:58:51.141464+0000","last_became_peered":"2024-04-26T23:58:51.141464+0000","last_unstale":"2024-04-27T00:00:46.387994+0000","last_undegraded":"2024-04-27T00:00:46.387994+0000","last_fullsized":"2024-04-27T00:00:46.387994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:37.663379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.546880+0000","last_change":"2024-04-26T23:58:51.157828+0000","last_active":"2024-04-27T00:00:46.546880+0000","last_peered":"2024-04-27T00:00:46.546880+0000","last_clean":"2024-04-27T00:00:46.546880+0000","last_became_active":"2024-04-26T23:58:51.156820+0000","last_became_peered":"2024-04-26T23:58:51.156820+0000","last_unstale":"2024-04-27T00:00:46.546880+0000","last_undegraded":"2024-04-27T00:00:46.546880+0000","last_fullsized":"2024-04-27T00:00:46.546880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:22:43.201795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.397129+0000","last_change":"2024-04-26T23:58:51.154108+0000","last_active":"2024-04-27T00:00:46.397129+0000","last_peered":"2024-04-27T00:00:46.397129+0000","last_clean":"2024-04-27T00:00:46.397129+0000","last_became_active":"2024-04-26T23:58:51.149740+0000","last_became_peered":"2024-04-26T23:58:51.149740+0000","last_unstale":"2024-04-27T00:00:46.397129+0000","last_undegraded":"2024-04-27T00:00:46.397129+0000","last_fullsized":"2024-04-27T00:00:46.397129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:23:58.519297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.453312+0000","last_change":"2024-04-27T00:00:46.453312+0000","last_active":"2024-04-27T00:00:46.453312+0000","last_peered":"2024-04-27T00:00:46.453312+0000","last_clean":"2024-04-27T00:00:46.453312+0000","last_became_active":"2024-04-27T00:00:46.451293+0000","last_became_peered":"2024-04-27T00:00:46.451293+0000","last_unstale":"2024-04-27T00:00:46.453312+0000","last_undegraded":"2024-04-27T00:00:46.453312+0000","last_fullsized":"2024-04-27T00:00:46.453312+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:51.188145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449540+0000","last_change":"2024-04-26T23:58:51.162811+0000","last_active":"2024-04-27T00:00:46.449540+0000","last_peered":"2024-04-27T00:00:46.449540+0000","last_clean":"2024-04-27T00:00:46.449540+0000","last_became_active":"2024-04-26T23:58:51.162464+0000","last_became_peered":"2024-04-26T23:58:51.162464+0000","last_unstale":"2024-04-27T00:00:46.449540+0000","last_undegraded":"2024-04-27T00:00:46.449540+0000","last_fullsized":"2024-04-27T00:00:46.449540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:50:26.127894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390369+0000","last_change":"2024-04-26T23:58:51.162363+0000","last_active":"2024-04-27T00:00:46.390369+0000","last_peered":"2024-04-27T00:00:46.390369+0000","last_clean":"2024-04-27T00:00:46.390369+0000","last_became_active":"2024-04-26T23:58:51.162033+0000","last_became_peered":"2024-04-26T23:58:51.162033+0000","last_unstale":"2024-04-27T00:00:46.390369+0000","last_undegraded":"2024-04-27T00:00:46.390369+0000","last_fullsized":"2024-04-27T00:00:46.390369+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:53.755338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.392506+0000","last_change":"2024-04-26T23:58:51.147373+0000","last_active":"2024-04-27T00:00:46.392506+0000","last_peered":"2024-04-27T00:00:46.392506+0000","last_clean":"2024-04-27T00:00:46.392506+0000","last_became_active":"2024-04-26T23:58:51.146993+0000","last_became_peered":"2024-04-26T23:58:51.146993+0000","last_unstale":"2024-04-27T00:00:46.392506+0000","last_undegraded":"2024-04-27T00:00:46.392506+0000","last_fullsized":"2024-04-27T00:00:46.392506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:20:14.887719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449262+0000","last_change":"2024-04-26T23:58:51.159553+0000","last_active":"2024-04-27T00:00:46.449262+0000","last_peered":"2024-04-27T00:00:46.449262+0000","last_clean":"2024-04-27T00:00:46.449262+0000","last_became_active":"2024-04-26T23:58:51.159252+0000","last_became_peered":"2024-04-26T23:58:51.159252+0000","last_unstale":"2024-04-27T00:00:46.449262+0000","last_undegraded":"2024-04-27T00:00:46.449262+0000","last_fullsized":"2024-04-27T00:00:46.449262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:56:17.408273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449131+0000","last_change":"2024-04-26T23:58:55.203263+0000","last_active":"2024-04-27T00:00:46.449131+0000","last_peered":"2024-04-27T00:00:46.449131+0000","last_clean":"2024-04-27T00:00:46.449131+0000","last_became_active":"2024-04-26T23:58:55.202832+0000","last_became_peered":"2024-04-26T23:58:55.202832+0000","last_unstale":"2024-04-27T00:00:46.449131+0000","last_undegraded":"2024-04-27T00:00:46.449131+0000","last_fullsized":"2024-04-27T00:00:46.449131+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:13:38.825916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.394167+0000","last_change":"2024-04-26T23:58:50.124003+0000","last_active":"2024-04-27T00:00:46.394167+0000","last_peered":"2024-04-27T00:00:46.394167+0000","last_clean":"2024-04-27T00:00:46.394167+0000","last_became_active":"2024-04-26T23:58:50.123653+0000","last_became_peered":"2024-04-26T23:58:50.123653+0000","last_unstale":"2024-04-27T00:00:46.394167+0000","last_undegraded":"2024-04-27T00:00:46.394167+0000","last_fullsized":"2024-04-27T00:00:46.394167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:16:57.688594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.458581+0000","last_change":"2024-04-27T00:00:46.455582+0000","last_active":"2024-04-27T00:00:46.458581+0000","last_peered":"2024-04-27T00:00:46.458581+0000","last_clean":"2024-04-27T00:00:46.458581+0000","last_became_active":"2024-04-27T00:00:46.451444+0000","last_became_peered":"2024-04-27T00:00:46.451444+0000","last_unstale":"2024-04-27T00:00:46.458581+0000","last_undegraded":"2024-04-27T00:00:46.458581+0000","last_fullsized":"2024-04-27T00:00:46.458581+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:24.173456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.549215+0000","last_change":"2024-04-27T00:00:46.549215+0000","last_active":"2024-04-27T00:00:46.549215+0000","last_peered":"2024-04-27T00:00:46.549215+0000","last_clean":"2024-04-27T00:00:46.549215+0000","last_became_active":"2024-04-27T00:00:46.548787+0000","last_became_peered":"2024-04-27T00:00:46.548787+0000","last_unstale":"2024-04-27T00:00:46.549215+0000","last_undegraded":"2024-04-27T00:00:46.549215+0000","last_fullsized":"2024-04-27T00:00:46.549215+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:35:21.068567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387797+0000","last_change":"2024-04-26T23:58:50.128773+0000","last_active":"2024-04-27T00:00:46.387797+0000","last_peered":"2024-04-27T00:00:46.387797+0000","last_clean":"2024-04-27T00:00:46.387797+0000","last_became_active":"2024-04-26T23:58:50.128445+0000","last_became_peered":"2024-04-26T23:58:50.128445+0000","last_unstale":"2024-04-27T00:00:46.387797+0000","last_undegraded":"2024-04-27T00:00:46.387797+0000","last_fullsized":"2024-04-27T00:00:46.387797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:27:57.577361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.399246+0000","last_change":"2024-04-26T23:58:51.164079+0000","last_active":"2024-04-27T00:00:46.399246+0000","last_peered":"2024-04-27T00:00:46.399246+0000","last_clean":"2024-04-27T00:00:46.399246+0000","last_became_active":"2024-04-26T23:58:51.162514+0000","last_became_peered":"2024-04-26T23:58:51.162514+0000","last_unstale":"2024-04-27T00:00:46.399246+0000","last_undegraded":"2024-04-27T00:00:46.399246+0000","last_fullsized":"2024-04-27T00:00:46.399246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:48:10.904737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387541+0000","last_change":"2024-04-26T23:58:55.211879+0000","last_active":"2024-04-27T00:00:46.387541+0000","last_peered":"2024-04-27T00:00:46.387541+0000","last_clean":"2024-04-27T00:00:46.387541+0000","last_became_active":"2024-04-26T23:58:55.211421+0000","last_became_peered":"2024-04-26T23:58:55.211421+0000","last_unstale":"2024-04-27T00:00:46.387541+0000","last_undegraded":"2024-04-27T00:00:46.387541+0000","last_fullsized":"2024-04-27T00:00:46.387541+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:45:12.754349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382584+0000","last_change":"2024-04-26T23:58:50.101893+0000","last_active":"2024-04-27T00:00:46.382584+0000","last_peered":"2024-04-27T00:00:46.382584+0000","last_clean":"2024-04-27T00:00:46.382584+0000","last_became_active":"2024-04-26T23:58:50.101625+0000","last_became_peered":"2024-04-26T23:58:50.101625+0000","last_unstale":"2024-04-27T00:00:46.382584+0000","last_undegraded":"2024-04-27T00:00:46.382584+0000","last_fullsized":"2024-04-27T00:00:46.382584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:05:24.365710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.450046+0000","last_change":"2024-04-26T23:58:51.146848+0000","last_active":"2024-04-27T00:00:46.450046+0000","last_peered":"2024-04-27T00:00:46.450046+0000","last_clean":"2024-04-27T00:00:46.450046+0000","last_became_active":"2024-04-26T23:58:51.146261+0000","last_became_peered":"2024-04-26T23:58:51.146261+0000","last_unstale":"2024-04-27T00:00:46.450046+0000","last_undegraded":"2024-04-27T00:00:46.450046+0000","last_fullsized":"2024-04-27T00:00:46.450046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:22.200191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387480+0000","last_change":"2024-04-26T23:58:55.196026+0000","last_active":"2024-04-27T00:00:46.387480+0000","last_peered":"2024-04-27T00:00:46.387480+0000","last_clean":"2024-04-27T00:00:46.387480+0000","last_became_active":"2024-04-26T23:58:55.195591+0000","last_became_peered":"2024-04-26T23:58:55.195591+0000","last_unstale":"2024-04-27T00:00:46.387480+0000","last_undegraded":"2024-04-27T00:00:46.387480+0000","last_fullsized":"2024-04-27T00:00:46.387480+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:06.980985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382090+0000","last_change":"2024-04-26T23:58:50.100655+0000","last_active":"2024-04-27T00:00:46.382090+0000","last_peered":"2024-04-27T00:00:46.382090+0000","last_clean":"2024-04-27T00:00:46.382090+0000","last_became_active":"2024-04-26T23:58:50.100314+0000","last_became_peered":"2024-04-26T23:58:50.100314+0000","last_unstale":"2024-04-27T00:00:46.382090+0000","last_undegraded":"2024-04-27T00:00:46.382090+0000","last_fullsized":"2024-04-27T00:00:46.382090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:10:06.230717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.391885+0000","last_change":"2024-04-26T23:58:51.142669+0000","last_active":"2024-04-27T00:00:46.391885+0000","last_peered":"2024-04-27T00:00:46.391885+0000","last_clean":"2024-04-27T00:00:46.391885+0000","last_became_active":"2024-04-26T23:58:51.142337+0000","last_became_peered":"2024-04-26T23:58:51.142337+0000","last_unstale":"2024-04-27T00:00:46.391885+0000","last_undegraded":"2024-04-27T00:00:46.391885+0000","last_fullsized":"2024-04-27T00:00:46.391885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:33:57.351429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449175+0000","last_change":"2024-04-26T23:58:55.184331+0000","last_active":"2024-04-27T00:00:46.449175+0000","last_peered":"2024-04-27T00:00:46.449175+0000","last_clean":"2024-04-27T00:00:46.449175+0000","last_became_active":"2024-04-26T23:58:55.183928+0000","last_became_peered":"2024-04-26T23:58:55.183928+0000","last_unstale":"2024-04-27T00:00:46.449175+0000","last_undegraded":"2024-04-27T00:00:46.449175+0000","last_fullsized":"2024-04-27T00:00:46.449175+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:47.303104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":121,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:49.514811+0000","last_change":"2024-04-26T23:58:50.101084+0000","last_active":"2024-04-27T00:00:49.514811+0000","last_peered":"2024-04-27T00:00:49.514811+0000","last_clean":"2024-04-27T00:00:49.514811+0000","last_became_active":"2024-04-26T23:58:50.100736+0000","last_became_peered":"2024-04-26T23:58:50.100736+0000","last_unstale":"2024-04-27T00:00:49.514811+0000","last_undegraded":"2024-04-27T00:00:49.514811+0000","last_fullsized":"2024-04-27T00:00:49.514811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:20:43.157122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382318+0000","last_change":"2024-04-26T23:58:51.143834+0000","last_active":"2024-04-27T00:00:46.382318+0000","last_peered":"2024-04-27T00:00:46.382318+0000","last_clean":"2024-04-27T00:00:46.382318+0000","last_became_active":"2024-04-26T23:58:51.143480+0000","last_became_peered":"2024-04-26T23:58:51.143480+0000","last_unstale":"2024-04-27T00:00:46.382318+0000","last_undegraded":"2024-04-27T00:00:46.382318+0000","last_fullsized":"2024-04-27T00:00:46.382318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:34:25.474562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449460+0000","last_change":"2024-04-26T23:58:55.215809+0000","last_active":"2024-04-27T00:00:46.449460+0000","last_peered":"2024-04-27T00:00:46.449460+0000","last_clean":"2024-04-27T00:00:46.449460+0000","last_became_active":"2024-04-26T23:58:55.215369+0000","last_became_peered":"2024-04-26T23:58:55.215369+0000","last_unstale":"2024-04-27T00:00:46.449460+0000","last_undegraded":"2024-04-27T00:00:46.449460+0000","last_fullsized":"2024-04-27T00:00:46.449460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:04.053963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.450710+0000","last_change":"2024-04-26T23:58:50.129626+0000","last_active":"2024-04-27T00:00:46.450710+0000","last_peered":"2024-04-27T00:00:46.450710+0000","last_clean":"2024-04-27T00:00:46.450710+0000","last_became_active":"2024-04-26T23:58:50.129386+0000","last_became_peered":"2024-04-26T23:58:50.129386+0000","last_unstale":"2024-04-27T00:00:46.450710+0000","last_undegraded":"2024-04-27T00:00:46.450710+0000","last_fullsized":"2024-04-27T00:00:46.450710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:31:54.432289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.391870+0000","last_change":"2024-04-26T23:58:51.166312+0000","last_active":"2024-04-27T00:00:46.391870+0000","last_peered":"2024-04-27T00:00:46.391870+0000","last_clean":"2024-04-27T00:00:46.391870+0000","last_became_active":"2024-04-26T23:58:51.165986+0000","last_became_peered":"2024-04-26T23:58:51.165986+0000","last_unstale":"2024-04-27T00:00:46.391870+0000","last_undegraded":"2024-04-27T00:00:46.391870+0000","last_fullsized":"2024-04-27T00:00:46.391870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:51:49.549912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390622+0000","last_change":"2024-04-26T23:58:55.204142+0000","last_active":"2024-04-27T00:00:46.390622+0000","last_peered":"2024-04-27T00:00:46.390622+0000","last_clean":"2024-04-27T00:00:46.390622+0000","last_became_active":"2024-04-26T23:58:55.203646+0000","last_became_peered":"2024-04-26T23:58:55.203646+0000","last_unstale":"2024-04-27T00:00:46.390622+0000","last_undegraded":"2024-04-27T00:00:46.390622+0000","last_fullsized":"2024-04-27T00:00:46.390622+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:40.614034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.546304+0000","last_change":"2024-04-26T23:58:50.132080+0000","last_active":"2024-04-27T00:00:46.546304+0000","last_peered":"2024-04-27T00:00:46.546304+0000","last_clean":"2024-04-27T00:00:46.546304+0000","last_became_active":"2024-04-26T23:58:50.131618+0000","last_became_peered":"2024-04-26T23:58:50.131618+0000","last_unstale":"2024-04-27T00:00:46.546304+0000","last_undegraded":"2024-04-27T00:00:46.546304+0000","last_fullsized":"2024-04-27T00:00:46.546304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:42:07.897166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.384995+0000","last_change":"2024-04-26T23:58:51.175083+0000","last_active":"2024-04-27T00:00:46.384995+0000","last_peered":"2024-04-27T00:00:46.384995+0000","last_clean":"2024-04-27T00:00:46.384995+0000","last_became_active":"2024-04-26T23:58:51.174159+0000","last_became_peered":"2024-04-26T23:58:51.174159+0000","last_unstale":"2024-04-27T00:00:46.384995+0000","last_undegraded":"2024-04-27T00:00:46.384995+0000","last_fullsized":"2024-04-27T00:00:46.384995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:51:06.583217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.397791+0000","last_change":"2024-04-26T23:58:55.200019+0000","last_active":"2024-04-27T00:00:46.397791+0000","last_peered":"2024-04-27T00:00:46.397791+0000","last_clean":"2024-04-27T00:00:46.397791+0000","last_became_active":"2024-04-26T23:58:55.199593+0000","last_became_peered":"2024-04-26T23:58:55.199593+0000","last_unstale":"2024-04-27T00:00:46.397791+0000","last_undegraded":"2024-04-27T00:00:46.397791+0000","last_fullsized":"2024-04-27T00:00:46.397791+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:41.465294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.391338+0000","last_change":"2024-04-26T23:58:50.100114+0000","last_active":"2024-04-27T00:00:46.391338+0000","last_peered":"2024-04-27T00:00:46.391338+0000","last_clean":"2024-04-27T00:00:46.391338+0000","last_became_active":"2024-04-26T23:58:50.099762+0000","last_became_peered":"2024-04-26T23:58:50.099762+0000","last_unstale":"2024-04-27T00:00:46.391338+0000","last_undegraded":"2024-04-27T00:00:46.391338+0000","last_fullsized":"2024-04-27T00:00:46.391338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:51:16.855014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.388891+0000","last_change":"2024-04-26T23:58:51.171236+0000","last_active":"2024-04-27T00:00:46.388891+0000","last_peered":"2024-04-27T00:00:46.388891+0000","last_clean":"2024-04-27T00:00:46.388891+0000","last_became_active":"2024-04-26T23:58:51.170709+0000","last_became_peered":"2024-04-26T23:58:51.170709+0000","last_unstale":"2024-04-27T00:00:46.388891+0000","last_undegraded":"2024-04-27T00:00:46.388891+0000","last_fullsized":"2024-04-27T00:00:46.388891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:11.404827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.385669+0000","last_change":"2024-04-26T23:58:55.201931+0000","last_active":"2024-04-27T00:00:46.385669+0000","last_peered":"2024-04-27T00:00:46.385669+0000","last_clean":"2024-04-27T00:00:46.385669+0000","last_became_active":"2024-04-26T23:58:55.196810+0000","last_became_peered":"2024-04-26T23:58:55.196810+0000","last_unstale":"2024-04-27T00:00:46.385669+0000","last_undegraded":"2024-04-27T00:00:46.385669+0000","last_fullsized":"2024-04-27T00:00:46.385669+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:04:39.775990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":115,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:49.514058+0000","last_change":"2024-04-26T23:58:50.126477+0000","last_active":"2024-04-27T00:00:49.514058+0000","last_peered":"2024-04-27T00:00:49.514058+0000","last_clean":"2024-04-27T00:00:49.514058+0000","last_became_active":"2024-04-26T23:58:50.126205+0000","last_became_peered":"2024-04-26T23:58:50.126205+0000","last_unstale":"2024-04-27T00:00:49.514058+0000","last_undegraded":"2024-04-27T00:00:49.514058+0000","last_fullsized":"2024-04-27T00:00:49.514058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:03:18.513036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.450467+0000","last_change":"2024-04-26T23:58:51.145782+0000","last_active":"2024-04-27T00:00:46.450467+0000","last_peered":"2024-04-27T00:00:46.450467+0000","last_clean":"2024-04-27T00:00:46.450467+0000","last_became_active":"2024-04-26T23:58:51.145344+0000","last_became_peered":"2024-04-26T23:58:51.145344+0000","last_unstale":"2024-04-27T00:00:46.450467+0000","last_undegraded":"2024-04-27T00:00:46.450467+0000","last_fullsized":"2024-04-27T00:00:46.450467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:32.062690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.389168+0000","last_change":"2024-04-26T23:58:55.198953+0000","last_active":"2024-04-27T00:00:46.389168+0000","last_peered":"2024-04-27T00:00:46.389168+0000","last_clean":"2024-04-27T00:00:46.389168+0000","last_became_active":"2024-04-26T23:58:55.198560+0000","last_became_peered":"2024-04-26T23:58:55.198560+0000","last_unstale":"2024-04-27T00:00:46.389168+0000","last_undegraded":"2024-04-27T00:00:46.389168+0000","last_fullsized":"2024-04-27T00:00:46.389168+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:57:07.109698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452448+0000","last_change":"2024-04-26T23:58:50.103489+0000","last_active":"2024-04-27T00:00:46.452448+0000","last_peered":"2024-04-27T00:00:46.452448+0000","last_clean":"2024-04-27T00:00:46.452448+0000","last_became_active":"2024-04-26T23:58:50.103068+0000","last_became_peered":"2024-04-26T23:58:50.103068+0000","last_unstale":"2024-04-27T00:00:46.452448+0000","last_undegraded":"2024-04-27T00:00:46.452448+0000","last_fullsized":"2024-04-27T00:00:46.452448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:22:26.490816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.399359+0000","last_change":"2024-04-26T23:58:51.163680+0000","last_active":"2024-04-27T00:00:46.399359+0000","last_peered":"2024-04-27T00:00:46.399359+0000","last_clean":"2024-04-27T00:00:46.399359+0000","last_became_active":"2024-04-26T23:58:51.158135+0000","last_became_peered":"2024-04-26T23:58:51.158135+0000","last_unstale":"2024-04-27T00:00:46.399359+0000","last_undegraded":"2024-04-27T00:00:46.399359+0000","last_fullsized":"2024-04-27T00:00:46.399359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:56.562260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452223+0000","last_change":"2024-04-27T00:00:46.452223+0000","last_active":"2024-04-27T00:00:46.452223+0000","last_peered":"2024-04-27T00:00:46.452223+0000","last_clean":"2024-04-27T00:00:46.452223+0000","last_became_active":"2024-04-27T00:00:46.451715+0000","last_became_peered":"2024-04-27T00:00:46.451715+0000","last_unstale":"2024-04-27T00:00:46.452223+0000","last_undegraded":"2024-04-27T00:00:46.452223+0000","last_fullsized":"2024-04-27T00:00:46.452223+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:59:28.551969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.391725+0000","last_change":"2024-04-26T23:58:50.116030+0000","last_active":"2024-04-27T00:00:46.391725+0000","last_peered":"2024-04-27T00:00:46.391725+0000","last_clean":"2024-04-27T00:00:46.391725+0000","last_became_active":"2024-04-26T23:58:50.115744+0000","last_became_peered":"2024-04-26T23:58:50.115744+0000","last_unstale":"2024-04-27T00:00:46.391725+0000","last_undegraded":"2024-04-27T00:00:46.391725+0000","last_fullsized":"2024-04-27T00:00:46.391725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:05:21.823842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.450301+0000","last_change":"2024-04-26T23:58:51.140652+0000","last_active":"2024-04-27T00:00:46.450301+0000","last_peered":"2024-04-27T00:00:46.450301+0000","last_clean":"2024-04-27T00:00:46.450301+0000","last_became_active":"2024-04-26T23:58:51.140330+0000","last_became_peered":"2024-04-26T23:58:51.140330+0000","last_unstale":"2024-04-27T00:00:46.450301+0000","last_undegraded":"2024-04-27T00:00:46.450301+0000","last_fullsized":"2024-04-27T00:00:46.450301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:59:56.037856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.455049+0000","last_change":"2024-04-27T00:00:46.455049+0000","last_active":"2024-04-27T00:00:46.455049+0000","last_peered":"2024-04-27T00:00:46.455049+0000","last_clean":"2024-04-27T00:00:46.455049+0000","last_became_active":"2024-04-27T00:00:46.452657+0000","last_became_peered":"2024-04-27T00:00:46.452657+0000","last_unstale":"2024-04-27T00:00:46.455049+0000","last_undegraded":"2024-04-27T00:00:46.455049+0000","last_fullsized":"2024-04-27T00:00:46.455049+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:42.755193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'32","reported_seq":238,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:50.525496+0000","last_change":"2024-04-26T23:58:50.101850+0000","last_active":"2024-04-27T00:00:50.525496+0000","last_peered":"2024-04-27T00:00:50.525496+0000","last_clean":"2024-04-27T00:00:50.525496+0000","last_became_active":"2024-04-26T23:58:50.101521+0000","last_became_peered":"2024-04-26T23:58:50.101521+0000","last_unstale":"2024-04-27T00:00:50.525496+0000","last_undegraded":"2024-04-27T00:00:50.525496+0000","last_fullsized":"2024-04-27T00:00:50.525496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:06.810644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55676,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":180,"num_read_kb":144,"num_write":33,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382478+0000","last_change":"2024-04-26T23:58:51.143441+0000","last_active":"2024-04-27T00:00:46.382478+0000","last_peered":"2024-04-27T00:00:46.382478+0000","last_clean":"2024-04-27T00:00:46.382478+0000","last_became_active":"2024-04-26T23:58:51.141779+0000","last_became_peered":"2024-04-26T23:58:51.141779+0000","last_unstale":"2024-04-27T00:00:46.382478+0000","last_undegraded":"2024-04-27T00:00:46.382478+0000","last_fullsized":"2024-04-27T00:00:46.382478+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:56:28.067685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.392155+0000","last_change":"2024-04-26T23:58:55.207277+0000","last_active":"2024-04-27T00:00:46.392155+0000","last_peered":"2024-04-27T00:00:46.392155+0000","last_clean":"2024-04-27T00:00:46.392155+0000","last_became_active":"2024-04-26T23:58:55.206905+0000","last_became_peered":"2024-04-26T23:58:55.206905+0000","last_unstale":"2024-04-27T00:00:46.392155+0000","last_undegraded":"2024-04-27T00:00:46.392155+0000","last_fullsized":"2024-04-27T00:00:46.392155+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:03:34.254752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449233+0000","last_change":"2024-04-26T23:58:50.115642+0000","last_active":"2024-04-27T00:00:46.449233+0000","last_peered":"2024-04-27T00:00:46.449233+0000","last_clean":"2024-04-27T00:00:46.449233+0000","last_became_active":"2024-04-26T23:58:50.115310+0000","last_became_peered":"2024-04-26T23:58:50.115310+0000","last_unstale":"2024-04-27T00:00:46.449233+0000","last_undegraded":"2024-04-27T00:00:46.449233+0000","last_fullsized":"2024-04-27T00:00:46.449233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:33.189230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.546588+0000","last_change":"2024-04-26T23:58:51.159397+0000","last_active":"2024-04-27T00:00:46.546588+0000","last_peered":"2024-04-27T00:00:46.546588+0000","last_clean":"2024-04-27T00:00:46.546588+0000","last_became_active":"2024-04-26T23:58:51.159054+0000","last_became_peered":"2024-04-26T23:58:51.159054+0000","last_unstale":"2024-04-27T00:00:46.546588+0000","last_undegraded":"2024-04-27T00:00:46.546588+0000","last_fullsized":"2024-04-27T00:00:46.546588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:54:13.306198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390398+0000","last_change":"2024-04-26T23:58:55.215794+0000","last_active":"2024-04-27T00:00:46.390398+0000","last_peered":"2024-04-27T00:00:46.390398+0000","last_clean":"2024-04-27T00:00:46.390398+0000","last_became_active":"2024-04-26T23:58:55.215366+0000","last_became_peered":"2024-04-26T23:58:55.215366+0000","last_unstale":"2024-04-27T00:00:46.390398+0000","last_undegraded":"2024-04-27T00:00:46.390398+0000","last_fullsized":"2024-04-27T00:00:46.390398+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:54:20.347552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.391887+0000","last_change":"2024-04-26T23:58:50.112207+0000","last_active":"2024-04-27T00:00:46.391887+0000","last_peered":"2024-04-27T00:00:46.391887+0000","last_clean":"2024-04-27T00:00:46.391887+0000","last_became_active":"2024-04-26T23:58:50.111975+0000","last_became_peered":"2024-04-26T23:58:50.111975+0000","last_unstale":"2024-04-27T00:00:46.391887+0000","last_undegraded":"2024-04-27T00:00:46.391887+0000","last_fullsized":"2024-04-27T00:00:46.391887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:29.091698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452067+0000","last_change":"2024-04-26T23:58:51.147154+0000","last_active":"2024-04-27T00:00:46.452067+0000","last_peered":"2024-04-27T00:00:46.452067+0000","last_clean":"2024-04-27T00:00:46.452067+0000","last_became_active":"2024-04-26T23:58:51.146809+0000","last_became_peered":"2024-04-26T23:58:51.146809+0000","last_unstale":"2024-04-27T00:00:46.452067+0000","last_undegraded":"2024-04-27T00:00:46.452067+0000","last_fullsized":"2024-04-27T00:00:46.452067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:22.308365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.393397+0000","last_change":"2024-04-26T23:58:55.214461+0000","last_active":"2024-04-27T00:00:46.393397+0000","last_peered":"2024-04-27T00:00:46.393397+0000","last_clean":"2024-04-27T00:00:46.393397+0000","last_became_active":"2024-04-26T23:58:55.213442+0000","last_became_peered":"2024-04-26T23:58:55.213442+0000","last_unstale":"2024-04-27T00:00:46.393397+0000","last_undegraded":"2024-04-27T00:00:46.393397+0000","last_fullsized":"2024-04-27T00:00:46.393397+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:07:03.114274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":119,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:49.579398+0000","last_change":"2024-04-26T23:58:50.100554+0000","last_active":"2024-04-27T00:00:49.579398+0000","last_peered":"2024-04-27T00:00:49.579398+0000","last_clean":"2024-04-27T00:00:49.579398+0000","last_became_active":"2024-04-26T23:58:50.100200+0000","last_became_peered":"2024-04-26T23:58:50.100200+0000","last_unstale":"2024-04-27T00:00:49.579398+0000","last_undegraded":"2024-04-27T00:00:49.579398+0000","last_fullsized":"2024-04-27T00:00:49.579398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:33:59.913424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451594+0000","last_change":"2024-04-26T23:58:51.141030+0000","last_active":"2024-04-27T00:00:46.451594+0000","last_peered":"2024-04-27T00:00:46.451594+0000","last_clean":"2024-04-27T00:00:46.451594+0000","last_became_active":"2024-04-26T23:58:51.140694+0000","last_became_peered":"2024-04-26T23:58:51.140694+0000","last_unstale":"2024-04-27T00:00:46.451594+0000","last_undegraded":"2024-04-27T00:00:46.451594+0000","last_fullsized":"2024-04-27T00:00:46.451594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:35:29.827020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.547621+0000","last_change":"2024-04-26T23:58:55.216556+0000","last_active":"2024-04-27T00:00:46.547621+0000","last_peered":"2024-04-27T00:00:46.547621+0000","last_clean":"2024-04-27T00:00:46.547621+0000","last_became_active":"2024-04-26T23:58:55.215798+0000","last_became_peered":"2024-04-26T23:58:55.215798+0000","last_unstale":"2024-04-27T00:00:46.547621+0000","last_undegraded":"2024-04-27T00:00:46.547621+0000","last_fullsized":"2024-04-27T00:00:46.547621+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:32:08.733655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.381919+0000","last_change":"2024-04-26T23:58:50.119495+0000","last_active":"2024-04-27T00:00:46.381919+0000","last_peered":"2024-04-27T00:00:46.381919+0000","last_clean":"2024-04-27T00:00:46.381919+0000","last_became_active":"2024-04-26T23:58:50.119174+0000","last_became_peered":"2024-04-26T23:58:50.119174+0000","last_unstale":"2024-04-27T00:00:46.381919+0000","last_undegraded":"2024-04-27T00:00:46.381919+0000","last_fullsized":"2024-04-27T00:00:46.381919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:53:23.408578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.391428+0000","last_change":"2024-04-26T23:58:51.141943+0000","last_active":"2024-04-27T00:00:46.391428+0000","last_peered":"2024-04-27T00:00:46.391428+0000","last_clean":"2024-04-27T00:00:46.391428+0000","last_became_active":"2024-04-26T23:58:51.141624+0000","last_became_peered":"2024-04-26T23:58:51.141624+0000","last_unstale":"2024-04-27T00:00:46.391428+0000","last_undegraded":"2024-04-27T00:00:46.391428+0000","last_fullsized":"2024-04-27T00:00:46.391428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:46:25.114059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.391485+0000","last_change":"2024-04-26T23:58:55.200247+0000","last_active":"2024-04-27T00:00:46.391485+0000","last_peered":"2024-04-27T00:00:46.391485+0000","last_clean":"2024-04-27T00:00:46.391485+0000","last_became_active":"2024-04-26T23:58:55.199854+0000","last_became_peered":"2024-04-26T23:58:55.199854+0000","last_unstale":"2024-04-27T00:00:46.391485+0000","last_undegraded":"2024-04-27T00:00:46.391485+0000","last_fullsized":"2024-04-27T00:00:46.391485+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:22:00.066549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382481+0000","last_change":"2024-04-26T23:58:50.102901+0000","last_active":"2024-04-27T00:00:46.382481+0000","last_peered":"2024-04-27T00:00:46.382481+0000","last_clean":"2024-04-27T00:00:46.382481+0000","last_became_active":"2024-04-26T23:58:50.102537+0000","last_became_peered":"2024-04-26T23:58:50.102537+0000","last_unstale":"2024-04-27T00:00:46.382481+0000","last_undegraded":"2024-04-27T00:00:46.382481+0000","last_fullsized":"2024-04-27T00:00:46.382481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:21:15.043327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.549175+0000","last_change":"2024-04-27T00:00:46.549175+0000","last_active":"2024-04-27T00:00:46.549175+0000","last_peered":"2024-04-27T00:00:46.549175+0000","last_clean":"2024-04-27T00:00:46.549175+0000","last_became_active":"2024-04-27T00:00:46.548691+0000","last_became_peered":"2024-04-27T00:00:46.548691+0000","last_unstale":"2024-04-27T00:00:46.549175+0000","last_undegraded":"2024-04-27T00:00:46.549175+0000","last_fullsized":"2024-04-27T00:00:46.549175+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:58:22.860610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382182+0000","last_change":"2024-04-26T23:58:55.203555+0000","last_active":"2024-04-27T00:00:46.382182+0000","last_peered":"2024-04-27T00:00:46.382182+0000","last_clean":"2024-04-27T00:00:46.382182+0000","last_became_active":"2024-04-26T23:58:55.202968+0000","last_became_peered":"2024-04-26T23:58:55.202968+0000","last_unstale":"2024-04-27T00:00:46.382182+0000","last_undegraded":"2024-04-27T00:00:46.382182+0000","last_fullsized":"2024-04-27T00:00:46.382182+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:01.149966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.392482+0000","last_change":"2024-04-26T23:58:50.129000+0000","last_active":"2024-04-27T00:00:46.392482+0000","last_peered":"2024-04-27T00:00:46.392482+0000","last_clean":"2024-04-27T00:00:46.392482+0000","last_became_active":"2024-04-26T23:58:50.128646+0000","last_became_peered":"2024-04-26T23:58:50.128646+0000","last_unstale":"2024-04-27T00:00:46.392482+0000","last_undegraded":"2024-04-27T00:00:46.392482+0000","last_fullsized":"2024-04-27T00:00:46.392482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:34.200578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.450930+0000","last_change":"2024-04-26T23:58:51.159557+0000","last_active":"2024-04-27T00:00:46.450930+0000","last_peered":"2024-04-27T00:00:46.450930+0000","last_clean":"2024-04-27T00:00:46.450930+0000","last_became_active":"2024-04-26T23:58:51.159211+0000","last_became_peered":"2024-04-26T23:58:51.159211+0000","last_unstale":"2024-04-27T00:00:46.450930+0000","last_undegraded":"2024-04-27T00:00:46.450930+0000","last_fullsized":"2024-04-27T00:00:46.450930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:48.740612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.546446+0000","last_change":"2024-04-26T23:58:55.193585+0000","last_active":"2024-04-27T00:00:46.546446+0000","last_peered":"2024-04-27T00:00:46.546446+0000","last_clean":"2024-04-27T00:00:46.546446+0000","last_became_active":"2024-04-26T23:58:55.193031+0000","last_became_peered":"2024-04-26T23:58:55.193031+0000","last_unstale":"2024-04-27T00:00:46.546446+0000","last_undegraded":"2024-04-27T00:00:46.546446+0000","last_fullsized":"2024-04-27T00:00:46.546446+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:20.689117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451772+0000","last_change":"2024-04-26T23:58:50.124301+0000","last_active":"2024-04-27T00:00:46.451772+0000","last_peered":"2024-04-27T00:00:46.451772+0000","last_clean":"2024-04-27T00:00:46.451772+0000","last_became_active":"2024-04-26T23:58:50.124016+0000","last_became_peered":"2024-04-26T23:58:50.124016+0000","last_unstale":"2024-04-27T00:00:46.451772+0000","last_undegraded":"2024-04-27T00:00:46.451772+0000","last_fullsized":"2024-04-27T00:00:46.451772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:00.835698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.450520+0000","last_change":"2024-04-26T23:58:51.164703+0000","last_active":"2024-04-27T00:00:46.450520+0000","last_peered":"2024-04-27T00:00:46.450520+0000","last_clean":"2024-04-27T00:00:46.450520+0000","last_became_active":"2024-04-26T23:58:51.164346+0000","last_became_peered":"2024-04-26T23:58:51.164346+0000","last_unstale":"2024-04-27T00:00:46.450520+0000","last_undegraded":"2024-04-27T00:00:46.450520+0000","last_fullsized":"2024-04-27T00:00:46.450520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:45:39.667034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.547474+0000","last_change":"2024-04-26T23:58:55.193795+0000","last_active":"2024-04-27T00:00:46.547474+0000","last_peered":"2024-04-27T00:00:46.547474+0000","last_clean":"2024-04-27T00:00:46.547474+0000","last_became_active":"2024-04-26T23:58:55.193323+0000","last_became_peered":"2024-04-26T23:58:55.193323+0000","last_unstale":"2024-04-27T00:00:46.547474+0000","last_undegraded":"2024-04-27T00:00:46.547474+0000","last_fullsized":"2024-04-27T00:00:46.547474+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:44.301311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.455477+0000","last_change":"2024-04-27T00:00:46.453286+0000","last_active":"2024-04-27T00:00:46.455477+0000","last_peered":"2024-04-27T00:00:46.455477+0000","last_clean":"2024-04-27T00:00:46.455477+0000","last_became_active":"2024-04-27T00:00:46.451159+0000","last_became_peered":"2024-04-27T00:00:46.451159+0000","last_unstale":"2024-04-27T00:00:46.455477+0000","last_undegraded":"2024-04-27T00:00:46.455477+0000","last_fullsized":"2024-04-27T00:00:46.455477+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:24.265297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":116,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:49.578338+0000","last_change":"2024-04-26T23:58:50.132099+0000","last_active":"2024-04-27T00:00:49.578338+0000","last_peered":"2024-04-27T00:00:49.578338+0000","last_clean":"2024-04-27T00:00:49.578338+0000","last_became_active":"2024-04-26T23:58:50.131712+0000","last_became_peered":"2024-04-26T23:58:50.131712+0000","last_unstale":"2024-04-27T00:00:49.578338+0000","last_undegraded":"2024-04-27T00:00:49.578338+0000","last_fullsized":"2024-04-27T00:00:49.578338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:40.849259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452179+0000","last_change":"2024-04-26T23:58:55.205614+0000","last_active":"2024-04-27T00:00:46.452179+0000","last_peered":"2024-04-27T00:00:46.452179+0000","last_clean":"2024-04-27T00:00:46.452179+0000","last_became_active":"2024-04-26T23:58:55.205206+0000","last_became_peered":"2024-04-26T23:58:55.205206+0000","last_unstale":"2024-04-27T00:00:46.452179+0000","last_undegraded":"2024-04-27T00:00:46.452179+0000","last_fullsized":"2024-04-27T00:00:46.452179+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:42:24.324563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.385799+0000","last_change":"2024-04-26T23:58:51.159808+0000","last_active":"2024-04-27T00:00:46.385799+0000","last_peered":"2024-04-27T00:00:46.385799+0000","last_clean":"2024-04-27T00:00:46.385799+0000","last_became_active":"2024-04-26T23:58:51.159444+0000","last_became_peered":"2024-04-26T23:58:51.159444+0000","last_unstale":"2024-04-27T00:00:46.385799+0000","last_undegraded":"2024-04-27T00:00:46.385799+0000","last_fullsized":"2024-04-27T00:00:46.385799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:14:57.401134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387605+0000","last_change":"2024-04-26T23:58:50.120100+0000","last_active":"2024-04-27T00:00:46.387605+0000","last_peered":"2024-04-27T00:00:46.387605+0000","last_clean":"2024-04-27T00:00:46.387605+0000","last_became_active":"2024-04-26T23:58:50.119613+0000","last_became_peered":"2024-04-26T23:58:50.119613+0000","last_unstale":"2024-04-27T00:00:46.387605+0000","last_undegraded":"2024-04-27T00:00:46.387605+0000","last_fullsized":"2024-04-27T00:00:46.387605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:32:18.116454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":348,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451438+0000","last_change":"2024-04-26T23:57:30.173979+0000","last_active":"2024-04-27T00:00:46.451438+0000","last_peered":"2024-04-27T00:00:46.451438+0000","last_clean":"2024-04-27T00:00:46.451438+0000","last_became_active":"2024-04-26T23:57:30.173549+0000","last_became_peered":"2024-04-26T23:57:30.173549+0000","last_unstale":"2024-04-27T00:00:46.451438+0000","last_undegraded":"2024-04-27T00:00:46.451438+0000","last_fullsized":"2024-04-27T00:00:46.451438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:53:33.634094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:53:33.634094+0000","last_clean_scrub_stamp":"2024-04-26T23:53:33.634094+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:48.026677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.549720+0000","last_change":"2024-04-27T00:00:46.549720+0000","last_active":"2024-04-27T00:00:46.549720+0000","last_peered":"2024-04-27T00:00:46.549720+0000","last_clean":"2024-04-27T00:00:46.549720+0000","last_became_active":"2024-04-27T00:00:46.549099+0000","last_became_peered":"2024-04-27T00:00:46.549099+0000","last_unstale":"2024-04-27T00:00:46.549720+0000","last_undegraded":"2024-04-27T00:00:46.549720+0000","last_fullsized":"2024-04-27T00:00:46.549720+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:34.196039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.545772+0000","last_change":"2024-04-26T23:58:51.152687+0000","last_active":"2024-04-27T00:00:46.545772+0000","last_peered":"2024-04-27T00:00:46.545772+0000","last_clean":"2024-04-27T00:00:46.545772+0000","last_became_active":"2024-04-26T23:58:51.149082+0000","last_became_peered":"2024-04-26T23:58:51.149082+0000","last_unstale":"2024-04-27T00:00:46.545772+0000","last_undegraded":"2024-04-27T00:00:46.545772+0000","last_fullsized":"2024-04-27T00:00:46.545772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:38:42.630214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451590+0000","last_change":"2024-04-26T23:58:50.104485+0000","last_active":"2024-04-27T00:00:46.451590+0000","last_peered":"2024-04-27T00:00:46.451590+0000","last_clean":"2024-04-27T00:00:46.451590+0000","last_became_active":"2024-04-26T23:58:50.104260+0000","last_became_peered":"2024-04-26T23:58:50.104260+0000","last_unstale":"2024-04-27T00:00:46.451590+0000","last_undegraded":"2024-04-27T00:00:46.451590+0000","last_fullsized":"2024-04-27T00:00:46.451590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:04:06.268993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390116+0000","last_change":"2024-04-26T23:58:55.190219+0000","last_active":"2024-04-27T00:00:46.390116+0000","last_peered":"2024-04-27T00:00:46.390116+0000","last_clean":"2024-04-27T00:00:46.390116+0000","last_became_active":"2024-04-26T23:58:55.189801+0000","last_became_peered":"2024-04-26T23:58:55.189801+0000","last_unstale":"2024-04-27T00:00:46.390116+0000","last_undegraded":"2024-04-27T00:00:46.390116+0000","last_fullsized":"2024-04-27T00:00:46.390116+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:53:42.170710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382415+0000","last_change":"2024-04-26T23:58:50.117442+0000","last_active":"2024-04-27T00:00:46.382415+0000","last_peered":"2024-04-27T00:00:46.382415+0000","last_clean":"2024-04-27T00:00:46.382415+0000","last_became_active":"2024-04-26T23:58:50.117112+0000","last_became_peered":"2024-04-26T23:58:50.117112+0000","last_unstale":"2024-04-27T00:00:46.382415+0000","last_undegraded":"2024-04-27T00:00:46.382415+0000","last_fullsized":"2024-04-27T00:00:46.382415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:18.352145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.547234+0000","last_change":"2024-04-26T23:58:51.156670+0000","last_active":"2024-04-27T00:00:46.547234+0000","last_peered":"2024-04-27T00:00:46.547234+0000","last_clean":"2024-04-27T00:00:46.547234+0000","last_became_active":"2024-04-26T23:58:51.155596+0000","last_became_peered":"2024-04-26T23:58:51.155596+0000","last_unstale":"2024-04-27T00:00:46.547234+0000","last_undegraded":"2024-04-27T00:00:46.547234+0000","last_fullsized":"2024-04-27T00:00:46.547234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:38.053814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.384716+0000","last_change":"2024-04-26T23:58:55.201644+0000","last_active":"2024-04-27T00:00:46.384716+0000","last_peered":"2024-04-27T00:00:46.384716+0000","last_clean":"2024-04-27T00:00:46.384716+0000","last_became_active":"2024-04-26T23:58:55.196421+0000","last_became_peered":"2024-04-26T23:58:55.196421+0000","last_unstale":"2024-04-27T00:00:46.384716+0000","last_undegraded":"2024-04-27T00:00:46.384716+0000","last_fullsized":"2024-04-27T00:00:46.384716+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:26:30.309773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'10","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.392707+0000","last_change":"2024-04-26T23:58:50.120805+0000","last_active":"2024-04-27T00:00:46.392707+0000","last_peered":"2024-04-27T00:00:46.392707+0000","last_clean":"2024-04-27T00:00:46.392707+0000","last_became_active":"2024-04-26T23:58:50.120403+0000","last_became_peered":"2024-04-26T23:58:50.120403+0000","last_unstale":"2024-04-27T00:00:46.392707+0000","last_undegraded":"2024-04-27T00:00:46.392707+0000","last_fullsized":"2024-04-27T00:00:46.392707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:53:28.950313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.386608+0000","last_change":"2024-04-26T23:58:51.167109+0000","last_active":"2024-04-27T00:00:46.386608+0000","last_peered":"2024-04-27T00:00:46.386608+0000","last_clean":"2024-04-27T00:00:46.386608+0000","last_became_active":"2024-04-26T23:58:51.164584+0000","last_became_peered":"2024-04-26T23:58:51.164584+0000","last_unstale":"2024-04-27T00:00:46.386608+0000","last_undegraded":"2024-04-27T00:00:46.386608+0000","last_fullsized":"2024-04-27T00:00:46.386608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:12:12.160553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.448812+0000","last_change":"2024-04-26T23:58:55.183138+0000","last_active":"2024-04-27T00:00:46.448812+0000","last_peered":"2024-04-27T00:00:46.448812+0000","last_clean":"2024-04-27T00:00:46.448812+0000","last_became_active":"2024-04-26T23:58:55.182779+0000","last_became_peered":"2024-04-26T23:58:55.182779+0000","last_unstale":"2024-04-27T00:00:46.448812+0000","last_undegraded":"2024-04-27T00:00:46.448812+0000","last_fullsized":"2024-04-27T00:00:46.448812+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:02:16.866145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.388141+0000","last_change":"2024-04-26T23:58:50.120401+0000","last_active":"2024-04-27T00:00:46.388141+0000","last_peered":"2024-04-27T00:00:46.388141+0000","last_clean":"2024-04-27T00:00:46.388141+0000","last_became_active":"2024-04-26T23:58:50.120036+0000","last_became_peered":"2024-04-26T23:58:50.120036+0000","last_unstale":"2024-04-27T00:00:46.388141+0000","last_undegraded":"2024-04-27T00:00:46.388141+0000","last_fullsized":"2024-04-27T00:00:46.388141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:49.852298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.455636+0000","last_change":"2024-04-27T00:00:46.455636+0000","last_active":"2024-04-27T00:00:46.455636+0000","last_peered":"2024-04-27T00:00:46.455636+0000","last_clean":"2024-04-27T00:00:46.455636+0000","last_became_active":"2024-04-27T00:00:46.454493+0000","last_became_peered":"2024-04-27T00:00:46.454493+0000","last_unstale":"2024-04-27T00:00:46.455636+0000","last_undegraded":"2024-04-27T00:00:46.455636+0000","last_fullsized":"2024-04-27T00:00:46.455636+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:25:57.077168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387997+0000","last_change":"2024-04-26T23:58:55.216273+0000","last_active":"2024-04-27T00:00:46.387997+0000","last_peered":"2024-04-27T00:00:46.387997+0000","last_clean":"2024-04-27T00:00:46.387997+0000","last_became_active":"2024-04-26T23:58:55.215630+0000","last_became_peered":"2024-04-26T23:58:55.215630+0000","last_unstale":"2024-04-27T00:00:46.387997+0000","last_undegraded":"2024-04-27T00:00:46.387997+0000","last_fullsized":"2024-04-27T00:00:46.387997+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:15:41.195389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449630+0000","last_change":"2024-04-26T23:58:50.111216+0000","last_active":"2024-04-27T00:00:46.449630+0000","last_peered":"2024-04-27T00:00:46.449630+0000","last_clean":"2024-04-27T00:00:46.449630+0000","last_became_active":"2024-04-26T23:58:50.110969+0000","last_became_peered":"2024-04-26T23:58:50.110969+0000","last_unstale":"2024-04-27T00:00:46.449630+0000","last_undegraded":"2024-04-27T00:00:46.449630+0000","last_fullsized":"2024-04-27T00:00:46.449630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:01.319448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.381941+0000","last_change":"2024-04-26T23:58:51.132643+0000","last_active":"2024-04-27T00:00:46.381941+0000","last_peered":"2024-04-27T00:00:46.381941+0000","last_clean":"2024-04-27T00:00:46.381941+0000","last_became_active":"2024-04-26T23:58:51.132250+0000","last_became_peered":"2024-04-26T23:58:51.132250+0000","last_unstale":"2024-04-27T00:00:46.381941+0000","last_undegraded":"2024-04-27T00:00:46.381941+0000","last_fullsized":"2024-04-27T00:00:46.381941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:00:11.180178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.393865+0000","last_change":"2024-04-26T23:58:55.204718+0000","last_active":"2024-04-27T00:00:46.393865+0000","last_peered":"2024-04-27T00:00:46.393865+0000","last_clean":"2024-04-27T00:00:46.393865+0000","last_became_active":"2024-04-26T23:58:55.204159+0000","last_became_peered":"2024-04-26T23:58:55.204159+0000","last_unstale":"2024-04-27T00:00:46.393865+0000","last_undegraded":"2024-04-27T00:00:46.393865+0000","last_fullsized":"2024-04-27T00:00:46.393865+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:42:52.199200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.384857+0000","last_change":"2024-04-26T23:58:50.126926+0000","last_active":"2024-04-27T00:00:46.384857+0000","last_peered":"2024-04-27T00:00:46.384857+0000","last_clean":"2024-04-27T00:00:46.384857+0000","last_became_active":"2024-04-26T23:58:50.126552+0000","last_became_peered":"2024-04-26T23:58:50.126552+0000","last_unstale":"2024-04-27T00:00:46.384857+0000","last_undegraded":"2024-04-27T00:00:46.384857+0000","last_fullsized":"2024-04-27T00:00:46.384857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:18:25.024805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.389942+0000","last_change":"2024-04-26T23:58:51.162722+0000","last_active":"2024-04-27T00:00:46.389942+0000","last_peered":"2024-04-27T00:00:46.389942+0000","last_clean":"2024-04-27T00:00:46.389942+0000","last_became_active":"2024-04-26T23:58:51.162387+0000","last_became_peered":"2024-04-26T23:58:51.162387+0000","last_unstale":"2024-04-27T00:00:46.389942+0000","last_undegraded":"2024-04-27T00:00:46.389942+0000","last_fullsized":"2024-04-27T00:00:46.389942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:07.143478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387842+0000","last_change":"2024-04-26T23:58:55.216201+0000","last_active":"2024-04-27T00:00:46.387842+0000","last_peered":"2024-04-27T00:00:46.387842+0000","last_clean":"2024-04-27T00:00:46.387842+0000","last_became_active":"2024-04-26T23:58:55.215730+0000","last_became_peered":"2024-04-26T23:58:55.215730+0000","last_unstale":"2024-04-27T00:00:46.387842+0000","last_undegraded":"2024-04-27T00:00:46.387842+0000","last_fullsized":"2024-04-27T00:00:46.387842+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:21.484879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449195+0000","last_change":"2024-04-26T23:58:50.111313+0000","last_active":"2024-04-27T00:00:46.449195+0000","last_peered":"2024-04-27T00:00:46.449195+0000","last_clean":"2024-04-27T00:00:46.449195+0000","last_became_active":"2024-04-26T23:58:50.110937+0000","last_became_peered":"2024-04-26T23:58:50.110937+0000","last_unstale":"2024-04-27T00:00:46.449195+0000","last_undegraded":"2024-04-27T00:00:46.449195+0000","last_fullsized":"2024-04-27T00:00:46.449195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:52:24.615527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.386058+0000","last_change":"2024-04-26T23:58:51.174152+0000","last_active":"2024-04-27T00:00:46.386058+0000","last_peered":"2024-04-27T00:00:46.386058+0000","last_clean":"2024-04-27T00:00:46.386058+0000","last_became_active":"2024-04-26T23:58:51.170956+0000","last_became_peered":"2024-04-26T23:58:51.170956+0000","last_unstale":"2024-04-27T00:00:46.386058+0000","last_undegraded":"2024-04-27T00:00:46.386058+0000","last_fullsized":"2024-04-27T00:00:46.386058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:09:09.253242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387993+0000","last_change":"2024-04-26T23:58:55.206248+0000","last_active":"2024-04-27T00:00:46.387993+0000","last_peered":"2024-04-27T00:00:46.387993+0000","last_clean":"2024-04-27T00:00:46.387993+0000","last_became_active":"2024-04-26T23:58:55.205820+0000","last_became_peered":"2024-04-26T23:58:55.205820+0000","last_unstale":"2024-04-27T00:00:46.387993+0000","last_undegraded":"2024-04-27T00:00:46.387993+0000","last_fullsized":"2024-04-27T00:00:46.387993+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:52:24.789706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.382089+0000","last_change":"2024-04-26T23:58:50.103874+0000","last_active":"2024-04-27T00:00:46.382089+0000","last_peered":"2024-04-27T00:00:46.382089+0000","last_clean":"2024-04-27T00:00:46.382089+0000","last_became_active":"2024-04-26T23:58:50.103590+0000","last_became_peered":"2024-04-26T23:58:50.103590+0000","last_unstale":"2024-04-27T00:00:46.382089+0000","last_undegraded":"2024-04-27T00:00:46.382089+0000","last_fullsized":"2024-04-27T00:00:46.382089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:34:59.817736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451886+0000","last_change":"2024-04-26T23:58:51.146396+0000","last_active":"2024-04-27T00:00:46.451886+0000","last_peered":"2024-04-27T00:00:46.451886+0000","last_clean":"2024-04-27T00:00:46.451886+0000","last_became_active":"2024-04-26T23:58:51.146046+0000","last_became_peered":"2024-04-26T23:58:51.146046+0000","last_unstale":"2024-04-27T00:00:46.451886+0000","last_undegraded":"2024-04-27T00:00:46.451886+0000","last_fullsized":"2024-04-27T00:00:46.451886+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:14.230111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449790+0000","last_change":"2024-04-26T23:58:55.199320+0000","last_active":"2024-04-27T00:00:46.449790+0000","last_peered":"2024-04-27T00:00:46.449790+0000","last_clean":"2024-04-27T00:00:46.449790+0000","last_became_active":"2024-04-26T23:58:55.198843+0000","last_became_peered":"2024-04-26T23:58:55.198843+0000","last_unstale":"2024-04-27T00:00:46.449790+0000","last_undegraded":"2024-04-27T00:00:46.449790+0000","last_fullsized":"2024-04-27T00:00:46.449790+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:56:54.643998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449645+0000","last_change":"2024-04-26T23:58:50.120892+0000","last_active":"2024-04-27T00:00:46.449645+0000","last_peered":"2024-04-27T00:00:46.449645+0000","last_clean":"2024-04-27T00:00:46.449645+0000","last_became_active":"2024-04-26T23:58:50.120396+0000","last_became_peered":"2024-04-26T23:58:50.120396+0000","last_unstale":"2024-04-27T00:00:46.449645+0000","last_undegraded":"2024-04-27T00:00:46.449645+0000","last_fullsized":"2024-04-27T00:00:46.449645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:16.670482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452329+0000","last_change":"2024-04-26T23:58:51.134917+0000","last_active":"2024-04-27T00:00:46.452329+0000","last_peered":"2024-04-27T00:00:46.452329+0000","last_clean":"2024-04-27T00:00:46.452329+0000","last_became_active":"2024-04-26T23:58:51.134470+0000","last_became_peered":"2024-04-26T23:58:51.134470+0000","last_unstale":"2024-04-27T00:00:46.452329+0000","last_undegraded":"2024-04-27T00:00:46.452329+0000","last_fullsized":"2024-04-27T00:00:46.452329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:03.696427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.398029+0000","last_change":"2024-04-26T23:58:55.213966+0000","last_active":"2024-04-27T00:00:46.398029+0000","last_peered":"2024-04-27T00:00:46.398029+0000","last_clean":"2024-04-27T00:00:46.398029+0000","last_became_active":"2024-04-26T23:58:55.213521+0000","last_became_peered":"2024-04-26T23:58:55.213521+0000","last_unstale":"2024-04-27T00:00:46.398029+0000","last_undegraded":"2024-04-27T00:00:46.398029+0000","last_fullsized":"2024-04-27T00:00:46.398029+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:10:02.998037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.392035+0000","last_change":"2024-04-26T23:58:50.118136+0000","last_active":"2024-04-27T00:00:46.392035+0000","last_peered":"2024-04-27T00:00:46.392035+0000","last_clean":"2024-04-27T00:00:46.392035+0000","last_became_active":"2024-04-26T23:58:50.117787+0000","last_became_peered":"2024-04-26T23:58:50.117787+0000","last_unstale":"2024-04-27T00:00:46.392035+0000","last_undegraded":"2024-04-27T00:00:46.392035+0000","last_fullsized":"2024-04-27T00:00:46.392035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:14.145327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.451956+0000","last_change":"2024-04-26T23:58:51.166491+0000","last_active":"2024-04-27T00:00:46.451956+0000","last_peered":"2024-04-27T00:00:46.451956+0000","last_clean":"2024-04-27T00:00:46.451956+0000","last_became_active":"2024-04-26T23:58:51.166106+0000","last_became_peered":"2024-04-26T23:58:51.166106+0000","last_unstale":"2024-04-27T00:00:46.451956+0000","last_undegraded":"2024-04-27T00:00:46.451956+0000","last_fullsized":"2024-04-27T00:00:46.451956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:18:58.690731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.391190+0000","last_change":"2024-04-26T23:58:55.184623+0000","last_active":"2024-04-27T00:00:46.391190+0000","last_peered":"2024-04-27T00:00:46.391190+0000","last_clean":"2024-04-27T00:00:46.391190+0000","last_became_active":"2024-04-26T23:58:55.184029+0000","last_became_peered":"2024-04-26T23:58:55.184029+0000","last_unstale":"2024-04-27T00:00:46.391190+0000","last_undegraded":"2024-04-27T00:00:46.391190+0000","last_fullsized":"2024-04-27T00:00:46.391190+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:54.156152+0000","last_clean_scrub_stamp":"2024-04-26T23:58:54.156152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:39:36.267352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.547443+0000","last_change":"2024-04-26T23:58:50.127630+0000","last_active":"2024-04-27T00:00:46.547443+0000","last_peered":"2024-04-27T00:00:46.547443+0000","last_clean":"2024-04-27T00:00:46.547443+0000","last_became_active":"2024-04-26T23:58:50.127134+0000","last_became_peered":"2024-04-26T23:58:50.127134+0000","last_unstale":"2024-04-27T00:00:46.547443+0000","last_undegraded":"2024-04-27T00:00:46.547443+0000","last_fullsized":"2024-04-27T00:00:46.547443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:49.076266+0000","last_clean_scrub_stamp":"2024-04-26T23:58:49.076266+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:15.061919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.452430+0000","last_change":"2024-04-26T23:58:51.134241+0000","last_active":"2024-04-27T00:00:46.452430+0000","last_peered":"2024-04-27T00:00:46.452430+0000","last_clean":"2024-04-27T00:00:46.452430+0000","last_became_active":"2024-04-26T23:58:51.133867+0000","last_became_peered":"2024-04-26T23:58:51.133867+0000","last_unstale":"2024-04-27T00:00:46.452430+0000","last_undegraded":"2024-04-27T00:00:46.452430+0000","last_fullsized":"2024-04-27T00:00:46.452430+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:12:20.171560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390031+0000","last_change":"2024-04-26T23:58:51.132681+0000","last_active":"2024-04-27T00:00:46.390031+0000","last_peered":"2024-04-27T00:00:46.390031+0000","last_clean":"2024-04-27T00:00:46.390031+0000","last_became_active":"2024-04-26T23:58:51.132213+0000","last_became_peered":"2024-04-26T23:58:51.132213+0000","last_unstale":"2024-04-27T00:00:46.390031+0000","last_undegraded":"2024-04-27T00:00:46.390031+0000","last_fullsized":"2024-04-27T00:00:46.390031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:26:52.080670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449455+0000","last_change":"2024-04-26T23:58:51.135731+0000","last_active":"2024-04-27T00:00:46.449455+0000","last_peered":"2024-04-27T00:00:46.449455+0000","last_clean":"2024-04-27T00:00:46.449455+0000","last_became_active":"2024-04-26T23:58:51.135352+0000","last_became_peered":"2024-04-26T23:58:51.135352+0000","last_unstale":"2024-04-27T00:00:46.449455+0000","last_undegraded":"2024-04-27T00:00:46.449455+0000","last_fullsized":"2024-04-27T00:00:46.449455+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:45:37.903396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449005+0000","last_change":"2024-04-26T23:58:51.160014+0000","last_active":"2024-04-27T00:00:46.449005+0000","last_peered":"2024-04-27T00:00:46.449005+0000","last_clean":"2024-04-27T00:00:46.449005+0000","last_became_active":"2024-04-26T23:58:51.159459+0000","last_became_peered":"2024-04-26T23:58:51.159459+0000","last_unstale":"2024-04-27T00:00:46.449005+0000","last_undegraded":"2024-04-27T00:00:46.449005+0000","last_fullsized":"2024-04-27T00:00:46.449005+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:12.757558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.398357+0000","last_change":"2024-04-26T23:58:51.166942+0000","last_active":"2024-04-27T00:00:46.398357+0000","last_peered":"2024-04-27T00:00:46.398357+0000","last_clean":"2024-04-27T00:00:46.398357+0000","last_became_active":"2024-04-26T23:58:51.162605+0000","last_became_peered":"2024-04-26T23:58:51.162605+0000","last_unstale":"2024-04-27T00:00:46.398357+0000","last_undegraded":"2024-04-27T00:00:46.398357+0000","last_fullsized":"2024-04-27T00:00:46.398357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:38:46.346098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.547279+0000","last_change":"2024-04-26T23:58:51.162638+0000","last_active":"2024-04-27T00:00:46.547279+0000","last_peered":"2024-04-27T00:00:46.547279+0000","last_clean":"2024-04-27T00:00:46.547279+0000","last_became_active":"2024-04-26T23:58:51.162165+0000","last_became_peered":"2024-04-26T23:58:51.162165+0000","last_unstale":"2024-04-27T00:00:46.547279+0000","last_undegraded":"2024-04-27T00:00:46.547279+0000","last_fullsized":"2024-04-27T00:00:46.547279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:26:32.277183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390359+0000","last_change":"2024-04-26T23:58:51.132667+0000","last_active":"2024-04-27T00:00:46.390359+0000","last_peered":"2024-04-27T00:00:46.390359+0000","last_clean":"2024-04-27T00:00:46.390359+0000","last_became_active":"2024-04-26T23:58:51.132316+0000","last_became_peered":"2024-04-26T23:58:51.132316+0000","last_unstale":"2024-04-27T00:00:46.390359+0000","last_undegraded":"2024-04-27T00:00:46.390359+0000","last_fullsized":"2024-04-27T00:00:46.390359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:41.427650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449600+0000","last_change":"2024-04-26T23:58:51.132614+0000","last_active":"2024-04-27T00:00:46.449600+0000","last_peered":"2024-04-27T00:00:46.449600+0000","last_clean":"2024-04-27T00:00:46.449600+0000","last_became_active":"2024-04-26T23:58:51.132274+0000","last_became_peered":"2024-04-26T23:58:51.132274+0000","last_unstale":"2024-04-27T00:00:46.449600+0000","last_undegraded":"2024-04-27T00:00:46.449600+0000","last_fullsized":"2024-04-27T00:00:46.449600+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:52.810401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.387183+0000","last_change":"2024-04-26T23:58:51.165470+0000","last_active":"2024-04-27T00:00:46.387183+0000","last_peered":"2024-04-27T00:00:46.387183+0000","last_clean":"2024-04-27T00:00:46.387183+0000","last_became_active":"2024-04-26T23:58:51.164998+0000","last_became_peered":"2024-04-26T23:58:51.164998+0000","last_unstale":"2024-04-27T00:00:46.387183+0000","last_undegraded":"2024-04-27T00:00:46.387183+0000","last_fullsized":"2024-04-27T00:00:46.387183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:02.703215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.391345+0000","last_change":"2024-04-26T23:58:51.166390+0000","last_active":"2024-04-27T00:00:46.391345+0000","last_peered":"2024-04-27T00:00:46.391345+0000","last_clean":"2024-04-27T00:00:46.391345+0000","last_became_active":"2024-04-26T23:58:51.166113+0000","last_became_peered":"2024-04-26T23:58:51.166113+0000","last_unstale":"2024-04-27T00:00:46.391345+0000","last_undegraded":"2024-04-27T00:00:46.391345+0000","last_fullsized":"2024-04-27T00:00:46.391345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:24:49.676207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.385295+0000","last_change":"2024-04-26T23:58:51.165272+0000","last_active":"2024-04-27T00:00:46.385295+0000","last_peered":"2024-04-27T00:00:46.385295+0000","last_clean":"2024-04-27T00:00:46.385295+0000","last_became_active":"2024-04-26T23:58:51.160355+0000","last_became_peered":"2024-04-26T23:58:51.160355+0000","last_unstale":"2024-04-27T00:00:46.385295+0000","last_undegraded":"2024-04-27T00:00:46.385295+0000","last_fullsized":"2024-04-27T00:00:46.385295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:32.910129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.449946+0000","last_change":"2024-04-26T23:58:51.159564+0000","last_active":"2024-04-27T00:00:46.449946+0000","last_peered":"2024-04-27T00:00:46.449946+0000","last_clean":"2024-04-27T00:00:46.449946+0000","last_became_active":"2024-04-26T23:58:51.159229+0000","last_became_peered":"2024-04-26T23:58:51.159229+0000","last_unstale":"2024-04-27T00:00:46.449946+0000","last_undegraded":"2024-04-27T00:00:46.449946+0000","last_fullsized":"2024-04-27T00:00:46.449946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:39.032595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.546453+0000","last_change":"2024-04-26T23:58:51.163868+0000","last_active":"2024-04-27T00:00:46.546453+0000","last_peered":"2024-04-27T00:00:46.546453+0000","last_clean":"2024-04-27T00:00:46.546453+0000","last_became_active":"2024-04-26T23:58:51.163450+0000","last_became_peered":"2024-04-26T23:58:51.163450+0000","last_unstale":"2024-04-27T00:00:46.546453+0000","last_undegraded":"2024-04-27T00:00:46.546453+0000","last_fullsized":"2024-04-27T00:00:46.546453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:42:06.945605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-27T00:00:46.390356+0000","last_change":"2024-04-26T23:58:51.136047+0000","last_active":"2024-04-27T00:00:46.390356+0000","last_peered":"2024-04-27T00:00:46.390356+0000","last_clean":"2024-04-27T00:00:46.390356+0000","last_became_active":"2024-04-26T23:58:51.135690+0000","last_became_peered":"2024-04-26T23:58:51.135690+0000","last_unstale":"2024-04-27T00:00:46.390356+0000","last_undegraded":"2024-04-27T00:00:46.390356+0000","last_fullsized":"2024-04-27T00:00:46.390356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:58:50.082876+0000","last_clean_scrub_stamp":"2024-04-26T23:58:50.082876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:55.721858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":60068,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":544,"num_read_kb":336,"num_write":166,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":180204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142,"ondisk_log_size":142,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351015,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":503795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612651,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28700,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716452,"statfs":{"total":95995035648,"available":95965646848,"internally_reserved":0,"allocated":1716224,"data_stored":1147130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776177,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1052672,"data_stored":508127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037816,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":503951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299452,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1642496,"data_stored":1100251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462980,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":499207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1089536,"data_stored":554793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1052672,"data_stored":500595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1114112,"data_stored":556534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":499401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888144,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":504248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1650688,"data_stored":1095448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":55676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":57417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":57645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T00:00:54.561 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-27T00:00:54.561 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-27T00:00:54.561 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-27T00:00:54.561 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-27T00:00:54.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:54 smithi040 ceph-mon[28095]: pgmap v386: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 7 op/s 2024-04-27T00:00:54.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:54 smithi080 ceph-mon[32362]: pgmap v386: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 7 op/s 2024-04-27T00:00:54.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:54 smithi103 ceph-mon[32952]: pgmap v386: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 7 op/s 2024-04-27T00:00:55.062 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:00:55.543 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:55 smithi040 ceph-mon[28095]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T00:00:55.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:55 smithi080 ceph-mon[32362]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T00:00:55.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:55 smithi103 ceph-mon[32952]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T00:00:56.649 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:56 smithi040 ceph-mon[28095]: pgmap v387: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 109 B/s wr, 6 op/s 2024-04-27T00:00:56.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:56 smithi080 ceph-mon[32362]: pgmap v387: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 109 B/s wr, 6 op/s 2024-04-27T00:00:56.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:56 smithi103 ceph-mon[32952]: pgmap v387: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 109 B/s wr, 6 op/s 2024-04-27T00:00:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.164+0000 7f9954e00700 1 -- 172.21.15.40:0/3378240506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9950103470 msgr2=0x7f99501038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.164+0000 7f9954e00700 1 --2- 172.21.15.40:0/3378240506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9950103470 0x7f99501038f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9938009990 tx=0x7f9938009ca0 comp rx=0 tx=0).stop 2024-04-27T00:00:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.164+0000 7f9954e00700 1 -- 172.21.15.40:0/3378240506 shutdown_connections 2024-04-27T00:00:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.164+0000 7f9954e00700 1 --2- 172.21.15.40:0/3378240506 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9950103e30 0x7f9950110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.164+0000 7f9954e00700 1 --2- 172.21.15.40:0/3378240506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99501094c0 0x7f99501098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.164+0000 7f9954e00700 1 --2- 172.21.15.40:0/3378240506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9950103470 0x7f99501038f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.164+0000 7f9954e00700 1 -- 172.21.15.40:0/3378240506 >> 172.21.15.40:0/3378240506 conn(0x7f995007a580 msgr2=0x7f995007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.164+0000 7f9954e00700 1 -- 172.21.15.40:0/3378240506 shutdown_connections 2024-04-27T00:00:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.164+0000 7f9954e00700 1 -- 172.21.15.40:0/3378240506 wait complete. 2024-04-27T00:00:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.165+0000 7f9954e00700 1 Processor -- start 2024-04-27T00:00:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.165+0000 7f9954e00700 1 -- start start 2024-04-27T00:00:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.166+0000 7f9954e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9950103470 0x7f995019bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.166+0000 7f9954e00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9950103e30 0x7f995019c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.166+0000 7f9954e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99501094c0 0x7f99501a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.166+0000 7f9954e00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9950113640 con 0x7f9950103470 2024-04-27T00:00:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.166+0000 7f9954e00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f99501134c0 con 0x7f9950103e30 2024-04-27T00:00:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.166+0000 7f9954e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f99501137c0 con 0x7f99501094c0 2024-04-27T00:00:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.166+0000 7f994de00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9950103470 0x7f995019bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.166+0000 7f994de00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9950103470 0x7f995019bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50046/0 (socket says 172.21.15.40:50046) 2024-04-27T00:00:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.166+0000 7f994de00700 1 -- 172.21.15.40:0/3697163936 learned_addr learned my addr 172.21.15.40:0/3697163936 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.166+0000 7f994e800700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99501094c0 0x7f99501a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.166+0000 7f9947e00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9950103e30 0x7f995019c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.167+0000 7f994de00700 1 -- 172.21.15.40:0/3697163936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99501094c0 msgr2=0x7f99501a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.167+0000 7f994de00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99501094c0 0x7f99501a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.167+0000 7f994de00700 1 -- 172.21.15.40:0/3697163936 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9950103e30 msgr2=0x7f995019c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.167+0000 7f994de00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9950103e30 0x7f995019c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.167+0000 7f994de00700 1 -- 172.21.15.40:0/3697163936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99501a0da0 con 0x7f9950103470 2024-04-27T00:00:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.167+0000 7f994e800700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99501094c0 0x7f99501a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.167+0000 7f994de00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9950103470 0x7f995019bd90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f994000e9a0 tx=0x7f994000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.167+0000 7f9946a00700 1 -- 172.21.15.40:0/3697163936 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994000ca10 con 0x7f9950103470 2024-04-27T00:00:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.167+0000 7f9946a00700 1 -- 172.21.15.40:0/3697163936 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f994000cbb0 con 0x7f9950103470 2024-04-27T00:00:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.167+0000 7f9954e00700 1 -- 172.21.15.40:0/3697163936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99501a1090 con 0x7f9950103470 2024-04-27T00:00:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.168+0000 7f9946a00700 1 -- 172.21.15.40:0/3697163936 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9940018680 con 0x7f9950103470 2024-04-27T00:00:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.168+0000 7f9954e00700 1 -- 172.21.15.40:0/3697163936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99501a8910 con 0x7f9950103470 2024-04-27T00:00:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.169+0000 7f9954e00700 1 -- 172.21.15.40:0/3697163936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f995006a110 con 0x7f9950103470 2024-04-27T00:00:57.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.171+0000 7f9946a00700 1 -- 172.21.15.40:0/3697163936 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9940010430 con 0x7f9950103470 2024-04-27T00:00:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.173+0000 7f9946a00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f99340a3cd0 0x7f99340a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.173+0000 7f9946a00700 1 -- 172.21.15.40:0/3697163936 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f99400d0d10 con 0x7f9950103470 2024-04-27T00:00:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.173+0000 7f9947e00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f99340a3cd0 0x7f99340a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.174+0000 7f9947e00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f99340a3cd0 0x7f99340a6190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f99501106d0 tx=0x7f9938005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.175+0000 7f9946a00700 1 -- 172.21.15.40:0/3697163936 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f994009d1f0 con 0x7f9950103470 2024-04-27T00:00:57.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.459+0000 7f9954e00700 1 -- 172.21.15.40:0/3697163936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f995019d0f0 con 0x7f9950103470 2024-04-27T00:00:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.461+0000 7f9946a00700 1 -- 172.21.15.40:0/3697163936 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f994009caf0 con 0x7f9950103470 2024-04-27T00:00:57.462 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:00:57.463 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-27T00:00:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.465+0000 7f9954e00700 1 -- 172.21.15.40:0/3697163936 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f99340a3cd0 msgr2=0x7f99340a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.465+0000 7f9954e00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f99340a3cd0 0x7f99340a6190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f99501106d0 tx=0x7f9938005fd0 comp rx=0 tx=0).stop 2024-04-27T00:00:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.465+0000 7f9954e00700 1 -- 172.21.15.40:0/3697163936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9950103470 msgr2=0x7f995019bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:57.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.465+0000 7f9954e00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9950103470 0x7f995019bd90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f994000e9a0 tx=0x7f994000ecb0 comp rx=0 tx=0).stop 2024-04-27T00:00:57.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.465+0000 7f9954e00700 1 -- 172.21.15.40:0/3697163936 shutdown_connections 2024-04-27T00:00:57.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.465+0000 7f9954e00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f99340a3cd0 0x7f99340a6190 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:57.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.465+0000 7f9954e00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99501094c0 0x7f99501a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:57.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.465+0000 7f9954e00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9950103470 0x7f995019bd90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:57.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.465+0000 7f9954e00700 1 --2- 172.21.15.40:0/3697163936 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9950103e30 0x7f995019c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:57.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.465+0000 7f9954e00700 1 -- 172.21.15.40:0/3697163936 >> 172.21.15.40:0/3697163936 conn(0x7f995007a580 msgr2=0x7f9950101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:57.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.466+0000 7f9954e00700 1 -- 172.21.15.40:0/3697163936 shutdown_connections 2024-04-27T00:00:57.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:57.466+0000 7f9954e00700 1 -- 172.21.15.40:0/3697163936 wait complete. 2024-04-27T00:00:58.332 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-27T00:00:58.333 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T00:00:58.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.554+0000 7f878e400640 1 -- 172.21.15.40:0/1424185431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8788106300 msgr2=0x7f8788106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:58.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.554+0000 7f878e400640 1 --2- 172.21.15.40:0/1424185431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8788106300 0x7f8788106780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8780009900 tx=0x7f878002a3f0 comp rx=0 tx=0).stop 2024-04-27T00:00:58.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.554+0000 7f878e400640 1 -- 172.21.15.40:0/1424185431 shutdown_connections 2024-04-27T00:00:58.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.554+0000 7f878e400640 1 --2- 172.21.15.40:0/1424185431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8788106300 0x7f8788106780 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:58.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.554+0000 7f878e400640 1 --2- 172.21.15.40:0/1424185431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8788103950 0x7f8788105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:58.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.554+0000 7f878e400640 1 --2- 172.21.15.40:0/1424185431 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8788100fe0 0x7f8788103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:58.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.554+0000 7f878e400640 1 -- 172.21.15.40:0/1424185431 >> 172.21.15.40:0/1424185431 conn(0x7f87880fcbc0 msgr2=0x7f87880ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:58.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.554+0000 7f878e400640 1 -- 172.21.15.40:0/1424185431 shutdown_connections 2024-04-27T00:00:58.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.554+0000 7f878e400640 1 -- 172.21.15.40:0/1424185431 wait complete. 2024-04-27T00:00:58.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.555+0000 7f878e400640 1 Processor -- start 2024-04-27T00:00:58.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.556+0000 7f878e400640 1 -- start start 2024-04-27T00:00:58.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.556+0000 7f878e400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8788100fe0 0x7f878819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:58.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.556+0000 7f878e400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8788103950 0x7f878819f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:58.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.556+0000 7f878e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8788106300 0x7f87881a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:58.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.556+0000 7f878e400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87881148e0 con 0x7f8788106300 2024-04-27T00:00:58.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.556+0000 7f878e400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f8788114760 con 0x7f8788103950 2024-04-27T00:00:58.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.556+0000 7f878e400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8788114a60 con 0x7f8788100fe0 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.556+0000 7f878d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8788100fe0 0x7f878819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.556+0000 7f878d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8788100fe0 0x7f878819eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:44474/0 (socket says 172.21.15.40:44474) 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.556+0000 7f878d000640 1 -- 172.21.15.40:0/1884959695 learned_addr learned my addr 172.21.15.40:0/1884959695 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.556+0000 7f878da00640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8788106300 0x7f87881a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.556+0000 7f8787e00640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8788103950 0x7f878819f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.557+0000 7f878d000640 1 -- 172.21.15.40:0/1884959695 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8788103950 msgr2=0x7f878819f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.557+0000 7f878d000640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8788103950 0x7f878819f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.557+0000 7f878d000640 1 -- 172.21.15.40:0/1884959695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8788106300 msgr2=0x7f87881a6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.557+0000 7f878d000640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8788106300 0x7f87881a6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.557+0000 7f878d000640 1 -- 172.21.15.40:0/1884959695 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87881a6ae0 con 0x7f8788100fe0 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.557+0000 7f878d000640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8788100fe0 0x7f878819eec0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f877800eb90 tx=0x7f877800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.557+0000 7f8785600640 1 -- 172.21.15.40:0/1884959695 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8778004d20 con 0x7f8788100fe0 2024-04-27T00:00:58.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.557+0000 7f8785600640 1 -- 172.21.15.40:0/1884959695 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f877800ee90 con 0x7f8788100fe0 2024-04-27T00:00:58.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.557+0000 7f878e400640 1 -- 172.21.15.40:0/1884959695 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87881a6dd0 con 0x7f8788100fe0 2024-04-27T00:00:58.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.557+0000 7f8785600640 1 -- 172.21.15.40:0/1884959695 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8778010670 con 0x7f8788100fe0 2024-04-27T00:00:58.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.557+0000 7f878e400640 1 -- 172.21.15.40:0/1884959695 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87881a7230 con 0x7f8788100fe0 2024-04-27T00:00:58.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.558+0000 7f878e400640 1 -- 172.21.15.40:0/1884959695 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87880fe6b0 con 0x7f8788100fe0 2024-04-27T00:00:58.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.560+0000 7f8785600640 1 -- 172.21.15.40:0/1884959695 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8778005020 con 0x7f8788100fe0 2024-04-27T00:00:58.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.562+0000 7f8785600640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f876c0a3cc0 0x7f876c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:58.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.562+0000 7f8785600640 1 -- 172.21.15.40:0/1884959695 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f87780d0f80 con 0x7f8788100fe0 2024-04-27T00:00:58.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.562+0000 7f8787e00640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f876c0a3cc0 0x7f876c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:58.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.563+0000 7f8787e00640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f876c0a3cc0 0x7f876c0a6180 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f87881a03d0 tx=0x7f8770005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:58.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.566+0000 7f8785600640 1 -- 172.21.15.40:0/1884959695 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f877809d460 con 0x7f8788100fe0 2024-04-27T00:00:58.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:58 smithi040 ceph-mon[28095]: pgmap v388: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 98 B/s wr, 6 op/s 2024-04-27T00:00:58.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:58 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3697163936' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T00:00:58.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.778+0000 7f878e400640 1 -- 172.21.15.40:0/1884959695 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f878802dd40 con 0x7f8788100fe0 2024-04-27T00:00:58.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.781+0000 7f8785600640 1 -- 172.21.15.40:0/1884959695 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24531 (secure 0 0 0) 0x7f877809cd60 con 0x7f8788100fe0 2024-04-27T00:00:58.782 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:00:58.783 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-27T00:00:46.365303+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9},{"from":5,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T00:00:58.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.783+0000 7f878e400640 1 -- 172.21.15.40:0/1884959695 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f876c0a3cc0 msgr2=0x7f876c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:58.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.783+0000 7f878e400640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f876c0a3cc0 0x7f876c0a6180 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f87881a03d0 tx=0x7f8770005f90 comp rx=0 tx=0).stop 2024-04-27T00:00:58.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.783+0000 7f878e400640 1 -- 172.21.15.40:0/1884959695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8788100fe0 msgr2=0x7f878819eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:58.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.783+0000 7f878e400640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8788100fe0 0x7f878819eec0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f877800eb90 tx=0x7f877800c6a0 comp rx=0 tx=0).stop 2024-04-27T00:00:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.784+0000 7f878e400640 1 -- 172.21.15.40:0/1884959695 shutdown_connections 2024-04-27T00:00:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.784+0000 7f878e400640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8788106300 0x7f87881a6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.784+0000 7f878e400640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8788103950 0x7f878819f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.784+0000 7f878e400640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f876c0a3cc0 0x7f876c0a6180 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.784+0000 7f878e400640 1 --2- 172.21.15.40:0/1884959695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8788100fe0 0x7f878819eec0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.784+0000 7f878e400640 1 -- 172.21.15.40:0/1884959695 >> 172.21.15.40:0/1884959695 conn(0x7f87880fcbc0 msgr2=0x7f8788101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.784+0000 7f878e400640 1 -- 172.21.15.40:0/1884959695 shutdown_connections 2024-04-27T00:00:58.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:58.784+0000 7f878e400640 1 -- 172.21.15.40:0/1884959695 wait complete. 2024-04-27T00:00:58.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:58 smithi080 ceph-mon[32362]: pgmap v388: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 98 B/s wr, 6 op/s 2024-04-27T00:00:58.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:58 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3697163936' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T00:00:58.806 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:53:32.708818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:58:48.506601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:58:49.540141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:58:53.179820+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-27T00:00:58.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-27T00:00:58.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:58 smithi103 ceph-mon[32952]: pgmap v388: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 98 B/s wr, 6 op/s 2024-04-27T00:00:58.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:58 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3697163936' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T00:00:59.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.027+0000 7f4bd3e00640 1 -- 172.21.15.40:0/2941139376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bd4109fc0 msgr2=0x7f4bd410a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.027+0000 7f4bd3e00640 1 --2- 172.21.15.40:0/2941139376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bd4109fc0 0x7f4bd410a440 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009900 tx=0x7f4bbc02a3f0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.027+0000 7f4bd3e00640 1 -- 172.21.15.40:0/2941139376 shutdown_connections 2024-04-27T00:00:59.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.027+0000 7f4bd3e00640 1 --2- 172.21.15.40:0/2941139376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd410a980 0x7f4bd41112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.027+0000 7f4bd3e00640 1 --2- 172.21.15.40:0/2941139376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bd4109fc0 0x7f4bd410a440 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.027+0000 7f4bd3e00640 1 --2- 172.21.15.40:0/2941139376 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd4108e70 0x7f4bd4109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.027+0000 7f4bd3e00640 1 -- 172.21.15.40:0/2941139376 >> 172.21.15.40:0/2941139376 conn(0x7f4bd4077e00 msgr2=0x7f4bd407a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:59.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.027+0000 7f4bd3e00640 1 -- 172.21.15.40:0/2941139376 shutdown_connections 2024-04-27T00:00:59.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.027+0000 7f4bd3e00640 1 -- 172.21.15.40:0/2941139376 wait complete. 2024-04-27T00:00:59.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.028+0000 7f4bd3e00640 1 Processor -- start 2024-04-27T00:00:59.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.028+0000 7f4bd3e00640 1 -- start start 2024-04-27T00:00:59.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.029+0000 7f4bd3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bd4108e70 0x7f4bd419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:59.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.029+0000 7f4bd3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd4109fc0 0x7f4bd419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:59.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.029+0000 7f4bd3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd410a980 0x7f4bd41a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:59.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.029+0000 7f4bd3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd4114a20 con 0x7f4bd4108e70 2024-04-27T00:00:59.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.029+0000 7f4bd3e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd41148a0 con 0x7f4bd4109fc0 2024-04-27T00:00:59.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.029+0000 7f4bd3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd4114ba0 con 0x7f4bd410a980 2024-04-27T00:00:59.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.029+0000 7f4bd2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bd4108e70 0x7f4bd419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:59.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.029+0000 7f4bd2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bd4108e70 0x7f4bd419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50092/0 (socket says 172.21.15.40:50092) 2024-04-27T00:00:59.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.029+0000 7f4bd2a00640 1 -- 172.21.15.40:0/669517400 learned_addr learned my addr 172.21.15.40:0/669517400 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:59.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.029+0000 7f4bd3400640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd410a980 0x7f4bd41a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:59.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.029+0000 7f4bcbe00640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd4109fc0 0x7f4bd419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:59.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.029+0000 7f4bd2a00640 1 -- 172.21.15.40:0/669517400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd410a980 msgr2=0x7f4bd41a6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.030+0000 7f4bd2a00640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd410a980 0x7f4bd41a6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.030+0000 7f4bd2a00640 1 -- 172.21.15.40:0/669517400 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd4109fc0 msgr2=0x7f4bd419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.030+0000 7f4bd2a00640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd4109fc0 0x7f4bd419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.030+0000 7f4bd2a00640 1 -- 172.21.15.40:0/669517400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd41a6c90 con 0x7f4bd4108e70 2024-04-27T00:00:59.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.030+0000 7f4bd3400640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd410a980 0x7f4bd41a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:59.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.030+0000 7f4bd2a00640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bd4108e70 0x7f4bd419f070 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400ea60 tx=0x7f4bc400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:59.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.030+0000 7f4bcb400640 1 -- 172.21.15.40:0/669517400 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc400cde0 con 0x7f4bd4108e70 2024-04-27T00:00:59.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.030+0000 7f4bcb400640 1 -- 172.21.15.40:0/669517400 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc40035f0 con 0x7f4bd4108e70 2024-04-27T00:00:59.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.030+0000 7f4bcb400640 1 -- 172.21.15.40:0/669517400 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4010690 con 0x7f4bd4108e70 2024-04-27T00:00:59.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.030+0000 7f4bd3e00640 1 -- 172.21.15.40:0/669517400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd41a6f80 con 0x7f4bd4108e70 2024-04-27T00:00:59.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.030+0000 7f4bd3e00640 1 -- 172.21.15.40:0/669517400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd41a73e0 con 0x7f4bd4108e70 2024-04-27T00:00:59.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.032+0000 7f4bd3e00640 1 -- 172.21.15.40:0/669517400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd4110180 con 0x7f4bd4108e70 2024-04-27T00:00:59.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.034+0000 7f4bcb400640 1 -- 172.21.15.40:0/669517400 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4bc40037a0 con 0x7f4bd4108e70 2024-04-27T00:00:59.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.035+0000 7f4bcb400640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4bb80a3c40 0x7f4bb80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:59.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.035+0000 7f4bcb400640 1 -- 172.21.15.40:0/669517400 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f4bc40d0bc0 con 0x7f4bd4108e70 2024-04-27T00:00:59.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.035+0000 7f4bcbe00640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4bb80a3c40 0x7f4bb80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:59.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.036+0000 7f4bcbe00640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4bb80a3c40 0x7f4bb80a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc005ab0 tx=0x7f4bbc005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:59.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.040+0000 7f4bcb400640 1 -- 172.21.15.40:0/669517400 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4bc409d0a0 con 0x7f4bd4108e70 2024-04-27T00:00:59.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.253+0000 7f4bd3e00640 1 -- 172.21.15.40:0/669517400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4bd4063530 con 0x7f4bd4108e70 2024-04-27T00:00:59.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.253+0000 7f4bcb400640 1 -- 172.21.15.40:0/669517400 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f4bc409c9a0 con 0x7f4bd4108e70 2024-04-27T00:00:59.255 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-27T00:00:59.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.257+0000 7f4bd3e00640 1 -- 172.21.15.40:0/669517400 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4bb80a3c40 msgr2=0x7f4bb80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.257+0000 7f4bd3e00640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4bb80a3c40 0x7f4bb80a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc005ab0 tx=0x7f4bbc005a20 comp rx=0 tx=0).stop 2024-04-27T00:00:59.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.257+0000 7f4bd3e00640 1 -- 172.21.15.40:0/669517400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bd4108e70 msgr2=0x7f4bd419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.257+0000 7f4bd3e00640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bd4108e70 0x7f4bd419f070 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400ea60 tx=0x7f4bc400ef20 comp rx=0 tx=0).stop 2024-04-27T00:00:59.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.257+0000 7f4bd3e00640 1 -- 172.21.15.40:0/669517400 shutdown_connections 2024-04-27T00:00:59.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.257+0000 7f4bd3e00640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4bb80a3c40 0x7f4bb80a6100 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.257+0000 7f4bd3e00640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd410a980 0x7f4bd41a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.257+0000 7f4bd3e00640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4bd4109fc0 0x7f4bd419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.257+0000 7f4bd3e00640 1 --2- 172.21.15.40:0/669517400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4bd4108e70 0x7f4bd419f070 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.257+0000 7f4bd3e00640 1 -- 172.21.15.40:0/669517400 >> 172.21.15.40:0/669517400 conn(0x7f4bd4077e00 msgr2=0x7f4bd410d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:59.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.258+0000 7f4bd3e00640 1 -- 172.21.15.40:0/669517400 shutdown_connections 2024-04-27T00:00:59.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.258+0000 7f4bd3e00640 1 -- 172.21.15.40:0/669517400 wait complete. 2024-04-27T00:00:59.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-27T00:00:59.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.479+0000 7f40fb200640 1 -- 172.21.15.40:0/1425957925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40fc076970 msgr2=0x7f40fc074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.479+0000 7f40fb200640 1 --2- 172.21.15.40:0/1425957925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40fc076970 0x7f40fc074dc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f40ec009ca0 tx=0x7f40ec02a8c0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.479+0000 7f40fb200640 1 -- 172.21.15.40:0/1425957925 shutdown_connections 2024-04-27T00:00:59.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.479+0000 7f40fb200640 1 --2- 172.21.15.40:0/1425957925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40fc10ee10 0x7f40fc111200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.479+0000 7f40fb200640 1 --2- 172.21.15.40:0/1425957925 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f40fc075390 0x7f40fc075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.479+0000 7f40fb200640 1 --2- 172.21.15.40:0/1425957925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40fc076970 0x7f40fc074dc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.479+0000 7f40fb200640 1 -- 172.21.15.40:0/1425957925 >> 172.21.15.40:0/1425957925 conn(0x7f40fc0fddd0 msgr2=0x7f40fc1001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:59.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.479+0000 7f40fb200640 1 -- 172.21.15.40:0/1425957925 shutdown_connections 2024-04-27T00:00:59.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.479+0000 7f40fb200640 1 -- 172.21.15.40:0/1425957925 wait complete. 2024-04-27T00:00:59.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.480+0000 7f40fb200640 1 Processor -- start 2024-04-27T00:00:59.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40fb200640 1 -- start start 2024-04-27T00:00:59.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40fb200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f40fc075390 0x7f40fc19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:59.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40fb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40fc076970 0x7f40fc19f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:59.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40fb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40fc10ee10 0x7f40fc1a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:59.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40fb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc114960 con 0x7f40fc10ee10 2024-04-27T00:00:59.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40fb200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc1147e0 con 0x7f40fc075390 2024-04-27T00:00:59.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40fb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc114ae0 con 0x7f40fc076970 2024-04-27T00:00:59.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40f9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40fc076970 0x7f40fc19f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:59.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40f9e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f40fc075390 0x7f40fc19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:59.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40fa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40fc10ee10 0x7f40fc1a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:59.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40f9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40fc076970 0x7f40fc19f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:44504/0 (socket says 172.21.15.40:44504) 2024-04-27T00:00:59.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40f9e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f40fc075390 0x7f40fc19f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:47244/0 (socket says 172.21.15.40:47244) 2024-04-27T00:00:59.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.481+0000 7f40f9400640 1 -- 172.21.15.40:0/2797537092 learned_addr learned my addr 172.21.15.40:0/2797537092 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:59.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.482+0000 7f40f9400640 1 -- 172.21.15.40:0/2797537092 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f40fc075390 msgr2=0x7f40fc19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.482+0000 7f40f9400640 1 --2- 172.21.15.40:0/2797537092 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f40fc075390 0x7f40fc19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.482+0000 7f40f9400640 1 -- 172.21.15.40:0/2797537092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40fc10ee10 msgr2=0x7f40fc1a6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.482+0000 7f40f9400640 1 --2- 172.21.15.40:0/2797537092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40fc10ee10 0x7f40fc1a6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.482+0000 7f40f9400640 1 -- 172.21.15.40:0/2797537092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40fc1a6cb0 con 0x7f40fc076970 2024-04-27T00:00:59.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.482+0000 7f40f9400640 1 --2- 172.21.15.40:0/2797537092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40fc076970 0x7f40fc19f5d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f40e800e9f0 tx=0x7f40e800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:59.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.482+0000 7f40f2a00640 1 -- 172.21.15.40:0/2797537092 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e800cd80 con 0x7f40fc076970 2024-04-27T00:00:59.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.482+0000 7f40f2a00640 1 -- 172.21.15.40:0/2797537092 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40e80054d0 con 0x7f40fc076970 2024-04-27T00:00:59.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.482+0000 7f40f2a00640 1 -- 172.21.15.40:0/2797537092 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8010690 con 0x7f40fc076970 2024-04-27T00:00:59.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.482+0000 7f40fb200640 1 -- 172.21.15.40:0/2797537092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40fc1a6fa0 con 0x7f40fc076970 2024-04-27T00:00:59.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.483+0000 7f40fb200640 1 -- 172.21.15.40:0/2797537092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40fc1a7460 con 0x7f40fc076970 2024-04-27T00:00:59.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.484+0000 7f40fb200640 1 -- 172.21.15.40:0/2797537092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40fc104fe0 con 0x7f40fc076970 2024-04-27T00:00:59.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.486+0000 7f40f2a00640 1 -- 172.21.15.40:0/2797537092 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f40e8004e90 con 0x7f40fc076970 2024-04-27T00:00:59.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.487+0000 7f40f2a00640 1 --2- 172.21.15.40:0/2797537092 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f40e00a3cc0 0x7f40e00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:59.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.488+0000 7f40f2a00640 1 -- 172.21.15.40:0/2797537092 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f40e80d0c90 con 0x7f40fc076970 2024-04-27T00:00:59.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.488+0000 7f40f9e00640 1 --2- 172.21.15.40:0/2797537092 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f40e00a3cc0 0x7f40e00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:59.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.488+0000 7f40f9e00640 1 --2- 172.21.15.40:0/2797537092 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f40e00a3cc0 0x7f40e00a6180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f40ec005e30 tx=0x7f40ec005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:59.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.492+0000 7f40f2a00640 1 -- 172.21.15.40:0/2797537092 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f40e809d170 con 0x7f40fc076970 2024-04-27T00:00:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.697+0000 7f40fb200640 1 -- 172.21.15.40:0/2797537092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f40fc02dd20 con 0x7f40fc076970 2024-04-27T00:00:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.698+0000 7f40f2a00640 1 -- 172.21.15.40:0/2797537092 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f40e809ca70 con 0x7f40fc076970 2024-04-27T00:00:59.699 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-27T00:00:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.700+0000 7f40fb200640 1 -- 172.21.15.40:0/2797537092 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f40e00a3cc0 msgr2=0x7f40e00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.700+0000 7f40fb200640 1 --2- 172.21.15.40:0/2797537092 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f40e00a3cc0 0x7f40e00a6180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f40ec005e30 tx=0x7f40ec005dc0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.700+0000 7f40fb200640 1 -- 172.21.15.40:0/2797537092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40fc076970 msgr2=0x7f40fc19f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.700+0000 7f40fb200640 1 --2- 172.21.15.40:0/2797537092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40fc076970 0x7f40fc19f5d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f40e800e9f0 tx=0x7f40e800edf0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.700+0000 7f40fb200640 1 -- 172.21.15.40:0/2797537092 shutdown_connections 2024-04-27T00:00:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.700+0000 7f40fb200640 1 --2- 172.21.15.40:0/2797537092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40fc10ee10 0x7f40fc1a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.700+0000 7f40fb200640 1 --2- 172.21.15.40:0/2797537092 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f40e00a3cc0 0x7f40e00a6180 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.700+0000 7f40fb200640 1 --2- 172.21.15.40:0/2797537092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40fc076970 0x7f40fc19f5d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.700+0000 7f40fb200640 1 --2- 172.21.15.40:0/2797537092 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f40fc075390 0x7f40fc19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.700+0000 7f40fb200640 1 -- 172.21.15.40:0/2797537092 >> 172.21.15.40:0/2797537092 conn(0x7f40fc0fddd0 msgr2=0x7f40fc1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.700+0000 7f40fb200640 1 -- 172.21.15.40:0/2797537092 shutdown_connections 2024-04-27T00:00:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.701+0000 7f40fb200640 1 -- 172.21.15.40:0/2797537092 wait complete. 2024-04-27T00:00:59.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:59 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1884959695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:00:59.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:00:59 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/669517400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:00:59.722 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-27T00:00:59.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:59 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1884959695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:00:59.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:00:59 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/669517400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:00:59.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:59 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1884959695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:00:59.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:00:59 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/669517400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:00:59.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.926+0000 7fe685c00640 1 -- 172.21.15.40:0/227298322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe680101670 msgr2=0x7fe680101a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.926+0000 7fe685c00640 1 --2- 172.21.15.40:0/227298322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe680101670 0x7fe680101a90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe670009900 tx=0x7fe67002a3f0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.926+0000 7fe685c00640 1 -- 172.21.15.40:0/227298322 shutdown_connections 2024-04-27T00:00:59.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.926+0000 7fe685c00640 1 --2- 172.21.15.40:0/227298322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe680103220 0x7fe680109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.926+0000 7fe685c00640 1 --2- 172.21.15.40:0/227298322 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe680102860 0x7fe680102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.926+0000 7fe685c00640 1 --2- 172.21.15.40:0/227298322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe680101670 0x7fe680101a90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.926+0000 7fe685c00640 1 -- 172.21.15.40:0/227298322 >> 172.21.15.40:0/227298322 conn(0x7fe6800fcc20 msgr2=0x7fe6800ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:00:59.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.926+0000 7fe685c00640 1 -- 172.21.15.40:0/227298322 shutdown_connections 2024-04-27T00:00:59.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.926+0000 7fe685c00640 1 -- 172.21.15.40:0/227298322 wait complete. 2024-04-27T00:00:59.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.927+0000 7fe685c00640 1 Processor -- start 2024-04-27T00:00:59.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.928+0000 7fe685c00640 1 -- start start 2024-04-27T00:00:59.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.928+0000 7fe685c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe680101670 0x7fe68019abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:59.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.928+0000 7fe685c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe680102860 0x7fe68019b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:59.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.928+0000 7fe685c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe680103220 0x7fe6801a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:59.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.928+0000 7fe685c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe68010c0f0 con 0x7fe680101670 2024-04-27T00:00:59.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.928+0000 7fe685c00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fe68010bf70 con 0x7fe680102860 2024-04-27T00:00:59.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.928+0000 7fe685c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe68010c270 con 0x7fe680103220 2024-04-27T00:00:59.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.928+0000 7fe67fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe680101670 0x7fe68019abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:59.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.928+0000 7fe67fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe680101670 0x7fe68019abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35134/0 (socket says 172.21.15.40:35134) 2024-04-27T00:00:59.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.928+0000 7fe67fe00640 1 -- 172.21.15.40:0/682877822 learned_addr learned my addr 172.21.15.40:0/682877822 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:00:59.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.928+0000 7fe677400640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe680102860 0x7fe68019b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:59.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.928+0000 7fe685200640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe680103220 0x7fe6801a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:59.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.929+0000 7fe67fe00640 1 -- 172.21.15.40:0/682877822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe680103220 msgr2=0x7fe6801a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.929+0000 7fe67fe00640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe680103220 0x7fe6801a2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.929+0000 7fe67fe00640 1 -- 172.21.15.40:0/682877822 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe680102860 msgr2=0x7fe68019b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:00:59.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.929+0000 7fe67fe00640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe680102860 0x7fe68019b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:00:59.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.929+0000 7fe67fe00640 1 -- 172.21.15.40:0/682877822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6801a27d0 con 0x7fe680101670 2024-04-27T00:00:59.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.929+0000 7fe685200640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe680103220 0x7fe6801a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:00:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.929+0000 7fe677400640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe680102860 0x7fe68019b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:00:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.929+0000 7fe67fe00640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe680101670 0x7fe68019abb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe670031500 tx=0x7fe670003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.929+0000 7fe67d600640 1 -- 172.21.15.40:0/682877822 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67003f070 con 0x7fe680101670 2024-04-27T00:00:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.930+0000 7fe67d600640 1 -- 172.21.15.40:0/682877822 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe670031c50 con 0x7fe680101670 2024-04-27T00:00:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.930+0000 7fe67d600640 1 -- 172.21.15.40:0/682877822 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670035030 con 0x7fe680101670 2024-04-27T00:00:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.930+0000 7fe685c00640 1 -- 172.21.15.40:0/682877822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6801a2a60 con 0x7fe680101670 2024-04-27T00:00:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.930+0000 7fe685c00640 1 -- 172.21.15.40:0/682877822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6801a2fa0 con 0x7fe680101670 2024-04-27T00:00:59.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.931+0000 7fe685c00640 1 -- 172.21.15.40:0/682877822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6801089e0 con 0x7fe680101670 2024-04-27T00:00:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.933+0000 7fe67d600640 1 -- 172.21.15.40:0/682877822 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe670042070 con 0x7fe680101670 2024-04-27T00:00:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.934+0000 7fe67d600640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe6600a3ca0 0x7fe6600a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:00:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.934+0000 7fe67d600640 1 -- 172.21.15.40:0/682877822 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fe6700f10c0 con 0x7fe680101670 2024-04-27T00:00:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.935+0000 7fe677400640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe6600a3ca0 0x7fe6600a6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:00:59.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.935+0000 7fe677400640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe6600a3ca0 0x7fe6600a6160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe68019c0c0 tx=0x7fe668005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:00:59.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:00:59.941+0000 7fe67d600640 1 -- 172.21.15.40:0/682877822 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe6700bd5a0 con 0x7fe680101670 2024-04-27T00:01:00.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.148+0000 7fe685c00640 1 -- 172.21.15.40:0/682877822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe680063530 con 0x7fe680101670 2024-04-27T00:01:00.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.149+0000 7fe67d600640 1 -- 172.21.15.40:0/682877822 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7fe6700bcea0 con 0x7fe680101670 2024-04-27T00:01:00.151 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-27T00:01:00.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.151+0000 7fe685c00640 1 -- 172.21.15.40:0/682877822 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe6600a3ca0 msgr2=0x7fe6600a6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:00.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.151+0000 7fe685c00640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe6600a3ca0 0x7fe6600a6160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe68019c0c0 tx=0x7fe668005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:00.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.151+0000 7fe685c00640 1 -- 172.21.15.40:0/682877822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe680101670 msgr2=0x7fe68019abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:00.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.151+0000 7fe685c00640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe680101670 0x7fe68019abb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe670031500 tx=0x7fe670003c10 comp rx=0 tx=0).stop 2024-04-27T00:01:00.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.151+0000 7fe685c00640 1 -- 172.21.15.40:0/682877822 shutdown_connections 2024-04-27T00:01:00.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.151+0000 7fe685c00640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe6600a3ca0 0x7fe6600a6160 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.151+0000 7fe685c00640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe680103220 0x7fe6801a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.151+0000 7fe685c00640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe680102860 0x7fe68019b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.151+0000 7fe685c00640 1 --2- 172.21.15.40:0/682877822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe680101670 0x7fe68019abb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.151+0000 7fe685c00640 1 -- 172.21.15.40:0/682877822 >> 172.21.15.40:0/682877822 conn(0x7fe6800fcc20 msgr2=0x7fe680105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:00.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.152+0000 7fe685c00640 1 -- 172.21.15.40:0/682877822 shutdown_connections 2024-04-27T00:01:00.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.152+0000 7fe685c00640 1 -- 172.21.15.40:0/682877822 wait complete. 2024-04-27T00:01:00.171 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-27T00:01:00.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.375+0000 7f3880400640 1 -- 172.21.15.40:0/892839364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3878104530 msgr2=0x7f387810adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:00.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.375+0000 7f3880400640 1 --2- 172.21.15.40:0/892839364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3878104530 0x7f387810adc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3870009860 tx=0x7f387002a340 comp rx=0 tx=0).stop 2024-04-27T00:01:00.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.375+0000 7f3880400640 1 -- 172.21.15.40:0/892839364 shutdown_connections 2024-04-27T00:01:00.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.375+0000 7f3880400640 1 --2- 172.21.15.40:0/892839364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3878104530 0x7f387810adc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.375+0000 7f3880400640 1 --2- 172.21.15.40:0/892839364 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3878103b70 0x7f3878103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.375+0000 7f3880400640 1 --2- 172.21.15.40:0/892839364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3878102980 0x7f3878102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.375+0000 7f3880400640 1 -- 172.21.15.40:0/892839364 >> 172.21.15.40:0/892839364 conn(0x7f38780fdf10 msgr2=0x7f3878100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:00.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.375+0000 7f3880400640 1 -- 172.21.15.40:0/892839364 shutdown_connections 2024-04-27T00:01:00.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.375+0000 7f3880400640 1 -- 172.21.15.40:0/892839364 wait complete. 2024-04-27T00:01:00.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.377+0000 7f3880400640 1 Processor -- start 2024-04-27T00:01:00.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.377+0000 7f3880400640 1 -- start start 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.377+0000 7f3880400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3878102980 0x7f3878198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.377+0000 7f3880400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3878103b70 0x7f38781990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.377+0000 7f3880400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3878104530 0x7f38781a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.377+0000 7f3880400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f387810e4b0 con 0x7f3878102980 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.377+0000 7f3880400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f387810e330 con 0x7f3878103b70 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.377+0000 7f3880400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f387810e630 con 0x7f3878104530 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.377+0000 7f387d000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3878103b70 0x7f38781990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.377+0000 7f387da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3878102980 0x7f3878198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.378+0000 7f387d000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3878103b70 0x7f38781990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:46546/0 (socket says 172.21.15.40:46546) 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.378+0000 7f387d000640 1 -- 172.21.15.40:0/1653555159 learned_addr learned my addr 172.21.15.40:0/1653555159 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.378+0000 7f387e400640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3878104530 0x7f38781a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.378+0000 7f387d000640 1 -- 172.21.15.40:0/1653555159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3878104530 msgr2=0x7f38781a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.378+0000 7f387d000640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3878104530 0x7f38781a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.378+0000 7f387d000640 1 -- 172.21.15.40:0/1653555159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3878102980 msgr2=0x7f3878198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:00.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.378+0000 7f387d000640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3878102980 0x7f3878198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.378+0000 7f387d000640 1 -- 172.21.15.40:0/1653555159 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38781a07c0 con 0x7f3878103b70 2024-04-27T00:01:00.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.378+0000 7f387d000640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3878103b70 0x7f38781990e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f386400e9c0 tx=0x7f386400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:00.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.379+0000 7f3876000640 1 -- 172.21.15.40:0/1653555159 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386400ccc0 con 0x7f3878103b70 2024-04-27T00:01:00.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.379+0000 7f3876000640 1 -- 172.21.15.40:0/1653555159 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38640054d0 con 0x7f3878103b70 2024-04-27T00:01:00.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.379+0000 7f3880400640 1 -- 172.21.15.40:0/1653555159 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38781a0ab0 con 0x7f3878103b70 2024-04-27T00:01:00.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.379+0000 7f3876000640 1 -- 172.21.15.40:0/1653555159 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3864010430 con 0x7f3878103b70 2024-04-27T00:01:00.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.379+0000 7f3880400640 1 -- 172.21.15.40:0/1653555159 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38781a1050 con 0x7f3878103b70 2024-04-27T00:01:00.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.380+0000 7f3880400640 1 -- 172.21.15.40:0/1653555159 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3878109dc0 con 0x7f3878103b70 2024-04-27T00:01:00.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.382+0000 7f3876000640 1 -- 172.21.15.40:0/1653555159 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3864004f80 con 0x7f3878103b70 2024-04-27T00:01:00.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.384+0000 7f3876000640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f38600a3cc0 0x7f38600a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:00.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.384+0000 7f3876000640 1 -- 172.21.15.40:0/1653555159 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f38640d0e20 con 0x7f3878103b70 2024-04-27T00:01:00.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.384+0000 7f387da00640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f38600a3cc0 0x7f38600a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:00.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.385+0000 7f387da00640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f38600a3cc0 0x7f38600a6180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f38781039d0 tx=0x7f386c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:00.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.386+0000 7f3876000640 1 -- 172.21.15.40:0/1653555159 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f386409d300 con 0x7f3878103b70 2024-04-27T00:01:00.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.594+0000 7f3880400640 1 -- 172.21.15.40:0/1653555159 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f387802dc60 con 0x7f3878103b70 2024-04-27T00:01:00.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.595+0000 7f3876000640 1 -- 172.21.15.40:0/1653555159 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f386409cc00 con 0x7f3878103b70 2024-04-27T00:01:00.597 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-27T00:01:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.598+0000 7f3880400640 1 -- 172.21.15.40:0/1653555159 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f38600a3cc0 msgr2=0x7f38600a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.598+0000 7f3880400640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f38600a3cc0 0x7f38600a6180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f38781039d0 tx=0x7f386c009500 comp rx=0 tx=0).stop 2024-04-27T00:01:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.598+0000 7f3880400640 1 -- 172.21.15.40:0/1653555159 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3878103b70 msgr2=0x7f38781990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.598+0000 7f3880400640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3878103b70 0x7f38781990e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f386400e9c0 tx=0x7f386400ee80 comp rx=0 tx=0).stop 2024-04-27T00:01:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.598+0000 7f3880400640 1 -- 172.21.15.40:0/1653555159 shutdown_connections 2024-04-27T00:01:00.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.598+0000 7f3880400640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f38600a3cc0 0x7f38600a6180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.598+0000 7f3880400640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3878104530 0x7f38781a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.598+0000 7f3880400640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3878103b70 0x7f38781990e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.598+0000 7f3880400640 1 --2- 172.21.15.40:0/1653555159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3878102980 0x7f3878198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.598+0000 7f3880400640 1 -- 172.21.15.40:0/1653555159 >> 172.21.15.40:0/1653555159 conn(0x7f38780fdf10 msgr2=0x7f3878106f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.598+0000 7f3880400640 1 -- 172.21.15.40:0/1653555159 shutdown_connections 2024-04-27T00:01:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.598+0000 7f3880400640 1 -- 172.21.15.40:0/1653555159 wait complete. 2024-04-27T00:01:00.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-27T00:01:00.695 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:00 smithi040 ceph-mon[28095]: pgmap v389: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-27T00:01:00.695 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:00 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2797537092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:00.695 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:00 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/682877822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:00.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:00 smithi080 ceph-mon[32362]: pgmap v389: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-27T00:01:00.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:00 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2797537092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:00.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:00 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/682877822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.830+0000 7f1967200640 1 -- 172.21.15.40:0/248105968 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f19680772f0 msgr2=0x7f1968077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.830+0000 7f1967200640 1 --2- 172.21.15.40:0/248105968 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f19680772f0 0x7f1968077710 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1958009860 tx=0x7f195802a340 comp rx=0 tx=0).stop 2024-04-27T00:01:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.830+0000 7f1967200640 1 -- 172.21.15.40:0/248105968 shutdown_connections 2024-04-27T00:01:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.830+0000 7f1967200640 1 --2- 172.21.15.40:0/248105968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1968078ed0 0x7f1968079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.830+0000 7f1967200640 1 --2- 172.21.15.40:0/248105968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1968078530 0x7f1968078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.830+0000 7f1967200640 1 --2- 172.21.15.40:0/248105968 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f19680772f0 0x7f1968077710 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.830+0000 7f1967200640 1 -- 172.21.15.40:0/248105968 >> 172.21.15.40:0/248105968 conn(0x7f19680fff00 msgr2=0x7f1968102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.830+0000 7f1967200640 1 -- 172.21.15.40:0/248105968 shutdown_connections 2024-04-27T00:01:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.830+0000 7f1967200640 1 -- 172.21.15.40:0/248105968 wait complete. 2024-04-27T00:01:00.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.832+0000 7f1967200640 1 Processor -- start 2024-04-27T00:01:00.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.832+0000 7f1967200640 1 -- start start 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.832+0000 7f1967200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19680772f0 0x7f196819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.832+0000 7f1967200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1968078530 0x7f196819f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.832+0000 7f1967200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1968078ed0 0x7f19681a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.832+0000 7f1967200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f19681148d0 con 0x7f1968078ed0 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.832+0000 7f1967200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1968114750 con 0x7f1968078530 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.832+0000 7f1967200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1968114a50 con 0x7f19680772f0 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.833+0000 7f1965400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1968078530 0x7f196819f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.833+0000 7f1965400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1968078530 0x7f196819f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:46570/0 (socket says 172.21.15.40:46570) 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.833+0000 7f1965400640 1 -- 172.21.15.40:0/2386120615 learned_addr learned my addr 172.21.15.40:0/2386120615 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.833+0000 7f1965e00640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19680772f0 0x7f196819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.833+0000 7f1966800640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1968078ed0 0x7f19681a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.833+0000 7f1965400640 1 -- 172.21.15.40:0/2386120615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19680772f0 msgr2=0x7f196819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.833+0000 7f1965400640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19680772f0 0x7f196819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.833+0000 7f1965400640 1 -- 172.21.15.40:0/2386120615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1968078ed0 msgr2=0x7f19681a6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:00.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.833+0000 7f1965400640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1968078ed0 0x7f19681a6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:00.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.833+0000 7f1965400640 1 -- 172.21.15.40:0/2386120615 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19681a6bc0 con 0x7f1968078530 2024-04-27T00:01:00.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.833+0000 7f1965400640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1968078530 0x7f196819f4e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f195400ea60 tx=0x7f195400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:00.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.834+0000 7f195ea00640 1 -- 172.21.15.40:0/2386120615 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195400cde0 con 0x7f1968078530 2024-04-27T00:01:00.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.834+0000 7f195ea00640 1 -- 172.21.15.40:0/2386120615 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19540035f0 con 0x7f1968078530 2024-04-27T00:01:00.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.834+0000 7f195ea00640 1 -- 172.21.15.40:0/2386120615 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954010690 con 0x7f1968078530 2024-04-27T00:01:00.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.834+0000 7f1967200640 1 -- 172.21.15.40:0/2386120615 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19681a6eb0 con 0x7f1968078530 2024-04-27T00:01:00.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.834+0000 7f1967200640 1 -- 172.21.15.40:0/2386120615 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19681a73c0 con 0x7f1968078530 2024-04-27T00:01:00.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.835+0000 7f1967200640 1 -- 172.21.15.40:0/2386120615 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1968111300 con 0x7f1968078530 2024-04-27T00:01:00.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.838+0000 7f195ea00640 1 -- 172.21.15.40:0/2386120615 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f19540037a0 con 0x7f1968078530 2024-04-27T00:01:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.839+0000 7f195ea00640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f194c0a3cc0 0x7f194c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.839+0000 7f195ea00640 1 -- 172.21.15.40:0/2386120615 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f19540d0d20 con 0x7f1968078530 2024-04-27T00:01:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.840+0000 7f1965e00640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f194c0a3cc0 0x7f194c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.840+0000 7f1965e00640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f194c0a3cc0 0x7f194c0a6180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1958000c00 tx=0x7f1958030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:00.841+0000 7f195ea00640 1 -- 172.21.15.40:0/2386120615 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f195409d200 con 0x7f1968078530 2024-04-27T00:01:00.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:00 smithi103 ceph-mon[32952]: pgmap v389: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-27T00:01:00.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:00 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2797537092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:00.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:00 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/682877822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:01.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.088+0000 7f1967200640 1 -- 172.21.15.40:0/2386120615 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f196802dad0 con 0x7f1968078530 2024-04-27T00:01:01.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.089+0000 7f195ea00640 1 -- 172.21.15.40:0/2386120615 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11872 (secure 0 0 0) 0x7f195409cb00 con 0x7f1968078530 2024-04-27T00:01:01.091 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:01.091 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/2265538751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":2265538751},{"type":"v1","addr":"172.21.15.40:6841","nonce":2265538751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:58:50.616677+0000","modified":"2024-04-26T23:59:25.374905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24523":{"gid":24523,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.80:6839/3536638444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6838","nonce":3536638444},{"type":"v1","addr":"172.21.15.80:6839","nonce":3536638444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24559,24560,24590]},"id":1}]} 2024-04-27T00:01:01.091 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 23 2024-04-27T00:01:01.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.092+0000 7f1967200640 1 -- 172.21.15.40:0/2386120615 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f194c0a3cc0 msgr2=0x7f194c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:01.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.092+0000 7f1967200640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f194c0a3cc0 0x7f194c0a6180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1958000c00 tx=0x7f1958030470 comp rx=0 tx=0).stop 2024-04-27T00:01:01.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.092+0000 7f1967200640 1 -- 172.21.15.40:0/2386120615 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1968078530 msgr2=0x7f196819f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:01.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.092+0000 7f1967200640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1968078530 0x7f196819f4e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f195400ea60 tx=0x7f195400ef20 comp rx=0 tx=0).stop 2024-04-27T00:01:01.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.092+0000 7f1967200640 1 -- 172.21.15.40:0/2386120615 shutdown_connections 2024-04-27T00:01:01.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.092+0000 7f1967200640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1968078ed0 0x7f19681a6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.092+0000 7f1967200640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1968078530 0x7f196819f4e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.092+0000 7f1967200640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f194c0a3cc0 0x7f194c0a6180 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.092+0000 7f1967200640 1 --2- 172.21.15.40:0/2386120615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19680772f0 0x7f196819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.092+0000 7f1967200640 1 -- 172.21.15.40:0/2386120615 >> 172.21.15.40:0/2386120615 conn(0x7f19680fff00 msgr2=0x7f1968101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:01.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.093+0000 7f1967200640 1 -- 172.21.15.40:0/2386120615 shutdown_connections 2024-04-27T00:01:01.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.093+0000 7f1967200640 1 -- 172.21.15.40:0/2386120615 wait complete. 2024-04-27T00:01:01.115 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T00:01:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.328+0000 7f2917200640 1 -- 172.21.15.40:0/3103603695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2918075330 msgr2=0x7f2918075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.328+0000 7f2917200640 1 --2- 172.21.15.40:0/3103603695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2918075330 0x7f2918075790 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2900009900 tx=0x7f290002a490 comp rx=0 tx=0).stop 2024-04-27T00:01:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.329+0000 7f2917200640 1 -- 172.21.15.40:0/3103603695 shutdown_connections 2024-04-27T00:01:01.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.329+0000 7f2917200640 1 --2- 172.21.15.40:0/3103603695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f291810a640 0x7f291810cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.329+0000 7f2917200640 1 --2- 172.21.15.40:0/3103603695 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2918075330 0x7f2918075790 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.329+0000 7f2917200640 1 --2- 172.21.15.40:0/3103603695 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2918076910 0x7f2918074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.329+0000 7f2917200640 1 -- 172.21.15.40:0/3103603695 >> 172.21.15.40:0/3103603695 conn(0x7f29180fdd40 msgr2=0x7f2918100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:01.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.329+0000 7f2917200640 1 -- 172.21.15.40:0/3103603695 shutdown_connections 2024-04-27T00:01:01.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.329+0000 7f2917200640 1 -- 172.21.15.40:0/3103603695 wait complete. 2024-04-27T00:01:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.330+0000 7f2917200640 1 Processor -- start 2024-04-27T00:01:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.330+0000 7f2917200640 1 -- start start 2024-04-27T00:01:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.330+0000 7f2917200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2918075330 0x7f291819ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2917200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2918076910 0x7f291819b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2917200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f291810a640 0x7f29181a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2917200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2918110530 con 0x7f291810a640 2024-04-27T00:01:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2917200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f29181103b0 con 0x7f2918076910 2024-04-27T00:01:01.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2917200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f29181106b0 con 0x7f2918075330 2024-04-27T00:01:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2915e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2918075330 0x7f291819ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f290d400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2918076910 0x7f291819b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2915e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2918075330 0x7f291819ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50396/0 (socket says 172.21.15.40:50396) 2024-04-27T00:01:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2915e00640 1 -- 172.21.15.40:0/1367804018 learned_addr learned my addr 172.21.15.40:0/1367804018 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2916800640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f291810a640 0x7f29181a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2915e00640 1 -- 172.21.15.40:0/1367804018 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2918076910 msgr2=0x7f291819b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2915e00640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2918076910 0x7f291819b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2915e00640 1 -- 172.21.15.40:0/1367804018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f291810a640 msgr2=0x7f29181a2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2915e00640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f291810a640 0x7f29181a2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.331+0000 7f2915e00640 1 -- 172.21.15.40:0/1367804018 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29181a2790 con 0x7f2918075330 2024-04-27T00:01:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.332+0000 7f290d400640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2918076910 0x7f291819b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.332+0000 7f2915e00640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2918075330 0x7f291819ab70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f290800e9f0 tx=0x7f290800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.332+0000 7f290f400640 1 -- 172.21.15.40:0/1367804018 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290800cd80 con 0x7f2918075330 2024-04-27T00:01:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.332+0000 7f2917200640 1 -- 172.21.15.40:0/1367804018 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29181a2a80 con 0x7f2918075330 2024-04-27T00:01:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.332+0000 7f290f400640 1 -- 172.21.15.40:0/1367804018 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29080054d0 con 0x7f2918075330 2024-04-27T00:01:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.332+0000 7f290f400640 1 -- 172.21.15.40:0/1367804018 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908010690 con 0x7f2918075330 2024-04-27T00:01:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.333+0000 7f2917200640 1 -- 172.21.15.40:0/1367804018 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29181a2fa0 con 0x7f2918075330 2024-04-27T00:01:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.334+0000 7f2917200640 1 -- 172.21.15.40:0/1367804018 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2918109400 con 0x7f2918075330 2024-04-27T00:01:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.336+0000 7f290f400640 1 -- 172.21.15.40:0/1367804018 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2908004e90 con 0x7f2918075330 2024-04-27T00:01:01.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.337+0000 7f290f400640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f28fc0a3cc0 0x7f28fc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:01.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.337+0000 7f290f400640 1 -- 172.21.15.40:0/1367804018 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f29080d0dc0 con 0x7f2918075330 2024-04-27T00:01:01.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.338+0000 7f290d400640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f28fc0a3cc0 0x7f28fc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.338+0000 7f290d400640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f28fc0a3cc0 0x7f28fc0a6180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2900031500 tx=0x7f2900005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:01.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.340+0000 7f290f400640 1 -- 172.21.15.40:0/1367804018 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f290809d2a0 con 0x7f2918075330 2024-04-27T00:01:01.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.547+0000 7f2917200640 1 -- 172.21.15.40:0/1367804018 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2918063560 con 0x7f2918075330 2024-04-27T00:01:01.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.549+0000 7f290f400640 1 -- 172.21.15.40:0/1367804018 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24531 (secure 0 0 0) 0x7f290809cba0 con 0x7f2918075330 2024-04-27T00:01:01.551 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:01.551 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-27T00:00:46.365303+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9},{"from":5,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T00:01:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.551+0000 7f2917200640 1 -- 172.21.15.40:0/1367804018 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f28fc0a3cc0 msgr2=0x7f28fc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.551+0000 7f2917200640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f28fc0a3cc0 0x7f28fc0a6180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2900031500 tx=0x7f2900005a20 comp rx=0 tx=0).stop 2024-04-27T00:01:01.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.552+0000 7f2917200640 1 -- 172.21.15.40:0/1367804018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2918075330 msgr2=0x7f291819ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:01.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.552+0000 7f2917200640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2918075330 0x7f291819ab70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f290800e9f0 tx=0x7f290800edf0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.552+0000 7f2917200640 1 -- 172.21.15.40:0/1367804018 shutdown_connections 2024-04-27T00:01:01.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.552+0000 7f2917200640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f291810a640 0x7f29181a2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.552+0000 7f2917200640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2918076910 0x7f291819b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.552+0000 7f2917200640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f28fc0a3cc0 0x7f28fc0a6180 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.552+0000 7f2917200640 1 --2- 172.21.15.40:0/1367804018 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2918075330 0x7f291819ab70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.552+0000 7f2917200640 1 -- 172.21.15.40:0/1367804018 >> 172.21.15.40:0/1367804018 conn(0x7f29180fdd40 msgr2=0x7f29181065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:01.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.552+0000 7f2917200640 1 -- 172.21.15.40:0/1367804018 shutdown_connections 2024-04-27T00:01:01.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.552+0000 7f2917200640 1 -- 172.21.15.40:0/1367804018 wait complete. 2024-04-27T00:01:01.573 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:53:32.708818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:58:48.506601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:58:49.540141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:58:53.179820+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-27T00:01:01.574 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-27T00:01:01.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:01 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1653555159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:01.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:01 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2386120615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:01.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.752+0000 7f4351e00640 1 -- 172.21.15.40:0/434736965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f434c108540 msgr2=0x7f434c10a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.752+0000 7f4351e00640 1 --2- 172.21.15.40:0/434736965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f434c108540 0x7f434c10a970 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4344009860 tx=0x7f434402a340 comp rx=0 tx=0).stop 2024-04-27T00:01:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.752+0000 7f4351e00640 1 -- 172.21.15.40:0/434736965 shutdown_connections 2024-04-27T00:01:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.752+0000 7f4351e00640 1 --2- 172.21.15.40:0/434736965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f434c108540 0x7f434c10a970 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.752+0000 7f4351e00640 1 --2- 172.21.15.40:0/434736965 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f434c105c10 0x7f434c108000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.752+0000 7f4351e00640 1 --2- 172.21.15.40:0/434736965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f434c1032e0 0x7f434c1056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.752+0000 7f4351e00640 1 -- 172.21.15.40:0/434736965 >> 172.21.15.40:0/434736965 conn(0x7f434c0fccd0 msgr2=0x7f434c0ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.752+0000 7f4351e00640 1 -- 172.21.15.40:0/434736965 shutdown_connections 2024-04-27T00:01:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.752+0000 7f4351e00640 1 -- 172.21.15.40:0/434736965 wait complete. 2024-04-27T00:01:01.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.753+0000 7f4351e00640 1 Processor -- start 2024-04-27T00:01:01.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.753+0000 7f4351e00640 1 -- start start 2024-04-27T00:01:01.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.753+0000 7f4351e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f434c1032e0 0x7f434c19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:01.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.753+0000 7f4351e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f434c105c10 0x7f434c19b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:01.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.753+0000 7f4351e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f434c108540 0x7f434c1a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:01.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.753+0000 7f4351e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f434c10e390 con 0x7f434c108540 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.753+0000 7f4351e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f434c10e210 con 0x7f434c105c10 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.753+0000 7f4351e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f434c10e510 con 0x7f434c1032e0 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.754+0000 7f4350a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f434c1032e0 0x7f434c19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.754+0000 7f4351400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f434c108540 0x7f434c1a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.754+0000 7f4350a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f434c1032e0 0x7f434c19abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50412/0 (socket says 172.21.15.40:50412) 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.754+0000 7f4350a00640 1 -- 172.21.15.40:0/4118931230 learned_addr learned my addr 172.21.15.40:0/4118931230 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.754+0000 7f434be00640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f434c105c10 0x7f434c19b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.754+0000 7f4350a00640 1 -- 172.21.15.40:0/4118931230 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f434c105c10 msgr2=0x7f434c19b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.754+0000 7f4350a00640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f434c105c10 0x7f434c19b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.754+0000 7f4350a00640 1 -- 172.21.15.40:0/4118931230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f434c108540 msgr2=0x7f434c1a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.754+0000 7f4350a00640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f434c108540 0x7f434c1a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.754+0000 7f4350a00640 1 -- 172.21.15.40:0/4118931230 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434c1a2800 con 0x7f434c1032e0 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.754+0000 7f4350a00640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f434c1032e0 0x7f434c19abe0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f433c00c420 tx=0x7f433c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.755+0000 7f4349600640 1 -- 172.21.15.40:0/4118931230 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c003970 con 0x7f434c1032e0 2024-04-27T00:01:01.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.755+0000 7f4349600640 1 -- 172.21.15.40:0/4118931230 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f433c004020 con 0x7f434c1032e0 2024-04-27T00:01:01.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.755+0000 7f4351e00640 1 -- 172.21.15.40:0/4118931230 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f434c1a2af0 con 0x7f434c1032e0 2024-04-27T00:01:01.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.755+0000 7f4349600640 1 -- 172.21.15.40:0/4118931230 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c00b980 con 0x7f434c1032e0 2024-04-27T00:01:01.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.755+0000 7f4351e00640 1 -- 172.21.15.40:0/4118931230 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f434c1a3040 con 0x7f434c1032e0 2024-04-27T00:01:01.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.756+0000 7f4351e00640 1 -- 172.21.15.40:0/4118931230 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f434c10a030 con 0x7f434c1032e0 2024-04-27T00:01:01.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.758+0000 7f4349600640 1 -- 172.21.15.40:0/4118931230 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f433c01d020 con 0x7f434c1032e0 2024-04-27T00:01:01.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.759+0000 7f4349600640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f43300a3c40 0x7f43300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:01.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.759+0000 7f4349600640 1 -- 172.21.15.40:0/4118931230 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f433c0d0d60 con 0x7f434c1032e0 2024-04-27T00:01:01.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.760+0000 7f434be00640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f43300a3c40 0x7f43300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:01.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.760+0000 7f434be00640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f43300a3c40 0x7f43300a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f434c19c0f0 tx=0x7f4334005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.762+0000 7f4349600640 1 -- 172.21.15.40:0/4118931230 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f433c09d240 con 0x7f434c1032e0 2024-04-27T00:01:01.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:01 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1653555159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:01.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:01 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2386120615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:01.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:01 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1653555159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:01.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:01 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2386120615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:01.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.958+0000 7f4351e00640 1 -- 172.21.15.40:0/4118931230 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f434c02dd20 con 0x7f434c1032e0 2024-04-27T00:01:01.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.960+0000 7f4349600640 1 -- 172.21.15.40:0/4118931230 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f433c09cb40 con 0x7f434c1032e0 2024-04-27T00:01:01.961 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-27T00:01:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.962+0000 7f4351e00640 1 -- 172.21.15.40:0/4118931230 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f43300a3c40 msgr2=0x7f43300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.962+0000 7f4351e00640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f43300a3c40 0x7f43300a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f434c19c0f0 tx=0x7f4334005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.962+0000 7f4351e00640 1 -- 172.21.15.40:0/4118931230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f434c1032e0 msgr2=0x7f434c19abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.962+0000 7f4351e00640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f434c1032e0 0x7f434c19abe0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f433c00c420 tx=0x7f433c009f90 comp rx=0 tx=0).stop 2024-04-27T00:01:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.962+0000 7f4351e00640 1 -- 172.21.15.40:0/4118931230 shutdown_connections 2024-04-27T00:01:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.962+0000 7f4351e00640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f434c108540 0x7f434c1a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.962+0000 7f4351e00640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f434c105c10 0x7f434c19b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.962+0000 7f4351e00640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f43300a3c40 0x7f43300a6100 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.962+0000 7f4351e00640 1 --2- 172.21.15.40:0/4118931230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f434c1032e0 0x7f434c19abe0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.962+0000 7f4351e00640 1 -- 172.21.15.40:0/4118931230 >> 172.21.15.40:0/4118931230 conn(0x7f434c0fccd0 msgr2=0x7f434c102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.962+0000 7f4351e00640 1 -- 172.21.15.40:0/4118931230 shutdown_connections 2024-04-27T00:01:01.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:01.962+0000 7f4351e00640 1 -- 172.21.15.40:0/4118931230 wait complete. 2024-04-27T00:01:01.982 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-27T00:01:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.200+0000 7f3c0a600640 1 -- 172.21.15.40:0/2729481459 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c04104a70 msgr2=0x7f3c04104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.200+0000 7f3c0a600640 1 --2- 172.21.15.40:0/2729481459 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c04104a70 0x7f3c04104e90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4009900 tx=0x7f3bf402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.200+0000 7f3c0a600640 1 -- 172.21.15.40:0/2729481459 shutdown_connections 2024-04-27T00:01:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.200+0000 7f3c0a600640 1 --2- 172.21.15.40:0/2729481459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c04106620 0x7f3c0410ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.200+0000 7f3c0a600640 1 --2- 172.21.15.40:0/2729481459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c04105c60 0x7f3c041060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.200+0000 7f3c0a600640 1 --2- 172.21.15.40:0/2729481459 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c04104a70 0x7f3c04104e90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.200+0000 7f3c0a600640 1 -- 172.21.15.40:0/2729481459 >> 172.21.15.40:0/2729481459 conn(0x7f3c040fffe0 msgr2=0x7f3c04102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.200+0000 7f3c0a600640 1 -- 172.21.15.40:0/2729481459 shutdown_connections 2024-04-27T00:01:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.200+0000 7f3c0a600640 1 -- 172.21.15.40:0/2729481459 wait complete. 2024-04-27T00:01:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.201+0000 7f3c0a600640 1 Processor -- start 2024-04-27T00:01:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.202+0000 7f3c0a600640 1 -- start start 2024-04-27T00:01:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.202+0000 7f3c0a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c04104a70 0x7f3c0419ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.202+0000 7f3c0a600640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c04105c60 0x7f3c0419b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.202+0000 7f3c0a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c04106620 0x7f3c041a2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.202+0000 7f3c0a600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c041105e0 con 0x7f3c04106620 2024-04-27T00:01:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.202+0000 7f3c0a600640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f3c04110460 con 0x7f3c04105c60 2024-04-27T00:01:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.202+0000 7f3c0a600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c04110760 con 0x7f3c04104a70 2024-04-27T00:01:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.202+0000 7f3bfbe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c04105c60 0x7f3c0419b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.202+0000 7f3bfbe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c04105c60 0x7f3c0419b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:46610/0 (socket says 172.21.15.40:46610) 2024-04-27T00:01:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.202+0000 7f3bfbe00640 1 -- 172.21.15.40:0/350854209 learned_addr learned my addr 172.21.15.40:0/350854209 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.203+0000 7f3c09c00640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c04106620 0x7f3c041a2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.203+0000 7f3c09200640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c04104a70 0x7f3c0419ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.203+0000 7f3bfbe00640 1 -- 172.21.15.40:0/350854209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c04104a70 msgr2=0x7f3c0419ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.203+0000 7f3bfbe00640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c04104a70 0x7f3c0419ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.203+0000 7f3bfbe00640 1 -- 172.21.15.40:0/350854209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c04106620 msgr2=0x7f3c041a2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.203+0000 7f3bfbe00640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c04106620 0x7f3c041a2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.203+0000 7f3bfbe00640 1 -- 172.21.15.40:0/350854209 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c041a2930 con 0x7f3c04105c60 2024-04-27T00:01:02.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.203+0000 7f3bfbe00640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c04105c60 0x7f3c0419b260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00e9f0 tx=0x7f3bec00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:02.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.203+0000 7f3c02000640 1 -- 172.21.15.40:0/350854209 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec00ccc0 con 0x7f3c04105c60 2024-04-27T00:01:02.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.204+0000 7f3c02000640 1 -- 172.21.15.40:0/350854209 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bec00ce60 con 0x7f3c04105c60 2024-04-27T00:01:02.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.204+0000 7f3c02000640 1 -- 172.21.15.40:0/350854209 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec010430 con 0x7f3c04105c60 2024-04-27T00:01:02.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.204+0000 7f3c0a600640 1 -- 172.21.15.40:0/350854209 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c041a2c20 con 0x7f3c04105c60 2024-04-27T00:01:02.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.204+0000 7f3c0a600640 1 -- 172.21.15.40:0/350854209 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c041a30f0 con 0x7f3c04105c60 2024-04-27T00:01:02.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.205+0000 7f3c0a600640 1 -- 172.21.15.40:0/350854209 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c0410be60 con 0x7f3c04105c60 2024-04-27T00:01:02.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.207+0000 7f3c02000640 1 -- 172.21.15.40:0/350854209 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3bec0046c0 con 0x7f3c04105c60 2024-04-27T00:01:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.208+0000 7f3c02000640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3be80a3c40 0x7f3be80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.208+0000 7f3c02000640 1 -- 172.21.15.40:0/350854209 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f3bec0d0d90 con 0x7f3c04105c60 2024-04-27T00:01:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.209+0000 7f3c09200640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3be80a3c40 0x7f3be80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:02.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.209+0000 7f3c09200640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3be80a3c40 0x7f3be80a6100 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4005ab0 tx=0x7f3bf4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:02.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.213+0000 7f3c02000640 1 -- 172.21.15.40:0/350854209 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3bec09d270 con 0x7f3c04105c60 2024-04-27T00:01:02.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.416+0000 7f3c0a600640 1 -- 172.21.15.40:0/350854209 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3c0402dd20 con 0x7f3c04105c60 2024-04-27T00:01:02.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.417+0000 7f3c02000640 1 -- 172.21.15.40:0/350854209 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f3bec09cb70 con 0x7f3c04105c60 2024-04-27T00:01:02.419 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-27T00:01:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.420+0000 7f3c0a600640 1 -- 172.21.15.40:0/350854209 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3be80a3c40 msgr2=0x7f3be80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.420+0000 7f3c0a600640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3be80a3c40 0x7f3be80a6100 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4005ab0 tx=0x7f3bf4005a20 comp rx=0 tx=0).stop 2024-04-27T00:01:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.420+0000 7f3c0a600640 1 -- 172.21.15.40:0/350854209 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c04105c60 msgr2=0x7f3c0419b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.420+0000 7f3c0a600640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c04105c60 0x7f3c0419b260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00e9f0 tx=0x7f3bec00edf0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.420+0000 7f3c0a600640 1 -- 172.21.15.40:0/350854209 shutdown_connections 2024-04-27T00:01:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.420+0000 7f3c0a600640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c04106620 0x7f3c041a2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.420+0000 7f3c0a600640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c04105c60 0x7f3c0419b260 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.420+0000 7f3c0a600640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3be80a3c40 0x7f3be80a6100 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.420+0000 7f3c0a600640 1 --2- 172.21.15.40:0/350854209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c04104a70 0x7f3c0419ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.420+0000 7f3c0a600640 1 -- 172.21.15.40:0/350854209 >> 172.21.15.40:0/350854209 conn(0x7f3c040fffe0 msgr2=0x7f3c04109030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:02.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.420+0000 7f3c0a600640 1 -- 172.21.15.40:0/350854209 shutdown_connections 2024-04-27T00:01:02.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.420+0000 7f3c0a600640 1 -- 172.21.15.40:0/350854209 wait complete. 2024-04-27T00:01:02.442 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-27T00:01:02.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.654+0000 7f6d16000640 1 -- 172.21.15.40:0/2528140381 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d101029a0 msgr2=0x7f6d10102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:02.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.654+0000 7f6d16000640 1 --2- 172.21.15.40:0/2528140381 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d101029a0 0x7f6d10102dc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009be0 tx=0x7f6d0002a6c0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.654+0000 7f6d16000640 1 -- 172.21.15.40:0/2528140381 shutdown_connections 2024-04-27T00:01:02.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.654+0000 7f6d16000640 1 --2- 172.21.15.40:0/2528140381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d10104550 0x7f6d1010ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.654+0000 7f6d16000640 1 --2- 172.21.15.40:0/2528140381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10103b90 0x7f6d10104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.654+0000 7f6d16000640 1 --2- 172.21.15.40:0/2528140381 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d101029a0 0x7f6d10102dc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.655+0000 7f6d16000640 1 -- 172.21.15.40:0/2528140381 >> 172.21.15.40:0/2528140381 conn(0x7f6d100fdef0 msgr2=0x7f6d10100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:02.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.655+0000 7f6d16000640 1 -- 172.21.15.40:0/2528140381 shutdown_connections 2024-04-27T00:01:02.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.655+0000 7f6d16000640 1 -- 172.21.15.40:0/2528140381 wait complete. 2024-04-27T00:01:02.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.656+0000 7f6d16000640 1 Processor -- start 2024-04-27T00:01:02.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.656+0000 7f6d16000640 1 -- start start 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.656+0000 7f6d16000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101029a0 0x7f6d100766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.656+0000 7f6d16000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d10103b90 0x7f6d10074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.656+0000 7f6d16000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10104550 0x7f6d10075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.656+0000 7f6d16000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1010e4a0 con 0x7f6d10104550 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.656+0000 7f6d16000640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1010e320 con 0x7f6d10103b90 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.656+0000 7f6d16000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1010e620 con 0x7f6d101029a0 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.656+0000 7f6d0fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10104550 0x7f6d10075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101029a0 0x7f6d100766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0ea00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d10103b90 0x7f6d10074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101029a0 0x7f6d100766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50452/0 (socket says 172.21.15.40:50452) 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10104550 0x7f6d10075260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35214/0 (socket says 172.21.15.40:35214) 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0f400640 1 -- 172.21.15.40:0/3449174703 learned_addr learned my addr 172.21.15.40:0/3449174703 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0ea00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d10103b90 0x7f6d10074d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:46624/0 (socket says 172.21.15.40:46624) 2024-04-27T00:01:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0f400640 1 -- 172.21.15.40:0/3449174703 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d10103b90 msgr2=0x7f6d10074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0f400640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d10103b90 0x7f6d10074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0f400640 1 -- 172.21.15.40:0/3449174703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10104550 msgr2=0x7f6d10075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0f400640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10104550 0x7f6d10075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0f400640 1 -- 172.21.15.40:0/3449174703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d10075af0 con 0x7f6d101029a0 2024-04-27T00:01:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0ea00640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d10103b90 0x7f6d10074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0fe00640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10104550 0x7f6d10075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.657+0000 7f6d0f400640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101029a0 0x7f6d100766e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6d00000f80 tx=0x7f6d00030d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.658+0000 7f6d07e00640 1 -- 172.21.15.40:0/3449174703 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d00041020 con 0x7f6d101029a0 2024-04-27T00:01:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.658+0000 7f6d07e00640 1 -- 172.21.15.40:0/3449174703 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d000318f0 con 0x7f6d101029a0 2024-04-27T00:01:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.658+0000 7f6d16000640 1 -- 172.21.15.40:0/3449174703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d101a2c60 con 0x7f6d101029a0 2024-04-27T00:01:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.658+0000 7f6d07e00640 1 -- 172.21.15.40:0/3449174703 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d00036040 con 0x7f6d101029a0 2024-04-27T00:01:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.658+0000 7f6d16000640 1 -- 172.21.15.40:0/3449174703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d101a31b0 con 0x7f6d101029a0 2024-04-27T00:01:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.659+0000 7f6d16000640 1 -- 172.21.15.40:0/3449174703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d10109d60 con 0x7f6d101029a0 2024-04-27T00:01:02.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.661+0000 7f6d07e00640 1 -- 172.21.15.40:0/3449174703 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6d00044020 con 0x7f6d101029a0 2024-04-27T00:01:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.662+0000 7f6d07e00640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6cf40a3c40 0x7f6cf40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.662+0000 7f6d07e00640 1 -- 172.21.15.40:0/3449174703 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f6d000f1930 con 0x7f6d101029a0 2024-04-27T00:01:02.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.663+0000 7f6d0ea00640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6cf40a3c40 0x7f6cf40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:02.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.663+0000 7f6d0ea00640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6cf40a3c40 0x7f6cf40a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6d10076450 tx=0x7f6cf8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:02.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.664+0000 7f6d07e00640 1 -- 172.21.15.40:0/3449174703 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6d0004b020 con 0x7f6d101029a0 2024-04-27T00:01:02.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:02 smithi040 ceph-mon[28095]: pgmap v390: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-27T00:01:02.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:02 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1367804018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:02.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:02 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/4118931230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:01:02.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:02 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/350854209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:02.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:02 smithi080 ceph-mon[32362]: pgmap v390: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-27T00:01:02.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:02 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1367804018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:02.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:02 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/4118931230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:01:02.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:02 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/350854209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:02.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:02 smithi103 ceph-mon[32952]: pgmap v390: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-27T00:01:02.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:02 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1367804018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:02.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:02 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/4118931230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:01:02.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:02 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/350854209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:02.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.873+0000 7f6d16000640 1 -- 172.21.15.40:0/3449174703 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6d1002dd20 con 0x7f6d101029a0 2024-04-27T00:01:02.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.874+0000 7f6d07e00640 1 -- 172.21.15.40:0/3449174703 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f6d000bde10 con 0x7f6d101029a0 2024-04-27T00:01:02.875 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-27T00:01:02.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.876+0000 7f6d16000640 1 -- 172.21.15.40:0/3449174703 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6cf40a3c40 msgr2=0x7f6cf40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:02.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.876+0000 7f6d16000640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6cf40a3c40 0x7f6cf40a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6d10076450 tx=0x7f6cf8009260 comp rx=0 tx=0).stop 2024-04-27T00:01:02.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.876+0000 7f6d16000640 1 -- 172.21.15.40:0/3449174703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101029a0 msgr2=0x7f6d100766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:02.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.876+0000 7f6d16000640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101029a0 0x7f6d100766e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6d00000f80 tx=0x7f6d00030d10 comp rx=0 tx=0).stop 2024-04-27T00:01:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.877+0000 7f6d16000640 1 -- 172.21.15.40:0/3449174703 shutdown_connections 2024-04-27T00:01:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.877+0000 7f6d16000640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10104550 0x7f6d10075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.877+0000 7f6d16000640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6d10103b90 0x7f6d10074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.877+0000 7f6d16000640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6cf40a3c40 0x7f6cf40a6100 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.877+0000 7f6d16000640 1 --2- 172.21.15.40:0/3449174703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101029a0 0x7f6d100766e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.877+0000 7f6d16000640 1 -- 172.21.15.40:0/3449174703 >> 172.21.15.40:0/3449174703 conn(0x7f6d100fdef0 msgr2=0x7f6d10106f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.877+0000 7f6d16000640 1 -- 172.21.15.40:0/3449174703 shutdown_connections 2024-04-27T00:01:02.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:02.877+0000 7f6d16000640 1 -- 172.21.15.40:0/3449174703 wait complete. 2024-04-27T00:01:02.898 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-27T00:01:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.104+0000 7f0a40e00640 1 -- 172.21.15.40:0/1275900118 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a3c10a980 msgr2=0x7f0a3c1112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.104+0000 7f0a40e00640 1 --2- 172.21.15.40:0/1275900118 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a3c10a980 0x7f0a3c1112a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0a34009be0 tx=0x7f0a3402a6c0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.104+0000 7f0a40e00640 1 -- 172.21.15.40:0/1275900118 shutdown_connections 2024-04-27T00:01:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.104+0000 7f0a40e00640 1 --2- 172.21.15.40:0/1275900118 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a3c10a980 0x7f0a3c1112a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.104+0000 7f0a40e00640 1 --2- 172.21.15.40:0/1275900118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a3c109fc0 0x7f0a3c10a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.104+0000 7f0a40e00640 1 --2- 172.21.15.40:0/1275900118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a3c108e70 0x7f0a3c109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.104+0000 7f0a40e00640 1 -- 172.21.15.40:0/1275900118 >> 172.21.15.40:0/1275900118 conn(0x7f0a3c077e00 msgr2=0x7f0a3c07a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:03.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.104+0000 7f0a40e00640 1 -- 172.21.15.40:0/1275900118 shutdown_connections 2024-04-27T00:01:03.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.104+0000 7f0a40e00640 1 -- 172.21.15.40:0/1275900118 wait complete. 2024-04-27T00:01:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.105+0000 7f0a40e00640 1 Processor -- start 2024-04-27T00:01:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a40e00640 1 -- start start 2024-04-27T00:01:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a40e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a3c108e70 0x7f0a3c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a40e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a3c109fc0 0x7f0a3c19f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a40e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a3c10a980 0x7f0a3c1a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a40e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c1149a0 con 0x7f0a3c10a980 2024-04-27T00:01:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a40e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c114820 con 0x7f0a3c109fc0 2024-04-27T00:01:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a40e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c114b20 con 0x7f0a3c108e70 2024-04-27T00:01:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a3be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a3c10a980 0x7f0a3c1a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a3b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a3c108e70 0x7f0a3c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:03.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a3aa00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a3c109fc0 0x7f0a3c19f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a3b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a3c108e70 0x7f0a3c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50458/0 (socket says 172.21.15.40:50458) 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a3aa00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a3c109fc0 0x7f0a3c19f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:46644/0 (socket says 172.21.15.40:46644) 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.106+0000 7f0a3b400640 1 -- 172.21.15.40:0/416649829 learned_addr learned my addr 172.21.15.40:0/416649829 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.107+0000 7f0a3aa00640 1 -- 172.21.15.40:0/416649829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a3c108e70 msgr2=0x7f0a3c19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.107+0000 7f0a3aa00640 1 --2- 172.21.15.40:0/416649829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a3c108e70 0x7f0a3c19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.107+0000 7f0a3aa00640 1 -- 172.21.15.40:0/416649829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a3c10a980 msgr2=0x7f0a3c1a65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.107+0000 7f0a3aa00640 1 --2- 172.21.15.40:0/416649829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a3c10a980 0x7f0a3c1a65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.107+0000 7f0a3aa00640 1 -- 172.21.15.40:0/416649829 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3c1a6c60 con 0x7f0a3c109fc0 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.107+0000 7f0a3aa00640 1 --2- 172.21.15.40:0/416649829 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a3c109fc0 0x7f0a3c19f580 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0a2800e9c0 tx=0x7f0a2800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.107+0000 7f0a33e00640 1 -- 172.21.15.40:0/416649829 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2800ccc0 con 0x7f0a3c109fc0 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.107+0000 7f0a33e00640 1 -- 172.21.15.40:0/416649829 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a280035f0 con 0x7f0a3c109fc0 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.107+0000 7f0a40e00640 1 -- 172.21.15.40:0/416649829 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a3c1a6f50 con 0x7f0a3c109fc0 2024-04-27T00:01:03.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.107+0000 7f0a33e00640 1 -- 172.21.15.40:0/416649829 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2800ce60 con 0x7f0a3c109fc0 2024-04-27T00:01:03.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.107+0000 7f0a40e00640 1 -- 172.21.15.40:0/416649829 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a3c1a7460 con 0x7f0a3c109fc0 2024-04-27T00:01:03.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.108+0000 7f0a40e00640 1 -- 172.21.15.40:0/416649829 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a3c110120 con 0x7f0a3c109fc0 2024-04-27T00:01:03.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.111+0000 7f0a33e00640 1 -- 172.21.15.40:0/416649829 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0a28004000 con 0x7f0a3c109fc0 2024-04-27T00:01:03.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.113+0000 7f0a33e00640 1 --2- 172.21.15.40:0/416649829 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0a200a3cc0 0x7f0a200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:03.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.113+0000 7f0a33e00640 1 -- 172.21.15.40:0/416649829 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f0a280d0c60 con 0x7f0a3c109fc0 2024-04-27T00:01:03.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.113+0000 7f0a3b400640 1 --2- 172.21.15.40:0/416649829 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0a200a3cc0 0x7f0a200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:03.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.114+0000 7f0a3b400640 1 --2- 172.21.15.40:0/416649829 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0a200a3cc0 0x7f0a200a6180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0058c0 tx=0x7f0a2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:03.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.117+0000 7f0a33e00640 1 -- 172.21.15.40:0/416649829 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0a2809d140 con 0x7f0a3c109fc0 2024-04-27T00:01:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.329+0000 7f0a40e00640 1 -- 172.21.15.40:0/416649829 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0a3c02dcf0 con 0x7f0a3c109fc0 2024-04-27T00:01:03.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.330+0000 7f0a33e00640 1 -- 172.21.15.40:0/416649829 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f0a2809ca40 con 0x7f0a3c109fc0 2024-04-27T00:01:03.332 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-27T00:01:03.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.334+0000 7f0a40e00640 1 -- 172.21.15.40:0/416649829 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0a200a3cc0 msgr2=0x7f0a200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:03.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.334+0000 7f0a40e00640 1 --2- 172.21.15.40:0/416649829 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0a200a3cc0 0x7f0a200a6180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0058c0 tx=0x7f0a2c009260 comp rx=0 tx=0).stop 2024-04-27T00:01:03.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.334+0000 7f0a40e00640 1 -- 172.21.15.40:0/416649829 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a3c109fc0 msgr2=0x7f0a3c19f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:03.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.334+0000 7f0a40e00640 1 --2- 172.21.15.40:0/416649829 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a3c109fc0 0x7f0a3c19f580 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0a2800e9c0 tx=0x7f0a2800ee80 comp rx=0 tx=0).stop 2024-04-27T00:01:03.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.334+0000 7f0a40e00640 1 -- 172.21.15.40:0/416649829 shutdown_connections 2024-04-27T00:01:03.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.334+0000 7f0a40e00640 1 --2- 172.21.15.40:0/416649829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a3c10a980 0x7f0a3c1a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.334+0000 7f0a40e00640 1 --2- 172.21.15.40:0/416649829 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a3c109fc0 0x7f0a3c19f580 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.334+0000 7f0a40e00640 1 --2- 172.21.15.40:0/416649829 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0a200a3cc0 0x7f0a200a6180 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.334+0000 7f0a40e00640 1 --2- 172.21.15.40:0/416649829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a3c108e70 0x7f0a3c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.334+0000 7f0a40e00640 1 -- 172.21.15.40:0/416649829 >> 172.21.15.40:0/416649829 conn(0x7f0a3c077e00 msgr2=0x7f0a3c10d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.334+0000 7f0a40e00640 1 -- 172.21.15.40:0/416649829 shutdown_connections 2024-04-27T00:01:03.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.335+0000 7f0a40e00640 1 -- 172.21.15.40:0/416649829 wait complete. 2024-04-27T00:01:03.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-27T00:01:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.576+0000 7fdf88c00640 1 -- 172.21.15.40:0/1166315378 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdf84108d90 msgr2=0x7fdf841091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.576+0000 7fdf88c00640 1 --2- 172.21.15.40:0/1166315378 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdf84108d90 0x7fdf841091b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009860 tx=0x7fdf7402a340 comp rx=0 tx=0).stop 2024-04-27T00:01:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.576+0000 7fdf88c00640 1 -- 172.21.15.40:0/1166315378 shutdown_connections 2024-04-27T00:01:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.576+0000 7fdf88c00640 1 --2- 172.21.15.40:0/1166315378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf8410a940 0x7fdf841111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.576+0000 7fdf88c00640 1 --2- 172.21.15.40:0/1166315378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf84109f80 0x7fdf8410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.576+0000 7fdf88c00640 1 --2- 172.21.15.40:0/1166315378 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdf84108d90 0x7fdf841091b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.576+0000 7fdf88c00640 1 -- 172.21.15.40:0/1166315378 >> 172.21.15.40:0/1166315378 conn(0x7fdf84077d30 msgr2=0x7fdf8407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:03.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.576+0000 7fdf88c00640 1 -- 172.21.15.40:0/1166315378 shutdown_connections 2024-04-27T00:01:03.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.576+0000 7fdf88c00640 1 -- 172.21.15.40:0/1166315378 wait complete. 2024-04-27T00:01:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.577+0000 7fdf88c00640 1 Processor -- start 2024-04-27T00:01:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.578+0000 7fdf88c00640 1 -- start start 2024-04-27T00:01:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.578+0000 7fdf88c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdf84109f80 0x7fdf8419f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.578+0000 7fdf88c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8410a940 0x7fdf8419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.578+0000 7fdf88c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf841a4760 0x7fdf841a6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.578+0000 7fdf88c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf84114890 con 0x7fdf8410a940 2024-04-27T00:01:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.578+0000 7fdf88c00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fdf84114710 con 0x7fdf84109f80 2024-04-27T00:01:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.578+0000 7fdf88c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf84114a10 con 0x7fdf841a4760 2024-04-27T00:01:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.578+0000 7fdf7aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8410a940 0x7fdf8419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.578+0000 7fdf7aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8410a940 0x7fdf8419f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35268/0 (socket says 172.21.15.40:35268) 2024-04-27T00:01:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.578+0000 7fdf7aa00640 1 -- 172.21.15.40:0/326079888 learned_addr learned my addr 172.21.15.40:0/326079888 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.578+0000 7fdf83400640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdf84109f80 0x7fdf8419f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.579+0000 7fdf83e00640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf841a4760 0x7fdf841a6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.579+0000 7fdf7aa00640 1 -- 172.21.15.40:0/326079888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf841a4760 msgr2=0x7fdf841a6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.579+0000 7fdf7aa00640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf841a4760 0x7fdf841a6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.579+0000 7fdf7aa00640 1 -- 172.21.15.40:0/326079888 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdf84109f80 msgr2=0x7fdf8419f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.579+0000 7fdf7aa00640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdf84109f80 0x7fdf8419f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.579+0000 7fdf7aa00640 1 -- 172.21.15.40:0/326079888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf841a71c0 con 0x7fdf8410a940 2024-04-27T00:01:03.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.579+0000 7fdf83e00640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf841a4760 0x7fdf841a6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:03.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.579+0000 7fdf7aa00640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8410a940 0x7fdf8419f750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c00e9c0 tx=0x7fdf6c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:03.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.579+0000 7fdf80c00640 1 -- 172.21.15.40:0/326079888 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c00ccc0 con 0x7fdf8410a940 2024-04-27T00:01:03.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.580+0000 7fdf80c00640 1 -- 172.21.15.40:0/326079888 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf6c0054d0 con 0x7fdf8410a940 2024-04-27T00:01:03.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.580+0000 7fdf88c00640 1 -- 172.21.15.40:0/326079888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf841a74b0 con 0x7fdf8410a940 2024-04-27T00:01:03.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.580+0000 7fdf80c00640 1 -- 172.21.15.40:0/326079888 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c010430 con 0x7fdf8410a940 2024-04-27T00:01:03.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.580+0000 7fdf88c00640 1 -- 172.21.15.40:0/326079888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf841a7a00 con 0x7fdf8410a940 2024-04-27T00:01:03.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.581+0000 7fdf88c00640 1 -- 172.21.15.40:0/326079888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf84110310 con 0x7fdf8410a940 2024-04-27T00:01:03.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.585+0000 7fdf80c00640 1 -- 172.21.15.40:0/326079888 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdf6c004f80 con 0x7fdf8410a940 2024-04-27T00:01:03.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.586+0000 7fdf80c00640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdf680a3c40 0x7fdf680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:03.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.586+0000 7fdf80c00640 1 -- 172.21.15.40:0/326079888 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fdf6c0d1140 con 0x7fdf8410a940 2024-04-27T00:01:03.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.586+0000 7fdf83400640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdf680a3c40 0x7fdf680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:03.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.587+0000 7fdf83400640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdf680a3c40 0x7fdf680a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009780 tx=0x7fdf74030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:03.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.589+0000 7fdf80c00640 1 -- 172.21.15.40:0/326079888 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdf6c09d620 con 0x7fdf8410a940 2024-04-27T00:01:03.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:03 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3449174703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:03.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:03 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/416649829' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:03.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:03 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3449174703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:03.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:03 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/416649829' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.836+0000 7fdf88c00640 1 -- 172.21.15.40:0/326079888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf8402dad0 con 0x7fdf8410a940 2024-04-27T00:01:03.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.837+0000 7fdf80c00640 1 -- 172.21.15.40:0/326079888 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11872 (secure 0 0 0) 0x7fdf6c09cf20 con 0x7fdf8410a940 2024-04-27T00:01:03.839 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:03.839 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/2265538751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":2265538751},{"type":"v1","addr":"172.21.15.40:6841","nonce":2265538751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:58:50.616677+0000","modified":"2024-04-26T23:59:25.374905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24523":{"gid":24523,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.80:6839/3536638444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6838","nonce":3536638444},{"type":"v1","addr":"172.21.15.80:6839","nonce":3536638444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24559,24560,24590]},"id":1}]} 2024-04-27T00:01:03.839 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 23 2024-04-27T00:01:03.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.839+0000 7fdf88c00640 1 -- 172.21.15.40:0/326079888 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdf680a3c40 msgr2=0x7fdf680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.839+0000 7fdf88c00640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdf680a3c40 0x7fdf680a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009780 tx=0x7fdf74030450 comp rx=0 tx=0).stop 2024-04-27T00:01:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.839+0000 7fdf88c00640 1 -- 172.21.15.40:0/326079888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8410a940 msgr2=0x7fdf8419f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.839+0000 7fdf88c00640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8410a940 0x7fdf8419f750 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c00e9c0 tx=0x7fdf6c00ee80 comp rx=0 tx=0).stop 2024-04-27T00:01:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.839+0000 7fdf88c00640 1 -- 172.21.15.40:0/326079888 shutdown_connections 2024-04-27T00:01:03.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.839+0000 7fdf88c00640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdf680a3c40 0x7fdf680a6100 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.839+0000 7fdf88c00640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf841a4760 0x7fdf841a6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.839+0000 7fdf88c00640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8410a940 0x7fdf8419f750 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.839+0000 7fdf88c00640 1 --2- 172.21.15.40:0/326079888 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdf84109f80 0x7fdf8419f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.839+0000 7fdf88c00640 1 -- 172.21.15.40:0/326079888 >> 172.21.15.40:0/326079888 conn(0x7fdf84077d30 msgr2=0x7fdf8410d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.840+0000 7fdf88c00640 1 -- 172.21.15.40:0/326079888 shutdown_connections 2024-04-27T00:01:03.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:03.840+0000 7fdf88c00640 1 -- 172.21.15.40:0/326079888 wait complete. 2024-04-27T00:01:03.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:03 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3449174703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:03.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:03 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/416649829' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:03.861 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T00:01:04.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.077+0000 7f6b9be00640 1 -- 172.21.15.40:0/74538998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c1049b0 msgr2=0x7f6b9c104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:04.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.077+0000 7f6b9be00640 1 --2- 172.21.15.40:0/74538998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c1049b0 0x7f6b9c104dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009900 tx=0x7f6b8c02a3f0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.077+0000 7f6b9be00640 1 -- 172.21.15.40:0/74538998 shutdown_connections 2024-04-27T00:01:04.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.077+0000 7f6b9be00640 1 --2- 172.21.15.40:0/74538998 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b9c106560 0x7f6b9c10cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.077+0000 7f6b9be00640 1 --2- 172.21.15.40:0/74538998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b9c105ba0 0x7f6b9c106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.077+0000 7f6b9be00640 1 --2- 172.21.15.40:0/74538998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c1049b0 0x7f6b9c104dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.077+0000 7f6b9be00640 1 -- 172.21.15.40:0/74538998 >> 172.21.15.40:0/74538998 conn(0x7f6b9c0fff40 msgr2=0x7f6b9c102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:04.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.077+0000 7f6b9be00640 1 -- 172.21.15.40:0/74538998 shutdown_connections 2024-04-27T00:01:04.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.077+0000 7f6b9be00640 1 -- 172.21.15.40:0/74538998 wait complete. 2024-04-27T00:01:04.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.078+0000 7f6b9be00640 1 Processor -- start 2024-04-27T00:01:04.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.078+0000 7f6b9be00640 1 -- start start 2024-04-27T00:01:04.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b9c1049b0 0x7f6b9c19abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:04.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b9c105ba0 0x7f6b9c19b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:04.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c106560 0x7f6b9c1a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:04.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c1104b0 con 0x7f6b9c106560 2024-04-27T00:01:04.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9be00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c110330 con 0x7f6b9c1049b0 2024-04-27T00:01:04.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c110630 con 0x7f6b9c105ba0 2024-04-27T00:01:04.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c106560 0x7f6b9c1a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:04.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9aa00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b9c1049b0 0x7f6b9c19abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c106560 0x7f6b9c1a2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35276/0 (socket says 172.21.15.40:35276) 2024-04-27T00:01:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9b400640 1 -- 172.21.15.40:0/2193462198 learned_addr learned my addr 172.21.15.40:0/2193462198 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9a000640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b9c105ba0 0x7f6b9c19b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9b400640 1 -- 172.21.15.40:0/2193462198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b9c105ba0 msgr2=0x7f6b9c19b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9b400640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b9c105ba0 0x7f6b9c19b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9b400640 1 -- 172.21.15.40:0/2193462198 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b9c1049b0 msgr2=0x7f6b9c19abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9b400640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b9c1049b0 0x7f6b9c19abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9b400640 1 -- 172.21.15.40:0/2193462198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9c1a27d0 con 0x7f6b9c106560 2024-04-27T00:01:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.079+0000 7f6b9aa00640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b9c1049b0 0x7f6b9c19abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.080+0000 7f6b9b400640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c106560 0x7f6b9c1a2160 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400efd0 tx=0x7f6b9400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.080+0000 7f6b93400640 1 -- 172.21.15.40:0/2193462198 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94003b70 con 0x7f6b9c106560 2024-04-27T00:01:04.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.080+0000 7f6b9be00640 1 -- 172.21.15.40:0/2193462198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9c1a2ac0 con 0x7f6b9c106560 2024-04-27T00:01:04.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.080+0000 7f6b93400640 1 -- 172.21.15.40:0/2193462198 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b94003d10 con 0x7f6b9c106560 2024-04-27T00:01:04.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.080+0000 7f6b93400640 1 -- 172.21.15.40:0/2193462198 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b940108a0 con 0x7f6b9c106560 2024-04-27T00:01:04.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.080+0000 7f6b9be00640 1 -- 172.21.15.40:0/2193462198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9c1a2fc0 con 0x7f6b9c106560 2024-04-27T00:01:04.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.082+0000 7f6b9be00640 1 -- 172.21.15.40:0/2193462198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b9c10bcf0 con 0x7f6b9c106560 2024-04-27T00:01:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.083+0000 7f6b93400640 1 -- 172.21.15.40:0/2193462198 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6b94004020 con 0x7f6b9c106560 2024-04-27T00:01:04.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.085+0000 7f6b93400640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b800a3c40 0x7f6b800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:04.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.085+0000 7f6b93400640 1 -- 172.21.15.40:0/2193462198 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f6b940d0e40 con 0x7f6b9c106560 2024-04-27T00:01:04.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.085+0000 7f6b9aa00640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b800a3c40 0x7f6b800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:04.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.086+0000 7f6b9aa00640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b800a3c40 0x7f6b800a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009230 tx=0x7f6b8c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:04.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.089+0000 7f6b93400640 1 -- 172.21.15.40:0/2193462198 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6b9409d320 con 0x7f6b9c106560 2024-04-27T00:01:04.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.298+0000 7f6b9be00640 1 -- 172.21.15.40:0/2193462198 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b9c02dcf0 con 0x7f6b9c106560 2024-04-27T00:01:04.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.300+0000 7f6b93400640 1 -- 172.21.15.40:0/2193462198 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24531 (secure 0 0 0) 0x7f6b9409cc20 con 0x7f6b9c106560 2024-04-27T00:01:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:04.302 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-27T00:00:46.365303+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9},{"from":5,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T00:01:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.303+0000 7f6b9be00640 1 -- 172.21.15.40:0/2193462198 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b800a3c40 msgr2=0x7f6b800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.303+0000 7f6b9be00640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b800a3c40 0x7f6b800a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009230 tx=0x7f6b8c0314a0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.303+0000 7f6b9be00640 1 -- 172.21.15.40:0/2193462198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c106560 msgr2=0x7f6b9c1a2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.303+0000 7f6b9be00640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c106560 0x7f6b9c1a2160 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400efd0 tx=0x7f6b9400c6a0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.303+0000 7f6b9be00640 1 -- 172.21.15.40:0/2193462198 shutdown_connections 2024-04-27T00:01:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.303+0000 7f6b9be00640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c106560 0x7f6b9c1a2160 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.303+0000 7f6b9be00640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b800a3c40 0x7f6b800a6100 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.303+0000 7f6b9be00640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b9c105ba0 0x7f6b9c19b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.303+0000 7f6b9be00640 1 --2- 172.21.15.40:0/2193462198 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b9c1049b0 0x7f6b9c19abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.303+0000 7f6b9be00640 1 -- 172.21.15.40:0/2193462198 >> 172.21.15.40:0/2193462198 conn(0x7f6b9c0fff40 msgr2=0x7f6b9c108ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.304+0000 7f6b9be00640 1 -- 172.21.15.40:0/2193462198 shutdown_connections 2024-04-27T00:01:04.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.304+0000 7f6b9be00640 1 -- 172.21.15.40:0/2193462198 wait complete. 2024-04-27T00:01:04.326 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-27T00:01:04.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.539+0000 7f1581800640 1 -- 172.21.15.40:0/1655051318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f157c103940 msgr2=0x7f157c103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:04.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.539+0000 7f1581800640 1 --2- 172.21.15.40:0/1655051318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f157c103940 0x7f157c103dc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1564009900 tx=0x7f156402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.539+0000 7f1581800640 1 -- 172.21.15.40:0/1655051318 shutdown_connections 2024-04-27T00:01:04.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.539+0000 7f1581800640 1 --2- 172.21.15.40:0/1655051318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f157c104300 0x7f157c10ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.539+0000 7f1581800640 1 --2- 172.21.15.40:0/1655051318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f157c103940 0x7f157c103dc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.539+0000 7f1581800640 1 --2- 172.21.15.40:0/1655051318 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f157c102750 0x7f157c102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.539+0000 7f1581800640 1 -- 172.21.15.40:0/1655051318 >> 172.21.15.40:0/1655051318 conn(0x7f157c0fdcc0 msgr2=0x7f157c100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:04.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.539+0000 7f1581800640 1 -- 172.21.15.40:0/1655051318 shutdown_connections 2024-04-27T00:01:04.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.539+0000 7f1581800640 1 -- 172.21.15.40:0/1655051318 wait complete. 2024-04-27T00:01:04.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.540+0000 7f1581800640 1 Processor -- start 2024-04-27T00:01:04.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f1581800640 1 -- start start 2024-04-27T00:01:04.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f1581800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f157c102750 0x7f157c19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:04.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f1581800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f157c103940 0x7f157c19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:04.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f1581800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f157c104300 0x7f157c1a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:04.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f1581800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f157c10e2d0 con 0x7f157c103940 2024-04-27T00:01:04.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f1581800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f157c10e150 con 0x7f157c102750 2024-04-27T00:01:04.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f1581800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f157c10e450 con 0x7f157c104300 2024-04-27T00:01:04.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f1573e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f157c103940 0x7f157c19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:04.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f1580e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f157c104300 0x7f157c1a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:04.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f157be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f157c102750 0x7f157c19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f1580e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f157c104300 0x7f157c1a2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50516/0 (socket says 172.21.15.40:50516) 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f1580e00640 1 -- 172.21.15.40:0/1355274955 learned_addr learned my addr 172.21.15.40:0/1355274955 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.541+0000 7f1573e00640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f157c103940 0x7f157c19b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35306/0 (socket says 172.21.15.40:35306) 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.542+0000 7f1580e00640 1 -- 172.21.15.40:0/1355274955 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f157c102750 msgr2=0x7f157c19ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.542+0000 7f1580e00640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f157c102750 0x7f157c19ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.542+0000 7f1580e00640 1 -- 172.21.15.40:0/1355274955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f157c103940 msgr2=0x7f157c19b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.542+0000 7f1580e00640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f157c103940 0x7f157c19b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.542+0000 7f1580e00640 1 -- 172.21.15.40:0/1355274955 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157c1a2780 con 0x7f157c104300 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.542+0000 7f157be00640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f157c102750 0x7f157c19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.542+0000 7f1580e00640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f157c104300 0x7f157c1a2110 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f157400eb10 tx=0x7f157400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.542+0000 7f1579600640 1 -- 172.21.15.40:0/1355274955 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1574004b20 con 0x7f157c104300 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.542+0000 7f1581800640 1 -- 172.21.15.40:0/1355274955 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f157c1a2a70 con 0x7f157c104300 2024-04-27T00:01:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.542+0000 7f1579600640 1 -- 172.21.15.40:0/1355274955 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1574004cc0 con 0x7f157c104300 2024-04-27T00:01:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.542+0000 7f1579600640 1 -- 172.21.15.40:0/1355274955 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1574010690 con 0x7f157c104300 2024-04-27T00:01:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.543+0000 7f1581800640 1 -- 172.21.15.40:0/1355274955 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f157c1a2fe0 con 0x7f157c104300 2024-04-27T00:01:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.544+0000 7f1581800640 1 -- 172.21.15.40:0/1355274955 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f157c1a32d0 con 0x7f157c104300 2024-04-27T00:01:04.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.546+0000 7f1579600640 1 -- 172.21.15.40:0/1355274955 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1574005020 con 0x7f157c104300 2024-04-27T00:01:04.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.547+0000 7f1579600640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f15600a3c40 0x7f15600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:04.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.547+0000 7f1579600640 1 -- 172.21.15.40:0/1355274955 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f15740d0e70 con 0x7f157c104300 2024-04-27T00:01:04.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.547+0000 7f157be00640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f15600a3c40 0x7f15600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:04.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.548+0000 7f157be00640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f15600a3c40 0x7f15600a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f157c1037a0 tx=0x7f156c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.552+0000 7f1579600640 1 -- 172.21.15.40:0/1355274955 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f157409d350 con 0x7f157c104300 2024-04-27T00:01:04.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:04 smithi040 ceph-mon[28095]: pgmap v391: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-27T00:01:04.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:04 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/326079888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:04.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:04 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2193462198' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:04.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:04 smithi080 ceph-mon[32362]: pgmap v391: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-27T00:01:04.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:04 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/326079888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:04.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:04 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2193462198' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:04.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.798+0000 7f1581800640 1 -- 172.21.15.40:0/1355274955 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f157c1a34a0 con 0x7f157c104300 2024-04-27T00:01:04.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.799+0000 7f1579600640 1 -- 172.21.15.40:0/1355274955 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11872 (secure 0 0 0) 0x7f157409cc50 con 0x7f157c104300 2024-04-27T00:01:04.801 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:04.801 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/2265538751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":2265538751},{"type":"v1","addr":"172.21.15.40:6841","nonce":2265538751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:58:50.616677+0000","modified":"2024-04-26T23:59:25.374905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24523":{"gid":24523,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.80:6839/3536638444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6838","nonce":3536638444},{"type":"v1","addr":"172.21.15.80:6839","nonce":3536638444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24559,24560,24590]},"id":1}]} 2024-04-27T00:01:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 23 2024-04-27T00:01:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.802+0000 7f1581800640 1 -- 172.21.15.40:0/1355274955 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f15600a3c40 msgr2=0x7f15600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.802+0000 7f1581800640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f15600a3c40 0x7f15600a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f157c1037a0 tx=0x7f156c005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.802+0000 7f1581800640 1 -- 172.21.15.40:0/1355274955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f157c104300 msgr2=0x7f157c1a2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.802+0000 7f1581800640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f157c104300 0x7f157c1a2110 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f157400eb10 tx=0x7f157400c2d0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.802+0000 7f1581800640 1 -- 172.21.15.40:0/1355274955 shutdown_connections 2024-04-27T00:01:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.802+0000 7f1581800640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f15600a3c40 0x7f15600a6100 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.802+0000 7f1581800640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f157c104300 0x7f157c1a2110 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.802+0000 7f1581800640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f157c103940 0x7f157c19b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.802+0000 7f1581800640 1 --2- 172.21.15.40:0/1355274955 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f157c102750 0x7f157c19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.802+0000 7f1581800640 1 -- 172.21.15.40:0/1355274955 >> 172.21.15.40:0/1355274955 conn(0x7f157c0fdcc0 msgr2=0x7f157c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:04.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.802+0000 7f1581800640 1 -- 172.21.15.40:0/1355274955 shutdown_connections 2024-04-27T00:01:04.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:04.802+0000 7f1581800640 1 -- 172.21.15.40:0/1355274955 wait complete. 2024-04-27T00:01:04.822 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T23:58:50.616677+0000', 'modified': '2024-04-26T23:59:25.374905+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24590, 'mds_1': 24560, 'mds_2': 24559}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24523': {'gid': 24523, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.103:6835/2596993377', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6834', 'nonce': 2596993377}, {'type': 'v1', 'addr': '172.21.15.103:6835', 'nonce': 2596993377}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.80:6837/4116866934', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6836', 'nonce': 4116866934}, {'type': 'v1', 'addr': '172.21.15.80:6837', 'nonce': 4116866934}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.103:6837/2206925949', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 2206925949}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 2206925949}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24559': {'gid': 24559, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.103:6839/1527392010', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6838', 'nonce': 1527392010}, {'type': 'v1', 'addr': '172.21.15.103:6839', 'nonce': 1527392010}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.40:6839/2205335602', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 2205335602}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 2205335602}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.80:6839/3536638444', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6838', 'nonce': 3536638444}, {'type': 'v1', 'addr': '172.21.15.80:6839', 'nonce': 3536638444}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24590, 'qdb_cluster': [24559, 24560, 24590]} 2024-04-27T00:01:04.822 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-27T00:01:04.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-27T00:01:04.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:04 smithi103 ceph-mon[32952]: pgmap v391: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-27T00:01:04.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:04 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/326079888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:04.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:04 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2193462198' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.042+0000 7f1bc6a00640 1 -- 172.21.15.40:0/818333831 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1bc0101680 msgr2=0x7f1bc0101aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.042+0000 7f1bc6a00640 1 --2- 172.21.15.40:0/818333831 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1bc0101680 0x7f1bc0101aa0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009860 tx=0x7f1bb402a340 comp rx=0 tx=0).stop 2024-04-27T00:01:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.042+0000 7f1bc6a00640 1 -- 172.21.15.40:0/818333831 shutdown_connections 2024-04-27T00:01:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.042+0000 7f1bc6a00640 1 --2- 172.21.15.40:0/818333831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bc0103230 0x7f1bc0109ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.042+0000 7f1bc6a00640 1 --2- 172.21.15.40:0/818333831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bc0102870 0x7f1bc0102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.042+0000 7f1bc6a00640 1 --2- 172.21.15.40:0/818333831 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1bc0101680 0x7f1bc0101aa0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.042+0000 7f1bc6a00640 1 -- 172.21.15.40:0/818333831 >> 172.21.15.40:0/818333831 conn(0x7f1bc00fcc10 msgr2=0x7f1bc00ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.042+0000 7f1bc6a00640 1 -- 172.21.15.40:0/818333831 shutdown_connections 2024-04-27T00:01:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.042+0000 7f1bc6a00640 1 -- 172.21.15.40:0/818333831 wait complete. 2024-04-27T00:01:05.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.043+0000 7f1bc6a00640 1 Processor -- start 2024-04-27T00:01:05.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.043+0000 7f1bc6a00640 1 -- start start 2024-04-27T00:01:05.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.043+0000 7f1bc6a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bc0101680 0x7f1bc019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.043+0000 7f1bc6a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1bc0102870 0x7f1bc019b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.043+0000 7f1bc6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bc0103230 0x7f1bc01a2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.043+0000 7f1bc6a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc010d180 con 0x7f1bc0103230 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.043+0000 7f1bc6a00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc010d000 con 0x7f1bc0102870 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.043+0000 7f1bc6a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc010d300 con 0x7f1bc0101680 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bc0103230 0x7f1bc01a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc5600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bc0101680 0x7f1bc019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc5600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bc0101680 0x7f1bc019abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50546/0 (socket says 172.21.15.40:50546) 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc4c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1bc0102870 0x7f1bc019b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bc0103230 0x7f1bc01a2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35316/0 (socket says 172.21.15.40:35316) 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc5600640 1 -- 172.21.15.40:0/126997787 learned_addr learned my addr 172.21.15.40:0/126997787 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc5600640 1 -- 172.21.15.40:0/126997787 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1bc0102870 msgr2=0x7f1bc019b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc5600640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1bc0102870 0x7f1bc019b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc5600640 1 -- 172.21.15.40:0/126997787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bc0103230 msgr2=0x7f1bc01a2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc5600640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bc0103230 0x7f1bc01a2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc5600640 1 -- 172.21.15.40:0/126997787 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc01a27f0 con 0x7f1bc0101680 2024-04-27T00:01:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc4c00640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1bc0102870 0x7f1bc019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc6000640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bc0103230 0x7f1bc01a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.044+0000 7f1bc5600640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bc0101680 0x7f1bc019abd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1bb40057c0 tx=0x7f1bb4032da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.045+0000 7f1bbe000640 1 -- 172.21.15.40:0/126997787 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4037040 con 0x7f1bc0101680 2024-04-27T00:01:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.045+0000 7f1bbe000640 1 -- 172.21.15.40:0/126997787 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bb4030bd0 con 0x7f1bc0101680 2024-04-27T00:01:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.045+0000 7f1bc6a00640 1 -- 172.21.15.40:0/126997787 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bc01a2a80 con 0x7f1bc0101680 2024-04-27T00:01:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.045+0000 7f1bbe000640 1 -- 172.21.15.40:0/126997787 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4005280 con 0x7f1bc0101680 2024-04-27T00:01:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.045+0000 7f1bc6a00640 1 -- 172.21.15.40:0/126997787 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bc01a2f10 con 0x7f1bc0101680 2024-04-27T00:01:05.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.048+0000 7f1bbe000640 1 -- 172.21.15.40:0/126997787 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1bb4043020 con 0x7f1bc0101680 2024-04-27T00:01:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.048+0000 7f1bcadb8740 1 Processor -- start 2024-04-27T00:01:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.048+0000 7f1bcadb8740 1 -- start start 2024-04-27T00:01:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.049+0000 7f1bcadb8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df32c0 0x55d194df36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.049+0000 7f1bcadb8740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194df87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.049+0000 7f1bcadb8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df8e20 0x55d194dfb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.049+0000 7f1bcadb8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d194d57600 con 0x55d194df32c0 2024-04-27T00:01:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.049+0000 7f1bcadb8740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55d194cf4100 con 0x55d194df3cb0 2024-04-27T00:01:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.049+0000 7f1bcadb8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d194d535c0 con 0x55d194df8e20 2024-04-27T00:01:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.049+0000 7f1bc6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df32c0 0x55d194df36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.049+0000 7f1bc6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df32c0 0x55d194df36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35326/0 (socket says 172.21.15.40:35326) 2024-04-27T00:01:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.049+0000 7f1bc6000640 1 -- 172.21.15.40:0/377985982 learned_addr learned my addr 172.21.15.40:0/377985982 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.049+0000 7f1bc5600640 1 --2- 172.21.15.40:0/377985982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194df87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.049+0000 7f1bc4c00640 1 --2- 172.21.15.40:0/377985982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df8e20 0x55d194dfb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bc6000640 1 -- 172.21.15.40:0/377985982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df8e20 msgr2=0x55d194dfb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bc6000640 1 --2- 172.21.15.40:0/377985982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df8e20 0x55d194dfb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bc6000640 1 -- 172.21.15.40:0/377985982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 msgr2=0x55d194df87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bc6000640 1 --2- 172.21.15.40:0/377985982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194df87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bc6000640 1 -- 172.21.15.40:0/377985982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d194a692d0 con 0x55d194df32c0 2024-04-27T00:01:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bc4c00640 1 --2- 172.21.15.40:0/377985982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df8e20 0x55d194dfb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bbe000640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1ba80a3c40 0x7f1ba80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bc6000640 1 --2- 172.21.15.40:0/377985982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df32c0 0x55d194df36e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80117e0 tx=0x7f1bb8011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e620d44f523ce1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bbe000640 1 -- 172.21.15.40:0/126997787 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f1bb40efd70 con 0x7f1bc0101680 2024-04-27T00:01:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bc5600640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1ba80a3c40 0x7f1ba80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bb2600640 1 -- 172.21.15.40:0/377985982 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8011310 con 0x55d194df32c0 2024-04-27T00:01:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bb2600640 1 -- 172.21.15.40:0/377985982 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bb8003e20 con 0x55d194df32c0 2024-04-27T00:01:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.050+0000 7f1bb2600640 1 -- 172.21.15.40:0/377985982 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb80044f0 con 0x55d194df32c0 2024-04-27T00:01:05.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.051+0000 7f1bcadb8740 1 -- 172.21.15.40:0/377985982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df32c0 msgr2=0x55d194df36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.051+0000 7f1bc5600640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1ba80a3c40 0x7f1ba80a6100 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55d194df40b0 tx=0x7f1bb40f3820 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.051+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/377985982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df32c0 0x55d194df36e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80117e0 tx=0x7f1bb8011ca0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.051+0000 7f1bcadb8740 1 -- 172.21.15.40:0/377985982 shutdown_connections 2024-04-27T00:01:05.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.051+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/377985982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df8e20 0x55d194dfb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.051+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/377985982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194df87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.051+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/377985982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df32c0 0x55d194df36e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.051+0000 7f1bcadb8740 1 -- 172.21.15.40:0/377985982 >> 172.21.15.40:0/377985982 conn(0x55d194d87d10 msgr2=0x55d194deed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.051+0000 7f1bcadb8740 1 -- 172.21.15.40:0/377985982 shutdown_connections 2024-04-27T00:01:05.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.051+0000 7f1bcadb8740 1 -- 172.21.15.40:0/377985982 wait complete. 2024-04-27T00:01:05.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.052+0000 7f1bcadb8740 1 -- start start 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.052+0000 7f1bcadb8740 1 Processor -- start 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bcadb8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 0x55d194dfd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bcadb8740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194dfdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bcadb8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 0x55d194e04cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bcadb8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d194dfbdd0 con 0x55d194df8e20 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bcadb8740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55d194cdef70 con 0x55d194df3cb0 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bcadb8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d194dfbf50 con 0x55d194df32c0 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bcadb8740 10 client.0 _ll_register_callbacks cb 0x55d194d4d480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bc6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 0x55d194e04cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bc6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 0x55d194e04cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35340/0 (socket says 172.21.15.40:35340) 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bc6000640 1 -- 172.21.15.40:0/4158149871 learned_addr learned my addr 172.21.15.40:0/4158149871 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bc4c00640 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194dfdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bc6000640 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 0x55d194dfd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bc6000640 1 -- 172.21.15.40:0/4158149871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 msgr2=0x55d194dfd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bc6000640 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 0x55d194dfd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bc6000640 1 -- 172.21.15.40:0/4158149871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 msgr2=0x55d194dfdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bc6000640 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194dfdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.053+0000 7f1bc6000640 1 -- 172.21.15.40:0/4158149871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d194b336b0 con 0x55d194df8e20 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.054+0000 7f1bc6000640 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 0x55d194e04cc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80099e0 tx=0x7f1bb8005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ef59df5e2dbf9e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.054+0000 7f1ba6a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.054+0000 7f1ba6a00640 1 -- 172.21.15.40:0/4158149871 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb800fd40 con 0x55d194df8e20 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.054+0000 7f1ba6a00640 1 -- 172.21.15.40:0/4158149871 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f1bb8004a40 con 0x55d194df8e20 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.054+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4158149871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bb8003e20 con 0x55d194df8e20 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.054+0000 7f1ba6a00640 1 -- 172.21.15.40:0/4158149871 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1bb8032050 con 0x55d194df8e20 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.054+0000 7f1bcadb8740 10 client.15033 fetch_fsmap learned FSMap version 23 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.054+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4158149871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d194e07ca0 con 0x55d194df8e20 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.054+0000 7f1ba6a00640 1 -- 172.21.15.40:0/4158149871 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bb8003e20 con 0x55d194df8e20 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.055+0000 7f1ba6a00640 1 -- 172.21.15.40:0/4158149871 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f1bb8028040 con 0x55d194df8e20 2024-04-27T00:01:05.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.055+0000 7f1bcadb8740 10 client.15033 fetch_fsmap finished waiting for FSMap version 23 2024-04-27T00:01:05.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.055+0000 7f1bcadb8740 10 client.15033 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-27T00:01:05.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.055+0000 7f1bcadb8740 20 client.15033 populate_metadata read hostname 'smithi040' 2024-04-27T00:01:05.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.055+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55d194e0a060 0x55d194e0b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.055+0000 7f1bcadb8740 4 client.15033 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:01:05.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.055+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4158149871 --> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d194d65210 con 0x55d194e0a060 2024-04-27T00:01:05.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.055+0000 7f1bc4c00640 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55d194e0a060 0x55d194e0b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.056+0000 7f1bc4c00640 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55d194e0a060 0x55d194e0b4d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4aad3fdfbdc5284d server_cookie=2844040671b929fb in_seq=0 out_seq=0 2024-04-27T00:01:05.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.056+0000 7f1ba6a00640 10 client.15033 ms_handle_connect on v2:172.21.15.103:6838/1527392010 2024-04-27T00:01:05.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1ba6a00640 1 -- 172.21.15.40:0/4158149871 <== mds.2 v2:172.21.15.103:6838/1527392010 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d194d65210 con 0x55d194e0a060 2024-04-27T00:01:05.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1ba6a00640 10 client.15033 handle_command_reply: tid=0 2024-04-27T00:01:05.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1bcadb8740 1 client.15033 shutdown 2024-04-27T00:01:05.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4158149871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 msgr2=0x55d194e04cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 0x55d194e04cc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80099e0 tx=0x7f1bb8005bd0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4158149871 shutdown_connections 2024-04-27T00:01:05.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55d194e0a060 0x55d194e0b4d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 0x55d194e04cc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194dfdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1ba6a00640 0 client.15033 ms_handle_reset on v2:172.21.15.103:6838/1527392010 2024-04-27T00:01:05.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/4158149871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 0x55d194dfd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4158149871 >> 172.21.15.40:0/4158149871 conn(0x55d194d87d10 msgr2=0x55d194dedf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4158149871 shutdown_connections 2024-04-27T00:01:05.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.057+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4158149871 wait complete. 2024-04-27T00:01:05.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.058+0000 7f1bcadb8740 20 client.15033 trim_cache size 0 max 16384 2024-04-27T00:01:05.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.077+0000 7f1bcadb8740 1 Processor -- start 2024-04-27T00:01:05.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.077+0000 7f1bcadb8740 1 -- start start 2024-04-27T00:01:05.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.077+0000 7f1bcadb8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 0x55d194dfd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.077+0000 7f1bcadb8740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194e04be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.077+0000 7f1bcadb8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 0x55d194dfd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.077+0000 7f1bcadb8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d194e09810 con 0x55d194df8e20 2024-04-27T00:01:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.077+0000 7f1bcadb8740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55d194e09ee0 con 0x55d194df3cb0 2024-04-27T00:01:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.077+0000 7f1bcadb8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d194e0a5b0 con 0x55d194df32c0 2024-04-27T00:01:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.078+0000 7f1bc6000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 0x55d194dfd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.078+0000 7f1bc4c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194e04be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.078+0000 7f1bc6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 0x55d194dfd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.078+0000 7f1bc4c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194e04be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:46732/0 (socket says 172.21.15.40:46732) 2024-04-27T00:01:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.078+0000 7f1bc4c00640 1 -- 172.21.15.40:0/4224848251 learned_addr learned my addr 172.21.15.40:0/4224848251 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.078+0000 7f1bc4c00640 1 -- 172.21.15.40:0/4224848251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 msgr2=0x55d194dfd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.078+0000 7f1bc4c00640 1 --2- 172.21.15.40:0/4224848251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 0x55d194dfd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.078+0000 7f1bc4c00640 1 -- 172.21.15.40:0/4224848251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 msgr2=0x55d194dfd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.078+0000 7f1bc4c00640 1 --2- 172.21.15.40:0/4224848251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 0x55d194dfd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.078+0000 7f1bc4c00640 1 -- 172.21.15.40:0/4224848251 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d194df83b0 con 0x55d194df3cb0 2024-04-27T00:01:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.078+0000 7f1bc6000640 1 --2- 172.21.15.40:0/4224848251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 0x55d194dfd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.079+0000 7f1bc4c00640 1 --2- 172.21.15.40:0/4224848251 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194e04be0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1bc019c0e0 tx=0x7f1bac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1962d4e769845d1c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.079+0000 7f1bb1c00640 1 -- 172.21.15.40:0/4224848251 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac00fa10 con 0x55d194df3cb0 2024-04-27T00:01:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.079+0000 7f1bb1c00640 1 -- 172.21.15.40:0/4224848251 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bac022020 con 0x55d194df3cb0 2024-04-27T00:01:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.079+0000 7f1bb1c00640 1 -- 172.21.15.40:0/4224848251 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac01d410 con 0x55d194df3cb0 2024-04-27T00:01:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.080+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4224848251 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 msgr2=0x55d194e04be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.080+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/4224848251 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194e04be0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1bc019c0e0 tx=0x7f1bac0046c0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.080+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4224848251 shutdown_connections 2024-04-27T00:01:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.080+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/4224848251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df8e20 0x55d194dfd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.080+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/4224848251 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df3cb0 0x55d194e04be0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.080+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/4224848251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 0x55d194dfd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.080+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4224848251 >> 172.21.15.40:0/4224848251 conn(0x55d194d87d10 msgr2=0x55d194de9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.080+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4224848251 shutdown_connections 2024-04-27T00:01:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.080+0000 7f1bcadb8740 1 -- 172.21.15.40:0/4224848251 wait complete. 2024-04-27T00:01:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.080+0000 7f1bcadb8740 1 -- start start 2024-04-27T00:01:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.080+0000 7f1bcadb8740 1 Processor -- start 2024-04-27T00:01:05.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.081+0000 7f1bcadb8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 0x55d194e008d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.081+0000 7f1bcadb8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df3cb0 0x55d194e00e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.081+0000 7f1bcadb8740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df8e20 0x55d194e03e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.081+0000 7f1bcadb8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d194e0b350 con 0x55d194df3cb0 2024-04-27T00:01:05.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.081+0000 7f1bcadb8740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55d194e0ac80 con 0x55d194df8e20 2024-04-27T00:01:05.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.081+0000 7f1bcadb8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d194e0ba20 con 0x55d194df32c0 2024-04-27T00:01:05.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.081+0000 7f1bcadb8740 10 client.0 _ll_register_callbacks cb 0x7f1bb40f36f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:01:05.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.081+0000 7f1bc4c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df3cb0 0x55d194e00e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.081+0000 7f1bc4c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df3cb0 0x55d194e00e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35360/0 (socket says 172.21.15.40:35360) 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.081+0000 7f1bc4c00640 1 -- 172.21.15.40:0/3506283223 learned_addr learned my addr 172.21.15.40:0/3506283223 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.081+0000 7f1bc6000640 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 0x55d194e008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.081+0000 7f1bc6000640 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df8e20 0x55d194e03e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1bc4c00640 1 -- 172.21.15.40:0/3506283223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 msgr2=0x55d194e008d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1bc4c00640 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 0x55d194e008d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1bc4c00640 1 -- 172.21.15.40:0/3506283223 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df8e20 msgr2=0x55d194e03e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1bc4c00640 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df8e20 0x55d194e03e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1bc4c00640 1 -- 172.21.15.40:0/3506283223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d194e04420 con 0x55d194df3cb0 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1bc6000640 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df8e20 0x55d194e03e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1bc4c00640 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df3cb0 0x55d194e00e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1bac000c00 tx=0x7f1bac010810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4f3a054d98fd6d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1ba6a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1ba6a00640 1 -- 172.21.15.40:0/3506283223 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac02b070 con 0x55d194df3cb0 2024-04-27T00:01:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1bcadb8740 1 -- 172.21.15.40:0/3506283223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bac022020 con 0x55d194df3cb0 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1ba6a00640 1 -- 172.21.15.40:0/3506283223 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f1bac00fe20 con 0x55d194df3cb0 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1ba6a00640 1 -- 172.21.15.40:0/3506283223 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1bac036050 con 0x55d194df3cb0 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.082+0000 7f1bcadb8740 10 client.15039 fetch_fsmap learned FSMap version 23 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.083+0000 7f1bcadb8740 1 -- 172.21.15.40:0/3506283223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d194e70610 con 0x55d194df3cb0 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.083+0000 7f1ba6a00640 1 -- 172.21.15.40:0/3506283223 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bac022020 con 0x55d194df3cb0 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.083+0000 7f1ba6a00640 1 -- 172.21.15.40:0/3506283223 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f1bac004730 con 0x55d194df3cb0 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.083+0000 7f1bcadb8740 10 client.15039 fetch_fsmap finished waiting for FSMap version 23 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.083+0000 7f1bcadb8740 10 client.15039 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.083+0000 7f1bcadb8740 20 client.15039 populate_metadata read hostname 'smithi040' 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.083+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55d194e74f10 0x55d194e73c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.083+0000 7f1bcadb8740 4 client.15039 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.083+0000 7f1bcadb8740 1 -- 172.21.15.40:0/3506283223 --> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] -- command(tid 0: {"prefix": "status"}) -- 0x55d194d5f080 con 0x55d194e74f10 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.083+0000 7f1bc6000640 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55d194e74f10 0x55d194e73c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1bc6000640 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55d194e74f10 0x55d194e73c20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4f19f303a4dc613f server_cookie=e425bef7c3a59633 in_seq=0 out_seq=0 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1ba6a00640 10 client.15039 ms_handle_connect on v2:172.21.15.103:6838/1527392010 2024-04-27T00:01:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1ba6a00640 1 -- 172.21.15.40:0/3506283223 <== mds.2 v2:172.21.15.103:6838/1527392010 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55d194d5f080 con 0x55d194e74f10 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1ba6a00640 10 client.15039 handle_command_reply: tid=0 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1bcadb8740 1 client.15039 shutdown 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1bcadb8740 1 -- 172.21.15.40:0/3506283223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df3cb0 msgr2=0x55d194e00e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df3cb0 0x55d194e00e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1bac000c00 tx=0x7f1bac010810 comp rx=0 tx=0).stop 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1bcadb8740 1 -- 172.21.15.40:0/3506283223 shutdown_connections 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55d194e74f10 0x55d194e73c20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d194df8e20 0x55d194e03e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d194df3cb0 0x55d194e00e10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1bcadb8740 1 --2- 172.21.15.40:0/3506283223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d194df32c0 0x55d194e008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1ba6a00640 0 client.15039 ms_handle_reset on v2:172.21.15.103:6838/1527392010 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.084+0000 7f1bcadb8740 1 -- 172.21.15.40:0/3506283223 >> 172.21.15.40:0/3506283223 conn(0x55d194d87d10 msgr2=0x55d194dfd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.085+0000 7f1bcadb8740 1 -- 172.21.15.40:0/3506283223 shutdown_connections 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.085+0000 7f1bcadb8740 1 -- 172.21.15.40:0/3506283223 wait complete. 2024-04-27T00:01:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.085+0000 7f1bcadb8740 20 client.15039 trim_cache size 0 max 16384 2024-04-27T00:01:05.093 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-27T00:01:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "78cff6ea-0427-11ef-bc93-c7b262605968", 2024-04-27T00:01:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-04-27T00:01:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24559, 2024-04-27T00:01:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-27T00:01:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-27T00:01:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-27T00:01:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 100.731076671, 2024-04-27T00:01:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 23, 2024-04-27T00:01:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 92, 2024-04-27T00:01:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-27T00:01:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 151.725220208 2024-04-27T00:01:05.094 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-27T00:01:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.085+0000 7f1bc6a00640 1 -- 172.21.15.40:0/126997787 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1ba80a3c40 msgr2=0x7f1ba80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.085+0000 7f1bc6a00640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1ba80a3c40 0x7f1ba80a6100 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55d194df40b0 tx=0x7f1bb40f3820 comp rx=0 tx=0).stop 2024-04-27T00:01:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.085+0000 7f1bc6a00640 1 -- 172.21.15.40:0/126997787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bc0101680 msgr2=0x7f1bc019abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.085+0000 7f1bc6a00640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bc0101680 0x7f1bc019abd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1bb40057c0 tx=0x7f1bb4032da0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.085+0000 7f1bc6a00640 1 -- 172.21.15.40:0/126997787 shutdown_connections 2024-04-27T00:01:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.085+0000 7f1bc6a00640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bc0103230 0x7f1bc01a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.085+0000 7f1bc6a00640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1bc0102870 0x7f1bc019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.085+0000 7f1bc6a00640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1ba80a3c40 0x7f1ba80a6100 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.085+0000 7f1bc6a00640 1 --2- 172.21.15.40:0/126997787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bc0101680 0x7f1bc019abd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.086+0000 7f1bc6a00640 1 -- 172.21.15.40:0/126997787 >> 172.21.15.40:0/126997787 conn(0x7f1bc00fcc10 msgr2=0x7f1bc006af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.086+0000 7f1bc6a00640 1 -- 172.21.15.40:0/126997787 shutdown_connections 2024-04-27T00:01:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.086+0000 7f1bc6a00640 1 -- 172.21.15.40:0/126997787 wait complete. 2024-04-27T00:01:05.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-27T00:01:05.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.309+0000 7fee3ae00640 1 -- 172.21.15.40:0/416454202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee34105d40 msgr2=0x7fee34108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.309+0000 7fee3ae00640 1 --2- 172.21.15.40:0/416454202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee34105d40 0x7fee34108130 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009900 tx=0x7fee1c02a400 comp rx=0 tx=0).stop 2024-04-27T00:01:05.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.309+0000 7fee3ae00640 1 -- 172.21.15.40:0/416454202 shutdown_connections 2024-04-27T00:01:05.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.309+0000 7fee3ae00640 1 --2- 172.21.15.40:0/416454202 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee34108670 0x7fee3410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.309+0000 7fee3ae00640 1 --2- 172.21.15.40:0/416454202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee34105d40 0x7fee34108130 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.309+0000 7fee3ae00640 1 --2- 172.21.15.40:0/416454202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee34103410 0x7fee34105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.309+0000 7fee3ae00640 1 -- 172.21.15.40:0/416454202 >> 172.21.15.40:0/416454202 conn(0x7fee340fce20 msgr2=0x7fee340ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.310+0000 7fee3ae00640 1 -- 172.21.15.40:0/416454202 shutdown_connections 2024-04-27T00:01:05.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.310+0000 7fee3ae00640 1 -- 172.21.15.40:0/416454202 wait complete. 2024-04-27T00:01:05.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.311+0000 7fee3ae00640 1 Processor -- start 2024-04-27T00:01:05.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.311+0000 7fee3ae00640 1 -- start start 2024-04-27T00:01:05.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.311+0000 7fee3ae00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee34103410 0x7fee3419ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.311+0000 7fee3ae00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee34105d40 0x7fee3419b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.311+0000 7fee3ae00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee34108670 0x7fee341a2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.311+0000 7fee3ae00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee3410e4d0 con 0x7fee34103410 2024-04-27T00:01:05.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.311+0000 7fee3ae00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fee3410e350 con 0x7fee34108670 2024-04-27T00:01:05.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.311+0000 7fee3ae00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee3410e650 con 0x7fee34105d40 2024-04-27T00:01:05.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.311+0000 7fee33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee34103410 0x7fee3419ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.311+0000 7fee38e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee34108670 0x7fee341a2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.311+0000 7fee33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee34103410 0x7fee3419ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35376/0 (socket says 172.21.15.40:35376) 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.311+0000 7fee33e00640 1 -- 172.21.15.40:0/4087114374 learned_addr learned my addr 172.21.15.40:0/4087114374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.312+0000 7fee33400640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee34105d40 0x7fee3419b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.312+0000 7fee33e00640 1 -- 172.21.15.40:0/4087114374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee34105d40 msgr2=0x7fee3419b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.312+0000 7fee33e00640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee34105d40 0x7fee3419b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.312+0000 7fee33e00640 1 -- 172.21.15.40:0/4087114374 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee34108670 msgr2=0x7fee341a2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.312+0000 7fee33e00640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee34108670 0x7fee341a2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.312+0000 7fee33e00640 1 -- 172.21.15.40:0/4087114374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee341a2a70 con 0x7fee34103410 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.312+0000 7fee33e00640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee34103410 0x7fee3419ae50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fee2400eb90 tx=0x7fee2400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.312+0000 7fee30c00640 1 -- 172.21.15.40:0/4087114374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee24004d20 con 0x7fee34103410 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.312+0000 7fee30c00640 1 -- 172.21.15.40:0/4087114374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee2400ee90 con 0x7fee34103410 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.312+0000 7fee30c00640 1 -- 172.21.15.40:0/4087114374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee24010670 con 0x7fee34103410 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.313+0000 7fee3ae00640 1 -- 172.21.15.40:0/4087114374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee341a2d60 con 0x7fee34103410 2024-04-27T00:01:05.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.313+0000 7fee3ae00640 1 -- 172.21.15.40:0/4087114374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee341a31d0 con 0x7fee34103410 2024-04-27T00:01:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.316+0000 7fee3ddca740 1 Processor -- start 2024-04-27T00:01:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.316+0000 7fee3ddca740 1 -- start start 2024-04-27T00:01:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.316+0000 7fee30c00640 1 -- 172.21.15.40:0/4087114374 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fee24005020 con 0x7fee34103410 2024-04-27T00:01:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.316+0000 7fee3ddca740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 0x556bcce46510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.316+0000 7fee3ddca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce46ae0 0x556bcce4b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.316+0000 7fee3ddca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce4bc50 0x556bcce4e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.316+0000 7fee3ddca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556bccdaa600 con 0x556bcce4bc50 2024-04-27T00:01:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.316+0000 7fee3ddca740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x556bccd47100 con 0x556bcce460f0 2024-04-27T00:01:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.316+0000 7fee3ddca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556bccda65c0 con 0x556bcce46ae0 2024-04-27T00:01:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.317+0000 7fee38e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 0x556bcce46510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.317+0000 7fee38e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 0x556bcce46510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:46770/0 (socket says 172.21.15.40:46770) 2024-04-27T00:01:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.317+0000 7fee33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce4bc50 0x556bcce4e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.317+0000 7fee38e00640 1 -- 172.21.15.40:0/2515605248 learned_addr learned my addr 172.21.15.40:0/2515605248 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.317+0000 7fee33e00640 1 --2- 172.21.15.40:0/2515605248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce46ae0 0x556bcce4b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.317+0000 7fee38e00640 1 -- 172.21.15.40:0/2515605248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce46ae0 msgr2=0x556bcce4b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.317+0000 7fee38e00640 1 --2- 172.21.15.40:0/2515605248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce46ae0 0x556bcce4b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.317+0000 7fee38e00640 1 -- 172.21.15.40:0/2515605248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce4bc50 msgr2=0x556bcce4e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.317+0000 7fee38e00640 1 --2- 172.21.15.40:0/2515605248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce4bc50 0x556bcce4e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.317+0000 7fee38e00640 1 -- 172.21.15.40:0/2515605248 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bccabc2d0 con 0x556bcce460f0 2024-04-27T00:01:05.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.317+0000 7fee38e00640 1 --2- 172.21.15.40:0/2515605248 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 0x556bcce46510 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009ca0 tx=0x7fee2c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=776ff8a8d2a06d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.318+0000 7fee29600640 1 -- 172.21.15.40:0/2515605248 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c01e070 con 0x556bcce460f0 2024-04-27T00:01:05.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.318+0000 7fee30c00640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fee180a3c40 0x7fee180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.318+0000 7fee29600640 1 -- 172.21.15.40:0/2515605248 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee2c009e50 con 0x556bcce460f0 2024-04-27T00:01:05.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.318+0000 7fee29600640 1 -- 172.21.15.40:0/2515605248 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00ea50 con 0x556bcce460f0 2024-04-27T00:01:05.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.318+0000 7fee30c00640 1 -- 172.21.15.40:0/4087114374 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fee240d0f10 con 0x7fee34103410 2024-04-27T00:01:05.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.318+0000 7fee33e00640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fee180a3c40 0x7fee180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.318+0000 7fee3ddca740 1 -- 172.21.15.40:0/2515605248 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 msgr2=0x556bcce46510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.318+0000 7fee3ddca740 1 --2- 172.21.15.40:0/2515605248 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 0x556bcce46510 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009ca0 tx=0x7fee2c011810 comp rx=0 tx=0).stop 2024-04-27T00:01:05.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.319+0000 7fee3ddca740 1 -- 172.21.15.40:0/2515605248 shutdown_connections 2024-04-27T00:01:05.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.319+0000 7fee3ddca740 1 --2- 172.21.15.40:0/2515605248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce4bc50 0x556bcce4e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.319+0000 7fee3ddca740 1 --2- 172.21.15.40:0/2515605248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce46ae0 0x556bcce4b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.319+0000 7fee3ddca740 1 --2- 172.21.15.40:0/2515605248 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 0x556bcce46510 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.319+0000 7fee3ddca740 1 -- 172.21.15.40:0/2515605248 >> 172.21.15.40:0/2515605248 conn(0x556bccddad10 msgr2=0x556bcce41b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.319+0000 7fee3ddca740 1 -- 172.21.15.40:0/2515605248 shutdown_connections 2024-04-27T00:01:05.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.319+0000 7fee3ddca740 1 -- 172.21.15.40:0/2515605248 wait complete. 2024-04-27T00:01:05.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.319+0000 7fee33e00640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fee180a3c40 0x7fee180a6100 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x556bcce46ee0 tx=0x7fee2400cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.319+0000 7fee3ddca740 1 -- start start 2024-04-27T00:01:05.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.319+0000 7fee3ddca740 1 Processor -- start 2024-04-27T00:01:05.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee3ddca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce460f0 0x556bcce50570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee3ddca740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce46ae0 0x556bcce50ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee3ddca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce4bc50 0x556bcce57af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee3ddca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556bcce4ec00 con 0x556bcce460f0 2024-04-27T00:01:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee3ddca740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x556bccd31f70 con 0x556bcce46ae0 2024-04-27T00:01:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee3ddca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556bcce4ed80 con 0x556bcce4bc50 2024-04-27T00:01:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee3ddca740 10 client.0 _ll_register_callbacks cb 0x556bccda0480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:01:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee38e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce460f0 0x556bcce50570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee38e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce460f0 0x556bcce50570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35398/0 (socket says 172.21.15.40:35398) 2024-04-27T00:01:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee38e00640 1 -- 172.21.15.40:0/3572855519 learned_addr learned my addr 172.21.15.40:0/3572855519 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee33400640 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce46ae0 0x556bcce50ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee38e00640 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce4bc50 0x556bcce57af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee38e00640 1 -- 172.21.15.40:0/3572855519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce4bc50 msgr2=0x556bcce57af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee38e00640 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce4bc50 0x556bcce57af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee38e00640 1 -- 172.21.15.40:0/3572855519 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce46ae0 msgr2=0x556bcce50ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee38e00640 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce46ae0 0x556bcce50ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.320+0000 7fee38e00640 1 -- 172.21.15.40:0/3572855519 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bccb866b0 con 0x556bcce460f0 2024-04-27T00:01:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.321+0000 7fee38e00640 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce460f0 0x556bcce50570 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00e4f0 tx=0x7fee2c00e460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6575be12944e192a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.321+0000 7fee22000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:01:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.321+0000 7fee22000640 1 -- 172.21.15.40:0/3572855519 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c01e040 con 0x556bcce460f0 2024-04-27T00:01:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.321+0000 7fee22000640 1 -- 172.21.15.40:0/3572855519 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fee2c004020 con 0x556bcce460f0 2024-04-27T00:01:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.321+0000 7fee3ddca740 1 -- 172.21.15.40:0/3572855519 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee2c009e50 con 0x556bcce460f0 2024-04-27T00:01:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.321+0000 7fee22000640 1 -- 172.21.15.40:0/3572855519 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fee2c00ee50 con 0x556bcce460f0 2024-04-27T00:01:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.321+0000 7fee3ddca740 10 client.15051 fetch_fsmap learned FSMap version 23 2024-04-27T00:01:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.321+0000 7fee3ddca740 1 -- 172.21.15.40:0/3572855519 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556bcce5aad0 con 0x556bcce460f0 2024-04-27T00:01:05.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.322+0000 7fee22000640 1 -- 172.21.15.40:0/3572855519 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee2c009e50 con 0x556bcce460f0 2024-04-27T00:01:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.322+0000 7fee22000640 1 -- 172.21.15.40:0/3572855519 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fee2c024070 con 0x556bcce460f0 2024-04-27T00:01:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.322+0000 7fee3ddca740 10 client.15051 fetch_fsmap finished waiting for FSMap version 23 2024-04-27T00:01:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.322+0000 7fee3ddca740 10 client.15051 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-27T00:01:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.322+0000 7fee3ddca740 20 client.15051 populate_metadata read hostname 'smithi040' 2024-04-27T00:01:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.322+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x556bcce5cde0 0x556bcce5e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.322+0000 7fee3ddca740 4 client.15051 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:01:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.322+0000 7fee3ddca740 1 -- 172.21.15.40:0/3572855519 --> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bccdb8210 con 0x556bcce5cde0 2024-04-27T00:01:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.323+0000 7fee33400640 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x556bcce5cde0 0x556bcce5e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.323+0000 7fee33400640 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x556bcce5cde0 0x556bcce5e250 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=410b9ae0bd9cefb6 server_cookie=6b26b804495cf34f in_seq=0 out_seq=0 2024-04-27T00:01:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.323+0000 7fee22000640 10 client.15051 ms_handle_connect on v2:172.21.15.40:6838/2205335602 2024-04-27T00:01:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.324+0000 7fee22000640 1 -- 172.21.15.40:0/3572855519 <== mds.1 v2:172.21.15.40:6838/2205335602 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556bccdb8210 con 0x556bcce5cde0 2024-04-27T00:01:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.324+0000 7fee22000640 10 client.15051 handle_command_reply: tid=0 2024-04-27T00:01:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.324+0000 7fee3ddca740 1 client.15051 shutdown 2024-04-27T00:01:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.324+0000 7fee3ddca740 1 -- 172.21.15.40:0/3572855519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce460f0 msgr2=0x556bcce50570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.324+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce460f0 0x556bcce50570 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00e4f0 tx=0x7fee2c00e460 comp rx=0 tx=0).stop 2024-04-27T00:01:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.325+0000 7fee3ddca740 1 -- 172.21.15.40:0/3572855519 shutdown_connections 2024-04-27T00:01:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.325+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x556bcce5cde0 0x556bcce5e250 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.325+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce4bc50 0x556bcce57af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.325+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce46ae0 0x556bcce50ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.325+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3572855519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce460f0 0x556bcce50570 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.325+0000 7fee22000640 0 client.15051 ms_handle_reset on v2:172.21.15.40:6838/2205335602 2024-04-27T00:01:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.325+0000 7fee3ddca740 1 -- 172.21.15.40:0/3572855519 >> 172.21.15.40:0/3572855519 conn(0x556bccddad10 msgr2=0x556bcce40eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.325+0000 7fee3ddca740 1 -- 172.21.15.40:0/3572855519 shutdown_connections 2024-04-27T00:01:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.325+0000 7fee3ddca740 1 -- 172.21.15.40:0/3572855519 wait complete. 2024-04-27T00:01:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.325+0000 7fee3ddca740 20 client.15051 trim_cache size 0 max 16384 2024-04-27T00:01:05.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.344+0000 7fee3ddca740 1 Processor -- start 2024-04-27T00:01:05.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.344+0000 7fee3ddca740 1 -- start start 2024-04-27T00:01:05.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.344+0000 7fee3ddca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce460f0 0x556bcce47f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.344+0000 7fee3ddca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 0x556bcce57ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.344+0000 7fee3ddca740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce4bc50 0x556bcce48480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.344+0000 7fee3ddca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556bcce5c590 con 0x556bcce46ae0 2024-04-27T00:01:05.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.344+0000 7fee3ddca740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x556bcce5cc60 con 0x556bcce4bc50 2024-04-27T00:01:05.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.344+0000 7fee3ddca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556bcce5d330 con 0x556bcce460f0 2024-04-27T00:01:05.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.345+0000 7fee33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 0x556bcce57ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.345+0000 7fee38e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce460f0 0x556bcce47f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.345+0000 7fee33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 0x556bcce57ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35412/0 (socket says 172.21.15.40:35412) 2024-04-27T00:01:05.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.345+0000 7fee33400640 1 -- 172.21.15.40:0/3366988875 learned_addr learned my addr 172.21.15.40:0/3366988875 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.345+0000 7fee38e00640 1 --2- 172.21.15.40:0/3366988875 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce4bc50 0x556bcce48480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.345+0000 7fee33400640 1 -- 172.21.15.40:0/3366988875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce460f0 msgr2=0x556bcce47f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.345+0000 7fee33400640 1 --2- 172.21.15.40:0/3366988875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce460f0 0x556bcce47f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.345+0000 7fee33400640 1 -- 172.21.15.40:0/3366988875 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce4bc50 msgr2=0x556bcce48480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.345+0000 7fee33400640 1 --2- 172.21.15.40:0/3366988875 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce4bc50 0x556bcce48480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.345+0000 7fee33400640 1 -- 172.21.15.40:0/3366988875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bcce3d8a0 con 0x556bcce46ae0 2024-04-27T00:01:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.345+0000 7fee38e00640 1 --2- 172.21.15.40:0/3366988875 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce4bc50 0x556bcce48480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.346+0000 7fee33400640 1 --2- 172.21.15.40:0/3366988875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 0x556bcce57ab0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fee3406b7a0 tx=0x7fee1c0314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2882e78d3254c29c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.346+0000 7fee28c00640 1 -- 172.21.15.40:0/3366988875 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c04a030 con 0x556bcce46ae0 2024-04-27T00:01:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.346+0000 7fee28c00640 1 -- 172.21.15.40:0/3366988875 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee1c0318f0 con 0x556bcce46ae0 2024-04-27T00:01:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.346+0000 7fee28c00640 1 -- 172.21.15.40:0/3366988875 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c035040 con 0x556bcce46ae0 2024-04-27T00:01:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.346+0000 7fee3ddca740 1 -- 172.21.15.40:0/3366988875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 msgr2=0x556bcce57ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.346+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3366988875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 0x556bcce57ab0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fee3406b7a0 tx=0x7fee1c0314a0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.347+0000 7fee3ddca740 1 -- 172.21.15.40:0/3366988875 shutdown_connections 2024-04-27T00:01:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.347+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3366988875 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce4bc50 0x556bcce48480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.347+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3366988875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 0x556bcce57ab0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.347+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3366988875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce460f0 0x556bcce47f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.347+0000 7fee3ddca740 1 -- 172.21.15.40:0/3366988875 >> 172.21.15.40:0/3366988875 conn(0x556bccddad10 msgr2=0x556bcce5be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.347+0000 7fee3ddca740 1 -- 172.21.15.40:0/3366988875 shutdown_connections 2024-04-27T00:01:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.347+0000 7fee3ddca740 1 -- 172.21.15.40:0/3366988875 wait complete. 2024-04-27T00:01:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.347+0000 7fee3ddca740 1 -- start start 2024-04-27T00:01:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.347+0000 7fee3ddca740 1 Processor -- start 2024-04-27T00:01:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee3ddca740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 0x556bcce53790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee3ddca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 0x556bcce53cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee3ddca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce4bc50 0x556bcce56cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee3ddca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556bcce5e0d0 con 0x556bcce46ae0 2024-04-27T00:01:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee3ddca740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x556bcce5da00 con 0x556bcce460f0 2024-04-27T00:01:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee3ddca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556bcce5e7a0 con 0x556bcce4bc50 2024-04-27T00:01:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee3ddca740 10 client.0 _ll_register_callbacks cb 0x7fee2c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:01:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 0x556bcce53cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee38e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 0x556bcce53790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee33400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 0x556bcce53cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35416/0 (socket says 172.21.15.40:35416) 2024-04-27T00:01:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee33400640 1 -- 172.21.15.40:0/3440025919 learned_addr learned my addr 172.21.15.40:0/3440025919 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee38e00640 1 --2- 172.21.15.40:0/3440025919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce4bc50 0x556bcce56cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee38e00640 1 -- 172.21.15.40:0/3440025919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce4bc50 msgr2=0x556bcce56cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee38e00640 1 --2- 172.21.15.40:0/3440025919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce4bc50 0x556bcce56cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee38e00640 1 -- 172.21.15.40:0/3440025919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 msgr2=0x556bcce53cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee38e00640 1 --2- 172.21.15.40:0/3440025919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 0x556bcce53cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.348+0000 7fee38e00640 1 -- 172.21.15.40:0/3440025919 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bcce572e0 con 0x556bcce460f0 2024-04-27T00:01:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.349+0000 7fee38e00640 1 --2- 172.21.15.40:0/3440025919 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 0x556bcce53790 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fee2c000c00 tx=0x7fee2c00e460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59cce5f34ee77599 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.349+0000 7fee22000640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:01:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.349+0000 7fee22000640 1 -- 172.21.15.40:0/3440025919 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c01e030 con 0x556bcce460f0 2024-04-27T00:01:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.349+0000 7fee3ddca740 1 -- 172.21.15.40:0/3440025919 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee1c0318f0 con 0x556bcce460f0 2024-04-27T00:01:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.349+0000 7fee22000640 1 -- 172.21.15.40:0/3440025919 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fee2c003890 con 0x556bcce460f0 2024-04-27T00:01:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.349+0000 7fee22000640 1 -- 172.21.15.40:0/3440025919 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fee2c009e50 con 0x556bcce460f0 2024-04-27T00:01:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.349+0000 7fee3ddca740 10 client.25184 fetch_fsmap learned FSMap version 23 2024-04-27T00:01:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.349+0000 7fee3ddca740 1 -- 172.21.15.40:0/3440025919 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556bccec3380 con 0x556bcce460f0 2024-04-27T00:01:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.349+0000 7fee22000640 1 -- 172.21.15.40:0/3440025919 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee1c0318f0 con 0x556bcce460f0 2024-04-27T00:01:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.350+0000 7fee22000640 1 -- 172.21.15.40:0/3440025919 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fee2c024040 con 0x556bcce460f0 2024-04-27T00:01:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.350+0000 7fee3ddca740 10 client.25184 fetch_fsmap finished waiting for FSMap version 23 2024-04-27T00:01:05.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.350+0000 7fee3ddca740 10 client.25184 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-27T00:01:05.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.350+0000 7fee3ddca740 20 client.25184 populate_metadata read hostname 'smithi040' 2024-04-27T00:01:05.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.350+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3440025919 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x556bccec7eb0 0x556bccec42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.350+0000 7fee3ddca740 4 client.25184 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-27T00:01:05.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.350+0000 7fee3ddca740 1 -- 172.21.15.40:0/3440025919 --> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] -- command(tid 0: {"prefix": "status"}) -- 0x556bccdb2080 con 0x556bccec7eb0 2024-04-27T00:01:05.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.350+0000 7fee33400640 1 --2- 172.21.15.40:0/3440025919 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x556bccec7eb0 0x556bccec42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee33400640 1 --2- 172.21.15.40:0/3440025919 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x556bccec7eb0 0x556bccec42f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1da6a0f756898e1c server_cookie=e512a78dc3259482 in_seq=0 out_seq=0 2024-04-27T00:01:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee22000640 10 client.25184 ms_handle_connect on v2:172.21.15.40:6838/2205335602 2024-04-27T00:01:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee22000640 1 -- 172.21.15.40:0/3440025919 <== mds.1 v2:172.21.15.40:6838/2205335602 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x556bccdb2080 con 0x556bccec7eb0 2024-04-27T00:01:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee22000640 10 client.25184 handle_command_reply: tid=0 2024-04-27T00:01:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee3ddca740 1 client.25184 shutdown 2024-04-27T00:01:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee3ddca740 1 -- 172.21.15.40:0/3440025919 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 msgr2=0x556bcce53790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3440025919 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 0x556bcce53790 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fee2c000c00 tx=0x7fee2c00e460 comp rx=0 tx=0).stop 2024-04-27T00:01:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee3ddca740 1 -- 172.21.15.40:0/3440025919 shutdown_connections 2024-04-27T00:01:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3440025919 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x556bccec7eb0 0x556bccec42f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3440025919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bcce4bc50 0x556bcce56cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee22000640 0 client.25184 ms_handle_reset on v2:172.21.15.40:6838/2205335602 2024-04-27T00:01:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3440025919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bcce46ae0 0x556bcce53cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee3ddca740 1 --2- 172.21.15.40:0/3440025919 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556bcce460f0 0x556bcce53790 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee3ddca740 1 -- 172.21.15.40:0/3440025919 >> 172.21.15.40:0/3440025919 conn(0x556bccddad10 msgr2=0x556bcce50180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.351+0000 7fee3ddca740 1 -- 172.21.15.40:0/3440025919 shutdown_connections 2024-04-27T00:01:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ddca740 1 -- 172.21.15.40:0/3440025919 wait complete. 2024-04-27T00:01:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ddca740 20 client.25184 trim_cache size 0 max 16384 2024-04-27T00:01:05.359 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-27T00:01:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "78cff6ea-0427-11ef-bc93-c7b262605968", 2024-04-27T00:01:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-04-27T00:01:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24560, 2024-04-27T00:01:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-27T00:01:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-27T00:01:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-27T00:01:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 102.01216395100001, 2024-04-27T00:01:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 23, 2024-04-27T00:01:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 92, 2024-04-27T00:01:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-27T00:01:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 147.96000731399999 2024-04-27T00:01:05.360 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-27T00:01:05.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ae00640 1 -- 172.21.15.40:0/4087114374 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fee180a3c40 msgr2=0x7fee180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ae00640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fee180a3c40 0x7fee180a6100 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x556bcce46ee0 tx=0x7fee2400cf90 comp rx=0 tx=0).stop 2024-04-27T00:01:05.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ae00640 1 -- 172.21.15.40:0/4087114374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee34103410 msgr2=0x7fee3419ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ae00640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee34103410 0x7fee3419ae50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fee2400eb90 tx=0x7fee2400c6a0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ae00640 1 -- 172.21.15.40:0/4087114374 shutdown_connections 2024-04-27T00:01:05.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ae00640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee34108670 0x7fee341a2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ae00640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fee180a3c40 0x7fee180a6100 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ae00640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee34105d40 0x7fee3419b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ae00640 1 --2- 172.21.15.40:0/4087114374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee34103410 0x7fee3419ae50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ae00640 1 -- 172.21.15.40:0/4087114374 >> 172.21.15.40:0/4087114374 conn(0x7fee340fce20 msgr2=0x7fee34102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.352+0000 7fee3ae00640 1 -- 172.21.15.40:0/4087114374 shutdown_connections 2024-04-27T00:01:05.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.353+0000 7fee3ae00640 1 -- 172.21.15.40:0/4087114374 wait complete. 2024-04-27T00:01:05.370 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-27T00:01:05.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.559+0000 7f1f53e00640 1 -- 172.21.15.40:0/1536297462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f54106540 msgr2=0x7f1f5410cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.559+0000 7f1f53e00640 1 --2- 172.21.15.40:0/1536297462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f54106540 0x7f1f5410cdd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c009be0 tx=0x7f1f4c02a6d0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.560+0000 7f1f53e00640 1 -- 172.21.15.40:0/1536297462 shutdown_connections 2024-04-27T00:01:05.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.560+0000 7f1f53e00640 1 --2- 172.21.15.40:0/1536297462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f54106540 0x7f1f5410cdd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.560+0000 7f1f53e00640 1 --2- 172.21.15.40:0/1536297462 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1f54105b80 0x7f1f54106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.560+0000 7f1f53e00640 1 --2- 172.21.15.40:0/1536297462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f54104990 0x7f1f54104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.560+0000 7f1f53e00640 1 -- 172.21.15.40:0/1536297462 >> 172.21.15.40:0/1536297462 conn(0x7f1f540fff00 msgr2=0x7f1f54102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.560+0000 7f1f53e00640 1 -- 172.21.15.40:0/1536297462 shutdown_connections 2024-04-27T00:01:05.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.560+0000 7f1f53e00640 1 -- 172.21.15.40:0/1536297462 wait complete. 2024-04-27T00:01:05.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f53e00640 1 Processor -- start 2024-04-27T00:01:05.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f53e00640 1 -- start start 2024-04-27T00:01:05.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f53e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1f54104990 0x7f1f5419ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f53e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f54105b80 0x7f1f5419b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f53e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f54106540 0x7f1f541a2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f53e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1f541104a0 con 0x7f1f54105b80 2024-04-27T00:01:05.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f53e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1f54110320 con 0x7f1f54104990 2024-04-27T00:01:05.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f53e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1f54110620 con 0x7f1f54106540 2024-04-27T00:01:05.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f52a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1f54104990 0x7f1f5419ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f52a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1f54104990 0x7f1f5419ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:46824/0 (socket says 172.21.15.40:46824) 2024-04-27T00:01:05.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f52a00640 1 -- 172.21.15.40:0/2637732440 learned_addr learned my addr 172.21.15.40:0/2637732440 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f52000640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f54105b80 0x7f1f5419b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.561+0000 7f1f53400640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f54106540 0x7f1f541a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.562+0000 7f1f52a00640 1 -- 172.21.15.40:0/2637732440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f54106540 msgr2=0x7f1f541a2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.562+0000 7f1f52a00640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f54106540 0x7f1f541a2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.562+0000 7f1f52a00640 1 -- 172.21.15.40:0/2637732440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f54105b80 msgr2=0x7f1f5419b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.562+0000 7f1f52a00640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f54105b80 0x7f1f5419b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.562+0000 7f1f52a00640 1 -- 172.21.15.40:0/2637732440 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f541a2770 con 0x7f1f54104990 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.562+0000 7f1f52a00640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1f54104990 0x7f1f5419ab50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400e9a0 tx=0x7f1f4400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.562+0000 7f1f4b400640 1 -- 172.21.15.40:0/2637732440 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4400cd80 con 0x7f1f54104990 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.563+0000 7f1f53e00640 1 -- 172.21.15.40:0/2637732440 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f541a2a60 con 0x7f1f54104990 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.563+0000 7f1f4b400640 1 -- 172.21.15.40:0/2637732440 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f440108e0 con 0x7f1f54104990 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.563+0000 7f1f4b400640 1 -- 172.21.15.40:0/2637732440 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44018960 con 0x7f1f54104990 2024-04-27T00:01:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.563+0000 7f1f53e00640 1 -- 172.21.15.40:0/2637732440 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f541a3000 con 0x7f1f54104990 2024-04-27T00:01:05.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.566+0000 7f1f4b400640 1 -- 172.21.15.40:0/2637732440 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1f44010a90 con 0x7f1f54104990 2024-04-27T00:01:05.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.566+0000 7f1f5c452740 1 Processor -- start 2024-04-27T00:01:05.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.566+0000 7f1f5c452740 1 -- start start 2024-04-27T00:01:05.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f5c452740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d685742c0 0x555d685746e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f5c452740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68574cb0 0x555d685797b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f5c452740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d68579e20 0x555d6857c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f5c452740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d684d8600 con 0x555d68574cb0 2024-04-27T00:01:05.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f5c452740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x555d68475100 con 0x555d685742c0 2024-04-27T00:01:05.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f5c452740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555d684d45c0 con 0x555d68579e20 2024-04-27T00:01:05.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f52a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68574cb0 0x555d685797b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f52a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68574cb0 0x555d685797b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35438/0 (socket says 172.21.15.40:35438) 2024-04-27T00:01:05.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f52a00640 1 -- 172.21.15.40:0/3731953523 learned_addr learned my addr 172.21.15.40:0/3731953523 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f53400640 1 --2- 172.21.15.40:0/3731953523 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d685742c0 0x555d685746e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f52000640 1 --2- 172.21.15.40:0/3731953523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d68579e20 0x555d6857c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f52a00640 1 -- 172.21.15.40:0/3731953523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d68579e20 msgr2=0x555d6857c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f52a00640 1 --2- 172.21.15.40:0/3731953523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d68579e20 0x555d6857c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f52a00640 1 -- 172.21.15.40:0/3731953523 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d685742c0 msgr2=0x555d685746e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f52a00640 1 --2- 172.21.15.40:0/3731953523 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d685742c0 0x555d685746e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.567+0000 7f1f52a00640 1 -- 172.21.15.40:0/3731953523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d681ea2d0 con 0x555d68574cb0 2024-04-27T00:01:05.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.568+0000 7f1f4b400640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1f380a3c40 0x7f1f380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.568+0000 7f1f52a00640 1 --2- 172.21.15.40:0/3731953523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68574cb0 0x555d685797b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1f440d4dd0 tx=0x7f1f440d9720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c021a5f2c30670 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.568+0000 7f1f53400640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1f380a3c40 0x7f1f380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.568+0000 7f1f4b400640 1 -- 172.21.15.40:0/2637732440 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f1f440d0c60 con 0x7f1f54104990 2024-04-27T00:01:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.568+0000 7f1f3f400640 1 -- 172.21.15.40:0/3731953523 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f44098ce0 con 0x555d68574cb0 2024-04-27T00:01:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.568+0000 7f1f3f400640 1 -- 172.21.15.40:0/3731953523 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f440d9e30 con 0x555d68574cb0 2024-04-27T00:01:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.568+0000 7f1f3f400640 1 -- 172.21.15.40:0/3731953523 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f440bd0c0 con 0x555d68574cb0 2024-04-27T00:01:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.569+0000 7f1f5c452740 1 -- 172.21.15.40:0/3731953523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68574cb0 msgr2=0x555d685797b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.569+0000 7f1f5c452740 1 --2- 172.21.15.40:0/3731953523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68574cb0 0x555d685797b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1f440d4dd0 tx=0x7f1f440d9720 comp rx=0 tx=0).stop 2024-04-27T00:01:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.569+0000 7f1f53400640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1f380a3c40 0x7f1f380a6100 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c004af0 tx=0x7f1f4c030990 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.569+0000 7f1f5c452740 1 -- 172.21.15.40:0/3731953523 shutdown_connections 2024-04-27T00:01:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.569+0000 7f1f5c452740 1 --2- 172.21.15.40:0/3731953523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d68579e20 0x555d6857c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.569+0000 7f1f5c452740 1 --2- 172.21.15.40:0/3731953523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68574cb0 0x555d685797b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.569+0000 7f1f5c452740 1 --2- 172.21.15.40:0/3731953523 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d685742c0 0x555d685746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.569+0000 7f1f5c452740 1 -- 172.21.15.40:0/3731953523 >> 172.21.15.40:0/3731953523 conn(0x555d68508d10 msgr2=0x555d6856fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.569+0000 7f1f5c452740 1 -- 172.21.15.40:0/3731953523 shutdown_connections 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.569+0000 7f1f5c452740 1 -- 172.21.15.40:0/3731953523 wait complete. 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.570+0000 7f1f5c452740 1 -- start start 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.570+0000 7f1f5c452740 1 Processor -- start 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.570+0000 7f1f5c452740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d6857e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f5c452740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d6857ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f5c452740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d68585c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f5c452740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d6857cdd0 con 0x555d68579e20 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f5c452740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x555d6845ff70 con 0x555d68574cb0 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f5c452740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555d6857cf50 con 0x555d685742c0 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f5c452740 10 client.0 _ll_register_callbacks cb 0x555d684ce480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f53400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d68585c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f52a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d6857e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f52a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d6857e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50658/0 (socket says 172.21.15.40:50658) 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f52a00640 1 -- 172.21.15.40:0/206245372 learned_addr learned my addr 172.21.15.40:0/206245372 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f52000640 1 --2- 172.21.15.40:0/206245372 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d6857ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f52a00640 1 -- 172.21.15.40:0/206245372 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 msgr2=0x555d6857ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f52a00640 1 --2- 172.21.15.40:0/206245372 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d6857ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f52a00640 1 -- 172.21.15.40:0/206245372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 msgr2=0x555d68585c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f52a00640 1 --2- 172.21.15.40:0/206245372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d68585c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.571+0000 7f1f52a00640 1 -- 172.21.15.40:0/206245372 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d682b46b0 con 0x555d685742c0 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.572+0000 7f1f52a00640 1 --2- 172.21.15.40:0/206245372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d6857e6f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1f440d4dd0 tx=0x7f1f44018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7699c355f7693ac2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.572+0000 7f1f33e00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.572+0000 7f1f33e00640 1 -- 172.21.15.40:0/206245372 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4409c3a0 con 0x555d685742c0 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.572+0000 7f1f5c452740 1 -- 172.21.15.40:0/206245372 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f440d9e30 con 0x555d685742c0 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.572+0000 7f1f33e00640 1 -- 172.21.15.40:0/206245372 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f1f440ce4c0 con 0x555d685742c0 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.572+0000 7f1f33e00640 1 -- 172.21.15.40:0/206245372 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1f4409c580 con 0x555d685742c0 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.572+0000 7f1f5c452740 10 client.25207 fetch_fsmap learned FSMap version 23 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.572+0000 7f1f5c452740 1 -- 172.21.15.40:0/206245372 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555d6857ce50 con 0x555d685742c0 2024-04-27T00:01:05.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.572+0000 7f1f33e00640 1 -- 172.21.15.40:0/206245372 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f440d9e30 con 0x555d685742c0 2024-04-27T00:01:05.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.573+0000 7f1f33e00640 1 -- 172.21.15.40:0/206245372 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f1f4409cca0 con 0x555d685742c0 2024-04-27T00:01:05.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.573+0000 7f1f5c452740 10 client.25207 fetch_fsmap finished waiting for FSMap version 23 2024-04-27T00:01:05.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.573+0000 7f1f5c452740 10 client.25207 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-27T00:01:05.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.573+0000 7f1f5c452740 20 client.25207 populate_metadata read hostname 'smithi040' 2024-04-27T00:01:05.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.573+0000 7f1f5c452740 1 --2- 172.21.15.40:0/206245372 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x555d6858ae00 0x555d6858c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.573+0000 7f1f5c452740 4 client.25207 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.573+0000 7f1f5c452740 1 -- 172.21.15.40:0/206245372 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d684e6210 con 0x555d6858ae00 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.573+0000 7f1f52000640 1 --2- 172.21.15.40:0/206245372 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x555d6858ae00 0x555d6858c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.574+0000 7f1f52000640 1 --2- 172.21.15.40:0/206245372 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x555d6858ae00 0x555d6858c270 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62a49656bc8365ee server_cookie=cb7dc2a42cf790a0 in_seq=0 out_seq=0 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.574+0000 7f1f33e00640 10 client.25207 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f33e00640 1 -- 172.21.15.40:0/206245372 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d684e6210 con 0x555d6858ae00 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f33e00640 10 client.25207 handle_command_reply: tid=0 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f5c452740 1 client.25207 shutdown 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f5c452740 1 -- 172.21.15.40:0/206245372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 msgr2=0x555d6857e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f5c452740 1 --2- 172.21.15.40:0/206245372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d6857e6f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1f440d4dd0 tx=0x7f1f44018420 comp rx=0 tx=0).stop 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f5c452740 1 -- 172.21.15.40:0/206245372 shutdown_connections 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f5c452740 1 --2- 172.21.15.40:0/206245372 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x555d6858ae00 0x555d6858c270 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f5c452740 1 --2- 172.21.15.40:0/206245372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d68585c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f33e00640 0 client.25207 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:01:05.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f5c452740 1 --2- 172.21.15.40:0/206245372 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d6857ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f5c452740 1 --2- 172.21.15.40:0/206245372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d6857e6f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f5c452740 1 -- 172.21.15.40:0/206245372 >> 172.21.15.40:0/206245372 conn(0x555d68508d10 msgr2=0x555d6856ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f5c452740 1 -- 172.21.15.40:0/206245372 shutdown_connections 2024-04-27T00:01:05.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f5c452740 1 -- 172.21.15.40:0/206245372 wait complete. 2024-04-27T00:01:05.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.575+0000 7f1f5c452740 20 client.25207 trim_cache size 0 max 16384 2024-04-27T00:01:05.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.596+0000 7f1f5c452740 1 Processor -- start 2024-04-27T00:01:05.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.596+0000 7f1f5c452740 1 -- start start 2024-04-27T00:01:05.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.596+0000 7f1f5c452740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d6857ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.596+0000 7f1f5c452740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d68585c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.596+0000 7f1f5c452740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d6856e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.596+0000 7f1f5c452740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d6858a5b0 con 0x555d68579e20 2024-04-27T00:01:05.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.596+0000 7f1f5c452740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x555d6858ac80 con 0x555d68574cb0 2024-04-27T00:01:05.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.596+0000 7f1f5c452740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555d6858b350 con 0x555d685742c0 2024-04-27T00:01:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.596+0000 7f1f53400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d6856e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.596+0000 7f1f53400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d6856e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35454/0 (socket says 172.21.15.40:35454) 2024-04-27T00:01:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.596+0000 7f1f53400640 1 -- 172.21.15.40:0/2540086559 learned_addr learned my addr 172.21.15.40:0/2540086559 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.597+0000 7f1f52000640 1 --2- 172.21.15.40:0/2540086559 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d68585c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.597+0000 7f1f52a00640 1 --2- 172.21.15.40:0/2540086559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d6857ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.597+0000 7f1f53400640 1 -- 172.21.15.40:0/2540086559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 msgr2=0x555d6857ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.597+0000 7f1f53400640 1 --2- 172.21.15.40:0/2540086559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d6857ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.597+0000 7f1f53400640 1 -- 172.21.15.40:0/2540086559 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 msgr2=0x555d68585c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.597+0000 7f1f53400640 1 --2- 172.21.15.40:0/2540086559 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d68585c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.597+0000 7f1f53400640 1 -- 172.21.15.40:0/2540086559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d6856b460 con 0x555d68579e20 2024-04-27T00:01:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.597+0000 7f1f53400640 1 --2- 172.21.15.40:0/2540086559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d6856e6d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c030d90 tx=0x7f1f4c040070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e51df1728d4eaa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.598+0000 7f1f3ea00640 1 -- 172.21.15.40:0/2540086559 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c036040 con 0x555d68579e20 2024-04-27T00:01:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.598+0000 7f1f3ea00640 1 -- 172.21.15.40:0/2540086559 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f4c050070 con 0x555d68579e20 2024-04-27T00:01:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.598+0000 7f1f3ea00640 1 -- 172.21.15.40:0/2540086559 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c055c20 con 0x555d68579e20 2024-04-27T00:01:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.598+0000 7f1f5c452740 1 -- 172.21.15.40:0/2540086559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 msgr2=0x555d6856e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.598+0000 7f1f5c452740 1 --2- 172.21.15.40:0/2540086559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d6856e6d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c030d90 tx=0x7f1f4c040070 comp rx=0 tx=0).stop 2024-04-27T00:01:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.598+0000 7f1f5c452740 1 -- 172.21.15.40:0/2540086559 shutdown_connections 2024-04-27T00:01:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.598+0000 7f1f5c452740 1 --2- 172.21.15.40:0/2540086559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d6856e6d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.598+0000 7f1f5c452740 1 --2- 172.21.15.40:0/2540086559 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d68585c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.598+0000 7f1f5c452740 1 --2- 172.21.15.40:0/2540086559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d6857ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.598+0000 7f1f5c452740 1 -- 172.21.15.40:0/2540086559 >> 172.21.15.40:0/2540086559 conn(0x555d68508d10 msgr2=0x555d6857e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.598+0000 7f1f5c452740 1 -- 172.21.15.40:0/2540086559 shutdown_connections 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.598+0000 7f1f5c452740 1 -- 172.21.15.40:0/2540086559 wait complete. 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.599+0000 7f1f5c452740 1 -- start start 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.599+0000 7f1f5c452740 1 Processor -- start 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.599+0000 7f1f5c452740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d68585640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f5c452740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d68585b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f5c452740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d685820e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f5c452740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d6858c0f0 con 0x555d68579e20 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f5c452740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x555d6858ba20 con 0x555d68574cb0 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f5c452740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555d6858c7c0 con 0x555d685742c0 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f5c452740 10 client.0 _ll_register_callbacks cb 0x7f1f40005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:01:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f53400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d685820e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f52a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d68585640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f53400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d685820e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35460/0 (socket says 172.21.15.40:35460) 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f53400640 1 -- 172.21.15.40:0/3358825675 learned_addr learned my addr 172.21.15.40:0/3358825675 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f52000640 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d68585b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f52a00640 1 -- 172.21.15.40:0/3358825675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 msgr2=0x555d68585b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f52a00640 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d68585b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f52a00640 1 -- 172.21.15.40:0/3358825675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 msgr2=0x555d685820e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f52a00640 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d685820e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f52a00640 1 -- 172.21.15.40:0/3358825675 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d68582910 con 0x555d685742c0 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f52000640 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d68585b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.600+0000 7f1f53400640 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d685820e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.601+0000 7f1f52a00640 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d68585640 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1f440a5180 tx=0x7f1f440d9720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdad82971dfe160c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.601+0000 7f1f33e00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.601+0000 7f1f33e00640 1 -- 172.21.15.40:0/3358825675 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4409c810 con 0x555d685742c0 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.601+0000 7f1f33e00640 1 -- 172.21.15.40:0/3358825675 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f1f4409cca0 con 0x555d685742c0 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.601+0000 7f1f5c452740 1 -- 172.21.15.40:0/3358825675 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f4c050070 con 0x555d685742c0 2024-04-27T00:01:05.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.601+0000 7f1f33e00640 1 -- 172.21.15.40:0/3358825675 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1f4409c3a0 con 0x555d685742c0 2024-04-27T00:01:05.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.601+0000 7f1f5c452740 10 client.25213 fetch_fsmap learned FSMap version 23 2024-04-27T00:01:05.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.601+0000 7f1f5c452740 1 -- 172.21.15.40:0/3358825675 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555d685f1aa0 con 0x555d685742c0 2024-04-27T00:01:05.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.601+0000 7f1f33e00640 1 -- 172.21.15.40:0/3358825675 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f4c050070 con 0x555d685742c0 2024-04-27T00:01:05.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.602+0000 7f1f33e00640 1 -- 172.21.15.40:0/3358825675 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f1f440d4070 con 0x555d685742c0 2024-04-27T00:01:05.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.602+0000 7f1f5c452740 10 client.25213 fetch_fsmap finished waiting for FSMap version 23 2024-04-27T00:01:05.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.602+0000 7f1f5c452740 10 client.25213 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-27T00:01:05.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.602+0000 7f1f5c452740 20 client.25213 populate_metadata read hostname 'smithi040' 2024-04-27T00:01:05.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.602+0000 7f1f5c452740 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x555d685f6410 0x555d685f2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:05.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.602+0000 7f1f5c452740 4 client.25213 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-27T00:01:05.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.602+0000 7f1f5c452740 1 -- 172.21.15.40:0/3358825675 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "status"}) -- 0x555d684e0080 con 0x555d685f6410 2024-04-27T00:01:05.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.602+0000 7f1f52000640 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x555d685f6410 0x555d685f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f52000640 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x555d685f6410 0x555d685f2850 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=682aae2d737954bf server_cookie=4ec6821ad9f6c435 in_seq=0 out_seq=0 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f33e00640 10 client.25213 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f33e00640 1 -- 172.21.15.40:0/3358825675 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x555d684e0080 con 0x555d685f6410 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f33e00640 10 client.25213 handle_command_reply: tid=0 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f5c452740 1 client.25213 shutdown 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f5c452740 1 -- 172.21.15.40:0/3358825675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 msgr2=0x555d68585640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f5c452740 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d68585640 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1f440a5180 tx=0x7f1f440d9720 comp rx=0 tx=0).stop 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f5c452740 1 -- 172.21.15.40:0/3358825675 shutdown_connections 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f5c452740 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x555d685f6410 0x555d685f2850 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f5c452740 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d68579e20 0x555d685820e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f5c452740 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x555d68574cb0 0x555d68585b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f5c452740 1 --2- 172.21.15.40:0/3358825675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d685742c0 0x555d68585640 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f5c452740 1 -- 172.21.15.40:0/3358825675 >> 172.21.15.40:0/3358825675 conn(0x555d68508d10 msgr2=0x555d68572d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f33e00640 0 client.25213 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:01:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f5c452740 1 -- 172.21.15.40:0/3358825675 shutdown_connections 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.603+0000 7f1f5c452740 1 -- 172.21.15.40:0/3358825675 wait complete. 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.604+0000 7f1f5c452740 20 client.25213 trim_cache size 0 max 16384 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "78cff6ea-0427-11ef-bc93-c7b262605968", 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24590, 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 134.48951536000001, 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 23, 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 92, 2024-04-27T00:01:05.609 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-27T00:01:05.610 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 140.89750538600001 2024-04-27T00:01:05.610 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-27T00:01:05.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.604+0000 7f1f53e00640 1 -- 172.21.15.40:0/2637732440 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1f380a3c40 msgr2=0x7f1f380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.604+0000 7f1f53e00640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1f380a3c40 0x7f1f380a6100 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c004af0 tx=0x7f1f4c030990 comp rx=0 tx=0).stop 2024-04-27T00:01:05.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.604+0000 7f1f53e00640 1 -- 172.21.15.40:0/2637732440 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1f54104990 msgr2=0x7f1f5419ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:05.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.604+0000 7f1f53e00640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1f54104990 0x7f1f5419ab50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400e9a0 tx=0x7f1f4400ee60 comp rx=0 tx=0).stop 2024-04-27T00:01:05.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.604+0000 7f1f53e00640 1 -- 172.21.15.40:0/2637732440 shutdown_connections 2024-04-27T00:01:05.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.604+0000 7f1f53e00640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1f380a3c40 0x7f1f380a6100 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.604+0000 7f1f53e00640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1f54106540 0x7f1f541a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.604+0000 7f1f53e00640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1f54105b80 0x7f1f5419b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.604+0000 7f1f53e00640 1 --2- 172.21.15.40:0/2637732440 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1f54104990 0x7f1f5419ab50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:05.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.604+0000 7f1f53e00640 1 -- 172.21.15.40:0/2637732440 >> 172.21.15.40:0/2637732440 conn(0x7f1f540fff00 msgr2=0x7f1f54108e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:05.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.604+0000 7f1f53e00640 1 -- 172.21.15.40:0/2637732440 shutdown_connections 2024-04-27T00:01:05.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:05.605+0000 7f1f53e00640 1 -- 172.21.15.40:0/2637732440 wait complete. 2024-04-27T00:01:05.622 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:05 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1355274955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:05.625 INFO:tasks.cephadm:Setup complete, yielding 2024-04-27T00:01:05.626 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T00:01:05.635 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-04-27T00:01:05.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-27T00:01:05.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:05 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1355274955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:05.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:05 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1355274955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:06.094 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:01:06.582 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:06 smithi040 ceph-mon[28095]: pgmap v392: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:06.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:06 smithi080 ceph-mon[32362]: pgmap v392: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:06.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:06 smithi103 ceph-mon[32952]: pgmap v392: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:08.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.360+0000 7f262c600700 1 -- 172.21.15.40:0/1038466894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2624109170 msgr2=0x7f262410b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:08.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.360+0000 7f262c600700 1 --2- 172.21.15.40:0/1038466894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2624109170 0x7f262410b5e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f261c00b0a0 tx=0x7f261c00b3b0 comp rx=0 tx=0).stop 2024-04-27T00:01:08.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.360+0000 7f262c600700 1 -- 172.21.15.40:0/1038466894 shutdown_connections 2024-04-27T00:01:08.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.360+0000 7f262c600700 1 --2- 172.21.15.40:0/1038466894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2624109170 0x7f262410b5e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:08.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.360+0000 7f262c600700 1 --2- 172.21.15.40:0/1038466894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2624103f10 0x7f2624106300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:08.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.360+0000 7f262c600700 1 --2- 172.21.15.40:0/1038466894 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2624106840 0x7f2624108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:08.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.360+0000 7f262c600700 1 -- 172.21.15.40:0/1038466894 >> 172.21.15.40:0/1038466894 conn(0x7f26240fd920 msgr2=0x7f26240ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:08.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.361+0000 7f262c600700 1 -- 172.21.15.40:0/1038466894 shutdown_connections 2024-04-27T00:01:08.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.361+0000 7f262c600700 1 -- 172.21.15.40:0/1038466894 wait complete. 2024-04-27T00:01:08.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.362+0000 7f262c600700 1 Processor -- start 2024-04-27T00:01:08.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.362+0000 7f262c600700 1 -- start start 2024-04-27T00:01:08.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.363+0000 7f262c600700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2624103f10 0x7f262419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:08.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.363+0000 7f262c600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2624106840 0x7f262419bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:08.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.363+0000 7f262c600700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2624109170 0x7f26241a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:08.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.363+0000 7f262c600700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f262410ef30 con 0x7f2624103f10 2024-04-27T00:01:08.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.363+0000 7f262c600700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f262410edb0 con 0x7f2624109170 2024-04-27T00:01:08.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.363+0000 7f262c600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f262410f0b0 con 0x7f2624106840 2024-04-27T00:01:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.363+0000 7f2629c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2624103f10 0x7f262419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.363+0000 7f2629c00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2624103f10 0x7f262419b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35470/0 (socket says 172.21.15.40:35470) 2024-04-27T00:01:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.363+0000 7f2629c00700 1 -- 172.21.15.40:0/1638378534 learned_addr learned my addr 172.21.15.40:0/1638378534 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.364+0000 7f2629200700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2624106840 0x7f262419bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.364+0000 7f262a600700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2624109170 0x7f26241a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.364+0000 7f2629c00700 1 -- 172.21.15.40:0/1638378534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2624106840 msgr2=0x7f262419bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.364+0000 7f2629c00700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2624106840 0x7f262419bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.364+0000 7f2629c00700 1 -- 172.21.15.40:0/1638378534 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2624109170 msgr2=0x7f26241a2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.364+0000 7f2629c00700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2624109170 0x7f26241a2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.364+0000 7f2629c00700 1 -- 172.21.15.40:0/1638378534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26241a34b0 con 0x7f2624103f10 2024-04-27T00:01:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.364+0000 7f2629200700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2624106840 0x7f262419bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:08.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.364+0000 7f2629c00700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2624103f10 0x7f262419b850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f261800d980 tx=0x7f261800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:08.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.365+0000 7f2622000700 1 -- 172.21.15.40:0/1638378534 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2618016070 con 0x7f2624103f10 2024-04-27T00:01:08.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.365+0000 7f262c600700 1 -- 172.21.15.40:0/1638378534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26241a37a0 con 0x7f2624103f10 2024-04-27T00:01:08.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.365+0000 7f2622000700 1 -- 172.21.15.40:0/1638378534 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2618009d80 con 0x7f2624103f10 2024-04-27T00:01:08.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.365+0000 7f262c600700 1 -- 172.21.15.40:0/1638378534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26241a3d30 con 0x7f2624103f10 2024-04-27T00:01:08.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.365+0000 7f2622000700 1 -- 172.21.15.40:0/1638378534 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2618020060 con 0x7f2624103f10 2024-04-27T00:01:08.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.366+0000 7f262c600700 1 -- 172.21.15.40:0/1638378534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f262406a110 con 0x7f2624103f10 2024-04-27T00:01:08.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.368+0000 7f2622000700 1 -- 172.21.15.40:0/1638378534 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2618019070 con 0x7f2624103f10 2024-04-27T00:01:08.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.369+0000 7f2622000700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f260c0a3cd0 0x7f260c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:08.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.370+0000 7f2622000700 1 -- 172.21.15.40:0/1638378534 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f26180d0750 con 0x7f2624103f10 2024-04-27T00:01:08.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.370+0000 7f2629200700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f260c0a3cd0 0x7f260c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:08.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.371+0000 7f2629200700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f260c0a3cd0 0x7f260c0a6190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2610005950 tx=0x7f261000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:08.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.375+0000 7f2622000700 1 -- 172.21.15.40:0/1638378534 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f261809cc30 con 0x7f2624103f10 2024-04-27T00:01:08.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.634+0000 7f262c600700 1 -- 172.21.15.40:0/1638378534 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f2624063860 con 0x7f260c0a3cd0 2024-04-27T00:01:08.636 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:08 smithi040 ceph-mon[28095]: pgmap v393: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:08.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.636+0000 7f2622000700 1 -- 172.21.15.40:0/1638378534 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f2624063860 con 0x7f260c0a3cd0 2024-04-27T00:01:08.638 INFO:teuthology.orchestra.run.smithi040.stdout:Backend: cephadm 2024-04-27T00:01:08.638 INFO:teuthology.orchestra.run.smithi040.stdout:Available: Yes 2024-04-27T00:01:08.638 INFO:teuthology.orchestra.run.smithi040.stdout:Paused: No 2024-04-27T00:01:08.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.641+0000 7f262c600700 1 -- 172.21.15.40:0/1638378534 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f260c0a3cd0 msgr2=0x7f260c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:08.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.641+0000 7f262c600700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f260c0a3cd0 0x7f260c0a6190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2610005950 tx=0x7f261000b410 comp rx=0 tx=0).stop 2024-04-27T00:01:08.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.641+0000 7f262c600700 1 -- 172.21.15.40:0/1638378534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2624103f10 msgr2=0x7f262419b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:08.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.641+0000 7f262c600700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2624103f10 0x7f262419b850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f261800d980 tx=0x7f261800dc90 comp rx=0 tx=0).stop 2024-04-27T00:01:08.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.641+0000 7f262c600700 1 -- 172.21.15.40:0/1638378534 shutdown_connections 2024-04-27T00:01:08.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.641+0000 7f262c600700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f260c0a3cd0 0x7f260c0a6190 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:08.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.641+0000 7f262c600700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2624109170 0x7f26241a2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:08.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.641+0000 7f262c600700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2624103f10 0x7f262419b850 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:08.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.641+0000 7f262c600700 1 --2- 172.21.15.40:0/1638378534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2624106840 0x7f262419bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:08.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.641+0000 7f262c600700 1 -- 172.21.15.40:0/1638378534 >> 172.21.15.40:0/1638378534 conn(0x7f26240fd920 msgr2=0x7f2624103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:08.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.641+0000 7f262c600700 1 -- 172.21.15.40:0/1638378534 shutdown_connections 2024-04-27T00:01:08.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:08.641+0000 7f262c600700 1 -- 172.21.15.40:0/1638378534 wait complete. 2024-04-27T00:01:08.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:08 smithi080 ceph-mon[32362]: pgmap v393: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:08.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:08 smithi103 ceph-mon[32952]: pgmap v393: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:09.504 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-27T00:01:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:01:10.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:10 smithi040 ceph-mon[28095]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:10.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:10 smithi040 ceph-mon[28095]: pgmap v394: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:10.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:10 smithi080 ceph-mon[32362]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:10.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:10 smithi080 ceph-mon[32362]: pgmap v394: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:10.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:10 smithi103 ceph-mon[32952]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:10.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:10 smithi103 ceph-mon[32952]: pgmap v394: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.170+0000 7f1631000700 1 -- 172.21.15.40:0/1942576761 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f162c1056c0 msgr2=0x7f162c105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.170+0000 7f1631000700 1 --2- 172.21.15.40:0/1942576761 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f162c1056c0 0x7f162c105ae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f161c009920 tx=0x7f161c009c30 comp rx=0 tx=0).stop 2024-04-27T00:01:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.171+0000 7f1631000700 1 -- 172.21.15.40:0/1942576761 shutdown_connections 2024-04-27T00:01:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.171+0000 7f1631000700 1 --2- 172.21.15.40:0/1942576761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162c107260 0x7f162c10db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.171+0000 7f1631000700 1 --2- 172.21.15.40:0/1942576761 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f162c1056c0 0x7f162c105ae0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.171+0000 7f1631000700 1 --2- 172.21.15.40:0/1942576761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f162c1068c0 0x7f162c106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:12.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.171+0000 7f1631000700 1 -- 172.21.15.40:0/1942576761 >> 172.21.15.40:0/1942576761 conn(0x7f162c100c80 msgr2=0x7f162c1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:12.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.171+0000 7f1631000700 1 -- 172.21.15.40:0/1942576761 shutdown_connections 2024-04-27T00:01:12.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.171+0000 7f1631000700 1 -- 172.21.15.40:0/1942576761 wait complete. 2024-04-27T00:01:12.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.172+0000 7f1631000700 1 Processor -- start 2024-04-27T00:01:12.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.172+0000 7f1631000700 1 -- start start 2024-04-27T00:01:12.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.173+0000 7f1631000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f162c1056c0 0x7f162c07b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:12.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.173+0000 7f1631000700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f162c1068c0 0x7f162c07b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:12.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.173+0000 7f1631000700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162c107260 0x7f162c077d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.173+0000 7f1631000700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f162c111180 con 0x7f162c107260 2024-04-27T00:01:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.173+0000 7f1631000700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f162c111000 con 0x7f162c1068c0 2024-04-27T00:01:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.173+0000 7f1631000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f162c111300 con 0x7f162c1056c0 2024-04-27T00:01:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.173+0000 7f162a800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162c107260 0x7f162c077d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.173+0000 7f1629e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f162c1056c0 0x7f162c07b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.173+0000 7f1629e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f162c1056c0 0x7f162c07b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:42082/0 (socket says 172.21.15.40:42082) 2024-04-27T00:01:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.173+0000 7f1629e00700 1 -- 172.21.15.40:0/3628558124 learned_addr learned my addr 172.21.15.40:0/3628558124 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.173+0000 7f1629400700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f162c1068c0 0x7f162c07b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.174+0000 7f1629e00700 1 -- 172.21.15.40:0/3628558124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f162c1068c0 msgr2=0x7f162c07b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.174+0000 7f1629e00700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f162c1068c0 0x7f162c07b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.174+0000 7f1629e00700 1 -- 172.21.15.40:0/3628558124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162c107260 msgr2=0x7f162c077d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:12.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.174+0000 7f1629e00700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162c107260 0x7f162c077d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:12.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.174+0000 7f1629e00700 1 -- 172.21.15.40:0/3628558124 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f162c078620 con 0x7f162c1056c0 2024-04-27T00:01:12.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.174+0000 7f1629400700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f162c1068c0 0x7f162c07b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:12.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.174+0000 7f1629e00700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f162c1056c0 0x7f162c07b1d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f161c009230 tx=0x7f161c005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:12.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.174+0000 7f1622a00700 1 -- 172.21.15.40:0/3628558124 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c027070 con 0x7f162c1056c0 2024-04-27T00:01:12.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.175+0000 7f1631000700 1 -- 172.21.15.40:0/3628558124 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f162c1a7cf0 con 0x7f162c1056c0 2024-04-27T00:01:12.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.175+0000 7f1622a00700 1 -- 172.21.15.40:0/3628558124 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f161c00ba10 con 0x7f162c1056c0 2024-04-27T00:01:12.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.175+0000 7f1622a00700 1 -- 172.21.15.40:0/3628558124 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c004930 con 0x7f162c1056c0 2024-04-27T00:01:12.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.175+0000 7f1631000700 1 -- 172.21.15.40:0/3628558124 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f162c1a8090 con 0x7f162c1056c0 2024-04-27T00:01:12.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.176+0000 7f1631000700 1 -- 172.21.15.40:0/3628558124 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f162c06a110 con 0x7f162c1056c0 2024-04-27T00:01:12.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.178+0000 7f1622a00700 1 -- 172.21.15.40:0/3628558124 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f161c02a070 con 0x7f162c1056c0 2024-04-27T00:01:12.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.179+0000 7f1622a00700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f16100a3d50 0x7f16100a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:12.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.179+0000 7f1622a00700 1 -- 172.21.15.40:0/3628558124 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f161c0d13b0 con 0x7f162c1056c0 2024-04-27T00:01:12.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.180+0000 7f1629400700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f16100a3d50 0x7f16100a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:12.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.180+0000 7f1629400700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f16100a3d50 0x7f16100a6210 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1618011490 tx=0x7f1618009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:12.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.184+0000 7f1622a00700 1 -- 172.21.15.40:0/3628558124 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f161c05f060 con 0x7f162c1056c0 2024-04-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.423+0000 7f1631000700 1 -- 172.21.15.40:0/3628558124 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f162c063860 con 0x7f16100a3d50 2024-04-27T00:01:12.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.472+0000 7f1622a00700 1 -- 172.21.15.40:0/3628558124 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f162c063860 con 0x7f16100a3d50 2024-04-27T00:01:12.475 INFO:teuthology.orchestra.run.smithi040.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T00:01:12.475 INFO:teuthology.orchestra.run.smithi040.stdout:mds.a smithi040 running (2m) 40s ago 2m 12.3M - 19.0.0-3306-gbc612ffb 55dc54019ad8 fab18cca5df1 2024-04-27T00:01:12.475 INFO:teuthology.orchestra.run.smithi040.stdout:mds.b smithi080 running (2m) 79s ago 2m 28.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 cb060557aa61 2024-04-27T00:01:12.475 INFO:teuthology.orchestra.run.smithi040.stdout:mds.c smithi103 running (2m) 79s ago 2m 16.7M - 19.0.0-3306-gbc612ffb 55dc54019ad8 6356b7125703 2024-04-27T00:01:12.475 INFO:teuthology.orchestra.run.smithi040.stdout:mds.d smithi040 running (2m) 40s ago 2m 12.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 885a4a093439 2024-04-27T00:01:12.475 INFO:teuthology.orchestra.run.smithi040.stdout:mds.e smithi080 running (2m) 79s ago 2m 15.6M - 19.0.0-3306-gbc612ffb 55dc54019ad8 908d3b1b27f6 2024-04-27T00:01:12.475 INFO:teuthology.orchestra.run.smithi040.stdout:mds.f smithi103 running (2m) 79s ago 2m 14.9M - 19.0.0-3306-gbc612ffb 55dc54019ad8 642aacff1d54 2024-04-27T00:01:12.475 INFO:teuthology.orchestra.run.smithi040.stdout:mds.g smithi040 running (2m) 40s ago 2m 16.3M - 19.0.0-3306-gbc612ffb 55dc54019ad8 337ff61668b7 2024-04-27T00:01:12.475 INFO:teuthology.orchestra.run.smithi040.stdout:mds.h smithi080 running (2m) 79s ago 2m 17.3M - 19.0.0-3306-gbc612ffb 55dc54019ad8 19f051a4dcd9 2024-04-27T00:01:12.475 INFO:teuthology.orchestra.run.smithi040.stdout:mds.i smithi103 running (2m) 79s ago 2m 19.6M - 19.0.0-3306-gbc612ffb 55dc54019ad8 ee1cb2f0787c 2024-04-27T00:01:12.475 INFO:teuthology.orchestra.run.smithi040.stdout:mds.j smithi040 running (2m) 40s ago 2m 12.7M - 19.0.0-3306-gbc612ffb 55dc54019ad8 9261d7109e4a 2024-04-27T00:01:12.475 INFO:teuthology.orchestra.run.smithi040.stdout:mds.k smithi080 running (3m) 79s ago 3m 14.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 bf6655a67f9f 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:mds.l smithi103 running (3m) 79s ago 3m 16.6M - 19.0.0-3306-gbc612ffb 55dc54019ad8 646bac59a435 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:mgr.x smithi040 *:9283,8765 running (11m) 40s ago 11m 515M - 19.0.0-3306-gbc612ffb 55dc54019ad8 24a7832aa9b8 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:mgr.y smithi080 *:8443,8765 running (9m) 79s ago 9m 455M - 19.0.0-3306-gbc612ffb 55dc54019ad8 c49e7b5b507a 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:mgr.z smithi103 *:8443,8765 running (9m) 79s ago 9m 455M - 19.0.0-3306-gbc612ffb 55dc54019ad8 3a8618c5cbde 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:mon.a smithi040 running (11m) 40s ago 11m 49.5M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 9cf2eb2d66cd 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:mon.b smithi080 running (9m) 79s ago 9m 41.1M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 766b7e4712e2 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:mon.c smithi103 running (9m) 79s ago 9m 42.1M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 22caa4266198 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 smithi040 running (8m) 40s ago 8m 52.4M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 717b9ecf56d9 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 smithi080 running (8m) 79s ago 8m 46.3M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 204147e7ecab 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 smithi103 running (7m) 79s ago 7m 55.4M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 6740ea3c4164 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3 smithi040 running (7m) 40s ago 7m 51.5M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 1072722a8a0e 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4 smithi080 running (6m) 79s ago 6m 50.5M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 db0fa9c13963 2024-04-27T00:01:12.476 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5 smithi103 running (6m) 79s ago 6m 44.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 3006c0859c1d 2024-04-27T00:01:12.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6 smithi040 running (5m) 40s ago 5m 51.3M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 22d627004579 2024-04-27T00:01:12.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 smithi080 running (5m) 79s ago 5m 44.0M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 3e7a6ef652d2 2024-04-27T00:01:12.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8 smithi103 running (4m) 79s ago 4m 51.1M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 4811ec89670a 2024-04-27T00:01:12.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9 smithi040 running (4m) 40s ago 4m 49.3M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 b9ac475e79cf 2024-04-27T00:01:12.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 smithi080 running (3m) 79s ago 3m 42.7M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 d3742a708cc2 2024-04-27T00:01:12.477 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11 smithi103 running (3m) 79s ago 3m 42.2M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 e8b53b452967 2024-04-27T00:01:12.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.476+0000 7f1631000700 1 -- 172.21.15.40:0/3628558124 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f16100a3d50 msgr2=0x7f16100a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:12.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.476+0000 7f1631000700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f16100a3d50 0x7f16100a6210 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1618011490 tx=0x7f1618009260 comp rx=0 tx=0).stop 2024-04-27T00:01:12.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.476+0000 7f1631000700 1 -- 172.21.15.40:0/3628558124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f162c1056c0 msgr2=0x7f162c07b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:12.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.476+0000 7f1631000700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f162c1056c0 0x7f162c07b1d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f161c009230 tx=0x7f161c005270 comp rx=0 tx=0).stop 2024-04-27T00:01:12.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.476+0000 7f1631000700 1 -- 172.21.15.40:0/3628558124 shutdown_connections 2024-04-27T00:01:12.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.476+0000 7f1631000700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f16100a3d50 0x7f16100a6210 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:12.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.476+0000 7f1631000700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f162c107260 0x7f162c077d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:12.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.476+0000 7f1631000700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f162c1056c0 0x7f162c07b1d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:12.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.476+0000 7f1631000700 1 --2- 172.21.15.40:0/3628558124 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f162c1068c0 0x7f162c07b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:12.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.476+0000 7f1631000700 1 -- 172.21.15.40:0/3628558124 >> 172.21.15.40:0/3628558124 conn(0x7f162c100c80 msgr2=0x7f162c109ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:12.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.476+0000 7f1631000700 1 -- 172.21.15.40:0/3628558124 shutdown_connections 2024-04-27T00:01:12.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:12.477+0000 7f1631000700 1 -- 172.21.15.40:0/3628558124 wait complete. 2024-04-27T00:01:12.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:12 smithi040 ceph-mon[28095]: pgmap v395: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:01:12.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:12 smithi080 ceph-mon[32362]: pgmap v395: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:01:12.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:12 smithi103 ceph-mon[32952]: pgmap v395: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:01:13.355 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-27T00:01:13.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:13 smithi040 ceph-mon[28095]: from='client.25225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:13.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:13 smithi080 ceph-mon[32362]: from='client.25225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:13.840 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:01:13.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:13 smithi103 ceph-mon[32952]: from='client.25225 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:14.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:14 smithi040 ceph-mon[28095]: pgmap v396: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:14 smithi080 ceph-mon[32362]: pgmap v396: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:14 smithi103 ceph-mon[32952]: pgmap v396: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.003+0000 7fcc7e800700 1 -- 172.21.15.40:0/3056663100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc78109ac0 msgr2=0x7fcc78109ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.003+0000 7fcc7e800700 1 --2- 172.21.15.40:0/3056663100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc78109ac0 0x7fcc78109ee0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009990 tx=0x7fcc68009ca0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.003+0000 7fcc7e800700 1 -- 172.21.15.40:0/3056663100 shutdown_connections 2024-04-27T00:01:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.003+0000 7fcc7e800700 1 --2- 172.21.15.40:0/3056663100 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcc7810b660 0x7fcc78111f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.003+0000 7fcc7e800700 1 --2- 172.21.15.40:0/3056663100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc78109ac0 0x7fcc78109ee0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.003+0000 7fcc7e800700 1 --2- 172.21.15.40:0/3056663100 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7810acc0 0x7fcc7810b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.003+0000 7fcc7e800700 1 -- 172.21.15.40:0/3056663100 >> 172.21.15.40:0/3056663100 conn(0x7fcc78078a90 msgr2=0x7fcc7807aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.003+0000 7fcc7e800700 1 -- 172.21.15.40:0/3056663100 shutdown_connections 2024-04-27T00:01:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.003+0000 7fcc7e800700 1 -- 172.21.15.40:0/3056663100 wait complete. 2024-04-27T00:01:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.005+0000 7fcc7e800700 1 Processor -- start 2024-04-27T00:01:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.005+0000 7fcc7e800700 1 -- start start 2024-04-27T00:01:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.005+0000 7fcc7e800700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcc78109ac0 0x7fcc7819fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.005+0000 7fcc7e800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7810acc0 0x7fcc781a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.005+0000 7fcc7e800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7810b660 0x7fcc781a71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.005+0000 7fcc7e800700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcc78115580 con 0x7fcc7810acc0 2024-04-27T00:01:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.005+0000 7fcc7e800700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fcc78115400 con 0x7fcc78109ac0 2024-04-27T00:01:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.005+0000 7fcc7e800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcc78115700 con 0x7fcc7810b660 2024-04-27T00:01:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.006+0000 7fcc6ea00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7810acc0 0x7fcc781a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.006+0000 7fcc77400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcc78109ac0 0x7fcc7819fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.006+0000 7fcc6ea00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7810acc0 0x7fcc781a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37368/0 (socket says 172.21.15.40:37368) 2024-04-27T00:01:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.006+0000 7fcc6ea00700 1 -- 172.21.15.40:0/2636963492 learned_addr learned my addr 172.21.15.40:0/2636963492 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.006+0000 7fcc77e00700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7810b660 0x7fcc781a71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:16.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.006+0000 7fcc6ea00700 1 -- 172.21.15.40:0/2636963492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7810b660 msgr2=0x7fcc781a71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:16.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.006+0000 7fcc6ea00700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7810b660 0x7fcc781a71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.006+0000 7fcc6ea00700 1 -- 172.21.15.40:0/2636963492 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcc78109ac0 msgr2=0x7fcc7819fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.006+0000 7fcc6ea00700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcc78109ac0 0x7fcc7819fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.006+0000 7fcc6ea00700 1 -- 172.21.15.40:0/2636963492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc781a7890 con 0x7fcc7810acc0 2024-04-27T00:01:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.006+0000 7fcc77400700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcc78109ac0 0x7fcc7819fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.006+0000 7fcc77e00700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7810b660 0x7fcc781a71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.007+0000 7fcc6ea00700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7810acc0 0x7fcc781a0180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcc6000ef10 tx=0x7fcc6000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.007+0000 7fcc74c00700 1 -- 172.21.15.40:0/2636963492 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6000c9c0 con 0x7fcc7810acc0 2024-04-27T00:01:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.007+0000 7fcc7e800700 1 -- 172.21.15.40:0/2636963492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc781a7b20 con 0x7fcc7810acc0 2024-04-27T00:01:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.007+0000 7fcc74c00700 1 -- 172.21.15.40:0/2636963492 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc60019070 con 0x7fcc7810acc0 2024-04-27T00:01:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.007+0000 7fcc74c00700 1 -- 172.21.15.40:0/2636963492 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc60014480 con 0x7fcc7810acc0 2024-04-27T00:01:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.007+0000 7fcc7e800700 1 -- 172.21.15.40:0/2636963492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc781a80b0 con 0x7fcc7810acc0 2024-04-27T00:01:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.009+0000 7fcc7e800700 1 -- 172.21.15.40:0/2636963492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc7806a110 con 0x7fcc7810acc0 2024-04-27T00:01:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.011+0000 7fcc74c00700 1 -- 172.21.15.40:0/2636963492 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcc60012050 con 0x7fcc7810acc0 2024-04-27T00:01:16.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.012+0000 7fcc74c00700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcc5c0a3d50 0x7fcc5c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.012+0000 7fcc74c00700 1 -- 172.21.15.40:0/2636963492 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fcc600d0ea0 con 0x7fcc7810acc0 2024-04-27T00:01:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.013+0000 7fcc77400700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcc5c0a3d50 0x7fcc5c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.013+0000 7fcc77400700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcc5c0a3d50 0x7fcc5c0a6210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcc781a1120 tx=0x7fcc6801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:16.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.015+0000 7fcc74c00700 1 -- 172.21.15.40:0/2636963492 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcc60010040 con 0x7fcc7810acc0 2024-04-27T00:01:16.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.255+0000 7fcc7e800700 1 -- 172.21.15.40:0/2636963492 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fcc78063860 con 0x7fcc5c0a3d50 2024-04-27T00:01:16.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.272+0000 7fcc74c00700 1 -- 172.21.15.40:0/2636963492 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fcc78063860 con 0x7fcc5c0a3d50 2024-04-27T00:01:16.273 INFO:teuthology.orchestra.run.smithi040.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-27T00:01:16.273 INFO:teuthology.orchestra.run.smithi040.stdout:mds.all 12/12 83s ago 3m smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi080=b;smithi080=e;smithi080=h;smithi080=k;smithi103=c;smithi103=f;smithi103=i;smithi103=l;count:12 2024-04-27T00:01:16.273 INFO:teuthology.orchestra.run.smithi040.stdout:mgr 3/3 83s ago 9m smithi040=x;smithi080=y;smithi103=z;count:3 2024-04-27T00:01:16.273 INFO:teuthology.orchestra.run.smithi040.stdout:mon 3/3 83s ago 9m smithi040:172.21.15.40=a;smithi080:172.21.15.80=b;smithi103:172.21.15.103=c;count:3 2024-04-27T00:01:16.273 INFO:teuthology.orchestra.run.smithi040.stdout:osd 12 83s ago - 2024-04-27T00:01:16.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.275+0000 7fcc7e800700 1 -- 172.21.15.40:0/2636963492 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcc5c0a3d50 msgr2=0x7fcc5c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:16.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.275+0000 7fcc7e800700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcc5c0a3d50 0x7fcc5c0a6210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcc781a1120 tx=0x7fcc6801a040 comp rx=0 tx=0).stop 2024-04-27T00:01:16.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.275+0000 7fcc7e800700 1 -- 172.21.15.40:0/2636963492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7810acc0 msgr2=0x7fcc781a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:16.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.275+0000 7fcc7e800700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7810acc0 0x7fcc781a0180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcc6000ef10 tx=0x7fcc6000c3a0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.275+0000 7fcc7e800700 1 -- 172.21.15.40:0/2636963492 shutdown_connections 2024-04-27T00:01:16.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.275+0000 7fcc7e800700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fcc5c0a3d50 0x7fcc5c0a6210 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.275+0000 7fcc7e800700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7810b660 0x7fcc781a71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.275+0000 7fcc7e800700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fcc78109ac0 0x7fcc7819fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.275+0000 7fcc7e800700 1 --2- 172.21.15.40:0/2636963492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7810acc0 0x7fcc781a0180 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.275+0000 7fcc7e800700 1 -- 172.21.15.40:0/2636963492 >> 172.21.15.40:0/2636963492 conn(0x7fcc78078a90 msgr2=0x7fcc7810e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:16.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.276+0000 7fcc7e800700 1 -- 172.21.15.40:0/2636963492 shutdown_connections 2024-04-27T00:01:16.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:16.276+0000 7fcc7e800700 1 -- 172.21.15.40:0/2636963492 wait complete. 2024-04-27T00:01:16.695 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:16 smithi040 ceph-mon[28095]: pgmap v397: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:16.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:16 smithi080 ceph-mon[32362]: pgmap v397: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:16.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:16 smithi103 ceph-mon[32952]: pgmap v397: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:17.100 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-27T00:01:17.573 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:01:17.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:17 smithi080 ceph-mon[32362]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:17.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:17 smithi103 ceph-mon[32952]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:17.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:17 smithi040 ceph-mon[28095]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:18.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:18 smithi040 ceph-mon[28095]: pgmap v398: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:18.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:18 smithi080 ceph-mon[32362]: pgmap v398: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:18.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:18 smithi103 ceph-mon[32952]: pgmap v398: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.862+0000 7f0e8ac00700 1 -- 172.21.15.40:0/1864026343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e84102370 msgr2=0x7f0e84102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.862+0000 7f0e8ac00700 1 --2- 172.21.15.40:0/1864026343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e84102370 0x7f0e84102790 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0e740099c0 tx=0x7f0e74009cd0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.862+0000 7f0e8ac00700 1 -- 172.21.15.40:0/1864026343 shutdown_connections 2024-04-27T00:01:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.862+0000 7f0e8ac00700 1 --2- 172.21.15.40:0/1864026343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e84103f10 0x7f0e8410a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.862+0000 7f0e8ac00700 1 --2- 172.21.15.40:0/1864026343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e84102370 0x7f0e84102790 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.862+0000 7f0e8ac00700 1 --2- 172.21.15.40:0/1864026343 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e84103570 0x7f0e841039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.862+0000 7f0e8ac00700 1 -- 172.21.15.40:0/1864026343 >> 172.21.15.40:0/1864026343 conn(0x7f0e840fd910 msgr2=0x7f0e840ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.862+0000 7f0e8ac00700 1 -- 172.21.15.40:0/1864026343 shutdown_connections 2024-04-27T00:01:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.862+0000 7f0e8ac00700 1 -- 172.21.15.40:0/1864026343 wait complete. 2024-04-27T00:01:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.863+0000 7f0e8ac00700 1 Processor -- start 2024-04-27T00:01:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.864+0000 7f0e8ac00700 1 -- start start 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.864+0000 7f0e8ac00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e84102370 0x7f0e841062b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.864+0000 7f0e8ac00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e84103570 0x7f0e84104950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.864+0000 7f0e8ac00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e84103f10 0x7f0e84104e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.864+0000 7f0e8ac00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8410de30 con 0x7f0e84102370 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.864+0000 7f0e8ac00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8410dcb0 con 0x7f0e84103f10 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.864+0000 7f0e8ac00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8410dfb0 con 0x7f0e84103570 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.864+0000 7f0e83400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e84103570 0x7f0e84104950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.864+0000 7f0e83e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e84102370 0x7f0e841062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.864+0000 7f0e83400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e84103570 0x7f0e84104950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50272/0 (socket says 172.21.15.40:50272) 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.864+0000 7f0e83400700 1 -- 172.21.15.40:0/2610757198 learned_addr learned my addr 172.21.15.40:0/2610757198 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.864+0000 7f0e88c00700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e84103f10 0x7f0e84104e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.865+0000 7f0e83400700 1 -- 172.21.15.40:0/2610757198 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e84103f10 msgr2=0x7f0e84104e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.865+0000 7f0e83400700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e84103f10 0x7f0e84104e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.865+0000 7f0e83400700 1 -- 172.21.15.40:0/2610757198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e84102370 msgr2=0x7f0e841062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.865+0000 7f0e83400700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e84102370 0x7f0e841062b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.865+0000 7f0e83400700 1 -- 172.21.15.40:0/2610757198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e84105760 con 0x7f0e84103570 2024-04-27T00:01:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.865+0000 7f0e83400700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e84103570 0x7f0e84104950 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000d950 tx=0x7f0e7000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.866+0000 7f0e80c00700 1 -- 172.21.15.40:0/2610757198 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e70016070 con 0x7f0e84103570 2024-04-27T00:01:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.866+0000 7f0e8ac00700 1 -- 172.21.15.40:0/2610757198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e841a3770 con 0x7f0e84103570 2024-04-27T00:01:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.866+0000 7f0e80c00700 1 -- 172.21.15.40:0/2610757198 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e7000fd40 con 0x7f0e84103570 2024-04-27T00:01:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.866+0000 7f0e80c00700 1 -- 172.21.15.40:0/2610757198 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7000e3b0 con 0x7f0e84103570 2024-04-27T00:01:19.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.866+0000 7f0e8ac00700 1 -- 172.21.15.40:0/2610757198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e841a3cd0 con 0x7f0e84103570 2024-04-27T00:01:19.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.867+0000 7f0e8ac00700 1 -- 172.21.15.40:0/2610757198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e8406a110 con 0x7f0e84103570 2024-04-27T00:01:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.870+0000 7f0e80c00700 1 -- 172.21.15.40:0/2610757198 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0e70019070 con 0x7f0e84103570 2024-04-27T00:01:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.871+0000 7f0e80c00700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0e680a3cd0 0x7f0e680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.871+0000 7f0e80c00700 1 -- 172.21.15.40:0/2610757198 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f0e700d0380 con 0x7f0e84103570 2024-04-27T00:01:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.871+0000 7f0e83e00700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0e680a3cd0 0x7f0e680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.872+0000 7f0e83e00700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0e680a3cd0 0x7f0e680a6190 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400b580 tx=0x7f0e74005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:19.876+0000 7f0e80c00700 1 -- 172.21.15.40:0/2610757198 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0e7009c8c0 con 0x7f0e84103570 2024-04-27T00:01:20.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.112+0000 7f0e8ac00700 1 -- 172.21.15.40:0/2610757198 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f0e8406c080 con 0x7f0e680a3cd0 2024-04-27T00:01:20.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.116+0000 7f0e80c00700 1 -- 172.21.15.40:0/2610757198 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f0e8406c080 con 0x7f0e680a3cd0 2024-04-27T00:01:20.116 INFO:teuthology.orchestra.run.smithi040.stdout:HOST ADDR LABELS STATUS 2024-04-27T00:01:20.116 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040 172.21.15.40 2024-04-27T00:01:20.117 INFO:teuthology.orchestra.run.smithi040.stdout:smithi080 172.21.15.80 2024-04-27T00:01:20.117 INFO:teuthology.orchestra.run.smithi040.stdout:smithi103 172.21.15.103 2024-04-27T00:01:20.117 INFO:teuthology.orchestra.run.smithi040.stdout:3 hosts in cluster 2024-04-27T00:01:20.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.119+0000 7f0e8ac00700 1 -- 172.21.15.40:0/2610757198 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0e680a3cd0 msgr2=0x7f0e680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:20.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.119+0000 7f0e8ac00700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0e680a3cd0 0x7f0e680a6190 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400b580 tx=0x7f0e74005fb0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.119+0000 7f0e8ac00700 1 -- 172.21.15.40:0/2610757198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e84103570 msgr2=0x7f0e84104950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:20.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.119+0000 7f0e8ac00700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e84103570 0x7f0e84104950 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000d950 tx=0x7f0e7000dc60 comp rx=0 tx=0).stop 2024-04-27T00:01:20.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.120+0000 7f0e8ac00700 1 -- 172.21.15.40:0/2610757198 shutdown_connections 2024-04-27T00:01:20.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.120+0000 7f0e8ac00700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0e680a3cd0 0x7f0e680a6190 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.120+0000 7f0e8ac00700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e84103f10 0x7f0e84104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.120+0000 7f0e8ac00700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e84102370 0x7f0e841062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.120+0000 7f0e8ac00700 1 --2- 172.21.15.40:0/2610757198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e84103570 0x7f0e84104950 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.120+0000 7f0e8ac00700 1 -- 172.21.15.40:0/2610757198 >> 172.21.15.40:0/2610757198 conn(0x7f0e840fd910 msgr2=0x7f0e8406b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:20.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.120+0000 7f0e8ac00700 1 -- 172.21.15.40:0/2610757198 shutdown_connections 2024-04-27T00:01:20.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:20.120+0000 7f0e8ac00700 1 -- 172.21.15.40:0/2610757198 wait complete. 2024-04-27T00:01:20.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:20 smithi080 ceph-mon[32362]: pgmap v399: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:20.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:20 smithi103 ceph-mon[32952]: pgmap v399: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:20.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:20 smithi040 ceph-mon[28095]: pgmap v399: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:20.916 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-27T00:01:21.357 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:01:21.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:21 smithi080 ceph-mon[32362]: from='client.25237 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:21.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:21 smithi040 ceph-mon[28095]: from='client.25237 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:21.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:21 smithi103 ceph-mon[32952]: from='client.25237 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:22.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:22 smithi040 ceph-mon[28095]: pgmap v400: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:01:22.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:22 smithi080 ceph-mon[32362]: pgmap v400: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:01:22.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:22 smithi103 ceph-mon[32952]: pgmap v400: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:01:23.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.727+0000 7f1784e00700 1 -- 172.21.15.40:0/2591604508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1780105010 msgr2=0x7f178010b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:23.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.727+0000 7f1784e00700 1 --2- 172.21.15.40:0/2591604508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1780105010 0x7f178010b8c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f177800b0d0 tx=0x7f177800b3e0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.727+0000 7f1784e00700 1 -- 172.21.15.40:0/2591604508 shutdown_connections 2024-04-27T00:01:23.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.727+0000 7f1784e00700 1 --2- 172.21.15.40:0/2591604508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1780105010 0x7f178010b8c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.727+0000 7f1784e00700 1 --2- 172.21.15.40:0/2591604508 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1780103470 0x7f1780103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.727+0000 7f1784e00700 1 --2- 172.21.15.40:0/2591604508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1780104670 0x7f1780104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.727+0000 7f1784e00700 1 -- 172.21.15.40:0/2591604508 >> 172.21.15.40:0/2591604508 conn(0x7f17800fea10 msgr2=0x7f1780100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:23.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.728+0000 7f1784e00700 1 -- 172.21.15.40:0/2591604508 shutdown_connections 2024-04-27T00:01:23.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.728+0000 7f1784e00700 1 -- 172.21.15.40:0/2591604508 wait complete. 2024-04-27T00:01:23.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.729+0000 7f1784e00700 1 Processor -- start 2024-04-27T00:01:23.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.729+0000 7f1784e00700 1 -- start start 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.729+0000 7f1784e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1780103470 0x7f17800771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.729+0000 7f1784e00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1780104670 0x7f1780075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.729+0000 7f1784e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1780105010 0x7f1780075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.729+0000 7f1784e00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f178010ef60 con 0x7f1780105010 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.729+0000 7f1784e00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f178010ede0 con 0x7f1780104670 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.729+0000 7f1784e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f178010f0e0 con 0x7f1780103470 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.730+0000 7f177d400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1780104670 0x7f1780075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.730+0000 7f177de00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1780103470 0x7f17800771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.730+0000 7f177e800700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1780105010 0x7f1780075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.730+0000 7f177d400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1780104670 0x7f1780075810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:42080/0 (socket says 172.21.15.40:42080) 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.730+0000 7f177d400700 1 -- 172.21.15.40:0/2008671882 learned_addr learned my addr 172.21.15.40:0/2008671882 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.730+0000 7f177e800700 1 -- 172.21.15.40:0/2008671882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1780103470 msgr2=0x7f17800771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.730+0000 7f177e800700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1780103470 0x7f17800771c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.730+0000 7f177e800700 1 -- 172.21.15.40:0/2008671882 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1780104670 msgr2=0x7f1780075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:23.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.730+0000 7f177e800700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1780104670 0x7f1780075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.730+0000 7f177e800700 1 -- 172.21.15.40:0/2008671882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1780076600 con 0x7f1780105010 2024-04-27T00:01:23.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.730+0000 7f177d400700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1780104670 0x7f1780075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:23.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.731+0000 7f177e800700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1780105010 0x7f1780075d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1778006070 tx=0x7f1778004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:23.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.731+0000 7f1776a00700 1 -- 172.21.15.40:0/2008671882 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177801f070 con 0x7f1780105010 2024-04-27T00:01:23.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.731+0000 7f1776a00700 1 -- 172.21.15.40:0/2008671882 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1778007b80 con 0x7f1780105010 2024-04-27T00:01:23.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.731+0000 7f1784e00700 1 -- 172.21.15.40:0/2008671882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17801a36b0 con 0x7f1780105010 2024-04-27T00:01:23.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.731+0000 7f1776a00700 1 -- 172.21.15.40:0/2008671882 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177801b690 con 0x7f1780105010 2024-04-27T00:01:23.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.731+0000 7f1784e00700 1 -- 172.21.15.40:0/2008671882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17801a3b00 con 0x7f1780105010 2024-04-27T00:01:23.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.732+0000 7f1784e00700 1 -- 172.21.15.40:0/2008671882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f178006a110 con 0x7f1780105010 2024-04-27T00:01:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.734+0000 7f1776a00700 1 -- 172.21.15.40:0/2008671882 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1778022070 con 0x7f1780105010 2024-04-27T00:01:23.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.736+0000 7f1776a00700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f17640a3cd0 0x7f17640a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:23.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.737+0000 7f1776a00700 1 -- 172.21.15.40:0/2008671882 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f177800f020 con 0x7f1780105010 2024-04-27T00:01:23.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.737+0000 7f177de00700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f17640a3cd0 0x7f17640a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:23.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.738+0000 7f177de00700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f17640a3cd0 0x7f17640a6190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f17801044d0 tx=0x7f1770009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:23.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.742+0000 7f1776a00700 1 -- 172.21.15.40:0/2008671882 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1778029020 con 0x7f1780105010 2024-04-27T00:01:23.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.985+0000 7f1784e00700 1 -- 172.21.15.40:0/2008671882 --> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f1780063860 con 0x7f17640a3cd0 2024-04-27T00:01:23.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.988+0000 7f1776a00700 1 -- 172.21.15.40:0/2008671882 <== mgr.14152 v2:172.21.15.40:6800/3777720410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f1780063860 con 0x7f17640a3cd0 2024-04-27T00:01:23.989 INFO:teuthology.orchestra.run.smithi040.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-27T00:01:23.989 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009J400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-27T00:01:23.989 INFO:teuthology.orchestra.run.smithi040.stdout:smithi080 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204014T400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-27T00:01:23.989 INFO:teuthology.orchestra.run.smithi040.stdout:smithi103 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204012Z400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-27T00:01:23.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.991+0000 7f1784e00700 1 -- 172.21.15.40:0/2008671882 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f17640a3cd0 msgr2=0x7f17640a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:23.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.991+0000 7f1784e00700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f17640a3cd0 0x7f17640a6190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f17801044d0 tx=0x7f1770009260 comp rx=0 tx=0).stop 2024-04-27T00:01:23.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.991+0000 7f1784e00700 1 -- 172.21.15.40:0/2008671882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1780105010 msgr2=0x7f1780075d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:23.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.991+0000 7f1784e00700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1780105010 0x7f1780075d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1778006070 tx=0x7f1778004c80 comp rx=0 tx=0).stop 2024-04-27T00:01:23.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.992+0000 7f1784e00700 1 -- 172.21.15.40:0/2008671882 shutdown_connections 2024-04-27T00:01:23.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.992+0000 7f1784e00700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f17640a3cd0 0x7f17640a6190 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.992+0000 7f1784e00700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1780105010 0x7f1780075d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.992+0000 7f1784e00700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1780103470 0x7f17800771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.992+0000 7f1784e00700 1 --2- 172.21.15.40:0/2008671882 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1780104670 0x7f1780075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.992+0000 7f1784e00700 1 -- 172.21.15.40:0/2008671882 >> 172.21.15.40:0/2008671882 conn(0x7f17800fea10 msgr2=0x7f1780107970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:23.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.992+0000 7f1784e00700 1 -- 172.21.15.40:0/2008671882 shutdown_connections 2024-04-27T00:01:23.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:23.992+0000 7f1784e00700 1 -- 172.21.15.40:0/2008671882 wait complete. 2024-04-27T00:01:24.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:24 smithi040 ceph-mon[28095]: pgmap v401: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:24.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:24 smithi080 ceph-mon[32362]: pgmap v401: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:24.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:24 smithi103 ceph-mon[32952]: pgmap v401: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:24.908 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T00:01:24.968 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-04-27T00:01:24.968 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-27T00:01:25.437 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:01:25.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:25 smithi080 ceph-mon[32362]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:25.800 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:25 smithi040 ceph-mon[28095]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:25.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:25 smithi103 ceph-mon[32952]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:26.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:26 smithi040 ceph-mon[28095]: pgmap v402: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:26.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:26 smithi103 ceph-mon[32952]: pgmap v402: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:27.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:26 smithi080 ceph-mon[32362]: pgmap v402: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:27.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.798+0000 7fe4adc00700 1 -- 172.21.15.40:0/1202667507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4a81068c0 msgr2=0x7fe4a8106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:27.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.798+0000 7fe4adc00700 1 --2- 172.21.15.40:0/1202667507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4a81068c0 0x7fe4a8106d20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe4900099c0 tx=0x7fe490009cd0 comp rx=0 tx=0).stop 2024-04-27T00:01:27.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.798+0000 7fe4adc00700 1 -- 172.21.15.40:0/1202667507 shutdown_connections 2024-04-27T00:01:27.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.798+0000 7fe4adc00700 1 --2- 172.21.15.40:0/1202667507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4a8107260 0x7fe4a810db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:27.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.798+0000 7fe4adc00700 1 --2- 172.21.15.40:0/1202667507 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4a81056c0 0x7fe4a8105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:27.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.798+0000 7fe4adc00700 1 --2- 172.21.15.40:0/1202667507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4a81068c0 0x7fe4a8106d20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:27.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.798+0000 7fe4adc00700 1 -- 172.21.15.40:0/1202667507 >> 172.21.15.40:0/1202667507 conn(0x7fe4a8100c80 msgr2=0x7fe4a81030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:27.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.798+0000 7fe4adc00700 1 -- 172.21.15.40:0/1202667507 shutdown_connections 2024-04-27T00:01:27.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.798+0000 7fe4adc00700 1 -- 172.21.15.40:0/1202667507 wait complete. 2024-04-27T00:01:27.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.799+0000 7fe4adc00700 1 Processor -- start 2024-04-27T00:01:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.799+0000 7fe4adc00700 1 -- start start 2024-04-27T00:01:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4adc00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4a81056c0 0x7fe4a819b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4adc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4a81068c0 0x7fe4a819bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4adc00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4a8107260 0x7fe4a81a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4adc00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a8111180 con 0x7fe4a8107260 2024-04-27T00:01:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4adc00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a8111000 con 0x7fe4a81056c0 2024-04-27T00:01:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4adc00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a8111300 con 0x7fe4a81068c0 2024-04-27T00:01:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4a6a00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4a81056c0 0x7fe4a819b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4a6a00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4a81056c0 0x7fe4a819b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:42084/0 (socket says 172.21.15.40:42084) 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4a6a00700 1 -- 172.21.15.40:0/2882364945 learned_addr learned my addr 172.21.15.40:0/2882364945 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe49e000700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4a81068c0 0x7fe4a819bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4a7400700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4a8107260 0x7fe4a81a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4a6a00700 1 -- 172.21.15.40:0/2882364945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4a81068c0 msgr2=0x7fe4a819bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4a6a00700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4a81068c0 0x7fe4a819bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4a6a00700 1 -- 172.21.15.40:0/2882364945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4a8107260 msgr2=0x7fe4a81a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4a6a00700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4a8107260 0x7fe4a81a2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4a6a00700 1 -- 172.21.15.40:0/2882364945 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a81a3400 con 0x7fe4a81056c0 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.800+0000 7fe4a7400700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4a8107260 0x7fe4a81a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.801+0000 7fe4a6a00700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4a81056c0 0x7fe4a819b7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe49800e970 tx=0x7fe49800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.801+0000 7fe49fe00700 1 -- 172.21.15.40:0/2882364945 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49800cbd0 con 0x7fe4a81056c0 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.801+0000 7fe4adc00700 1 -- 172.21.15.40:0/2882364945 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a81a36f0 con 0x7fe4a81056c0 2024-04-27T00:01:27.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.801+0000 7fe49fe00700 1 -- 172.21.15.40:0/2882364945 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4980044d0 con 0x7fe4a81056c0 2024-04-27T00:01:27.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.801+0000 7fe49fe00700 1 -- 172.21.15.40:0/2882364945 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe498003df0 con 0x7fe4a81056c0 2024-04-27T00:01:27.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.801+0000 7fe4adc00700 1 -- 172.21.15.40:0/2882364945 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a81a3c80 con 0x7fe4a81056c0 2024-04-27T00:01:27.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.802+0000 7fe4adc00700 1 -- 172.21.15.40:0/2882364945 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4a806a110 con 0x7fe4a81056c0 2024-04-27T00:01:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.804+0000 7fe49fe00700 1 -- 172.21.15.40:0/2882364945 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe498003f90 con 0x7fe4a81056c0 2024-04-27T00:01:27.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.806+0000 7fe49fe00700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe48c0a3cd0 0x7fe48c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:27.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.806+0000 7fe49fe00700 1 -- 172.21.15.40:0/2882364945 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fe4980d0b80 con 0x7fe4a81056c0 2024-04-27T00:01:27.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.806+0000 7fe49e000700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe48c0a3cd0 0x7fe48c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:27.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.807+0000 7fe49e000700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe48c0a3cd0 0x7fe48c0a6190 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe4900036f0 tx=0x7fe490005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:27.810+0000 7fe49fe00700 1 -- 172.21.15.40:0/2882364945 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe49809d060 con 0x7fe4a81056c0 2024-04-27T00:01:27.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:27 smithi040 ceph-mon[28095]: pgmap v403: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:28.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:27 smithi080 ceph-mon[32362]: pgmap v403: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:28.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.087+0000 7fe4adc00700 1 -- 172.21.15.40:0/2882364945 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fe4a8064030 con 0x7fe4a81056c0 2024-04-27T00:01:28.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.088+0000 7fe49fe00700 1 -- 172.21.15.40:0/2882364945 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) ==== 76+0+3052 (secure 0 0 0) 0x7fe49809c960 con 0x7fe4a81056c0 2024-04-27T00:01:28.090 INFO:teuthology.orchestra.run.smithi040.stdout:e23 2024-04-27T00:01:28.090 INFO:teuthology.orchestra.run.smithi040.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T00:01:28.090 INFO:teuthology.orchestra.run.smithi040.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-27T00:01:28.090 INFO:teuthology.orchestra.run.smithi040.stdout:legacy client fscid: 1 2024-04-27T00:01:28.091 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:28.091 INFO:teuthology.orchestra.run.smithi040.stdout:Filesystem 'cephfs' (1) 2024-04-27T00:01:28.092 INFO:teuthology.orchestra.run.smithi040.stdout:fs_name cephfs 2024-04-27T00:01:28.092 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 23 2024-04-27T00:01:28.092 INFO:teuthology.orchestra.run.smithi040.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-27T00:01:28.092 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-04-26T23:58:50.616677+0000 2024-04-27T00:01:28.092 INFO:teuthology.orchestra.run.smithi040.stdout:modified 2024-04-26T23:59:25.374905+0000 2024-04-27T00:01:28.092 INFO:teuthology.orchestra.run.smithi040.stdout:tableserver 0 2024-04-27T00:01:28.093 INFO:teuthology.orchestra.run.smithi040.stdout:root 0 2024-04-27T00:01:28.093 INFO:teuthology.orchestra.run.smithi040.stdout:session_timeout 300 2024-04-27T00:01:28.093 INFO:teuthology.orchestra.run.smithi040.stdout:session_autoclose 300 2024-04-27T00:01:28.093 INFO:teuthology.orchestra.run.smithi040.stdout:max_file_size 1099511627776 2024-04-27T00:01:28.093 INFO:teuthology.orchestra.run.smithi040.stdout:max_xattr_size 65536 2024-04-27T00:01:28.093 INFO:teuthology.orchestra.run.smithi040.stdout:required_client_features {} 2024-04-27T00:01:28.094 INFO:teuthology.orchestra.run.smithi040.stdout:last_failure 0 2024-04-27T00:01:28.094 INFO:teuthology.orchestra.run.smithi040.stdout:last_failure_osd_epoch 0 2024-04-27T00:01:28.094 INFO:teuthology.orchestra.run.smithi040.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-27T00:01:28.094 INFO:teuthology.orchestra.run.smithi040.stdout:max_mds 3 2024-04-27T00:01:28.094 INFO:teuthology.orchestra.run.smithi040.stdout:in 0,1,2 2024-04-27T00:01:28.094 INFO:teuthology.orchestra.run.smithi040.stdout:up {0=24590,1=24560,2=24559} 2024-04-27T00:01:28.094 INFO:teuthology.orchestra.run.smithi040.stdout:failed 2024-04-27T00:01:28.094 INFO:teuthology.orchestra.run.smithi040.stdout:damaged 2024-04-27T00:01:28.094 INFO:teuthology.orchestra.run.smithi040.stdout:stopped 2024-04-27T00:01:28.095 INFO:teuthology.orchestra.run.smithi040.stdout:data_pools [3,4] 2024-04-27T00:01:28.095 INFO:teuthology.orchestra.run.smithi040.stdout:metadata_pool 2 2024-04-27T00:01:28.095 INFO:teuthology.orchestra.run.smithi040.stdout:inline_data disabled 2024-04-27T00:01:28.095 INFO:teuthology.orchestra.run.smithi040.stdout:balancer 2024-04-27T00:01:28.095 INFO:teuthology.orchestra.run.smithi040.stdout:bal_rank_mask -1 2024-04-27T00:01:28.095 INFO:teuthology.orchestra.run.smithi040.stdout:standby_count_wanted 0 2024-04-27T00:01:28.095 INFO:teuthology.orchestra.run.smithi040.stdout:qdb_cluster leader: 24590 members: 24559,24560,24590 2024-04-27T00:01:28.095 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.b{0:24590} state up:active seq 3 addr [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] compat {c=[1],r=[1],i=[1fff]}] 2024-04-27T00:01:28.095 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.c{0:24535} state up:standby-replay seq 1 addr [v2:172.21.15.103:6836/2206925949,v1:172.21.15.103:6837/2206925949] compat {c=[1],r=[1],i=[1fff]}] 2024-04-27T00:01:28.095 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.g{1:24560} state up:active seq 13 addr [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] compat {c=[1],r=[1],i=[1fff]}] 2024-04-27T00:01:28.095 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.h{1:24533} state up:standby-replay seq 1 addr [v2:172.21.15.80:6836/4116866934,v1:172.21.15.80:6837/4116866934] compat {c=[1],r=[1],i=[1fff]}] 2024-04-27T00:01:28.095 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.i{2:24559} state up:active seq 14 addr [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] compat {c=[1],r=[1],i=[1fff]}] 2024-04-27T00:01:28.096 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.f{2:24523} state up:standby-replay seq 1 addr [v2:172.21.15.103:6834/2596993377,v1:172.21.15.103:6835/2596993377] compat {c=[1],r=[1],i=[1fff]}] 2024-04-27T00:01:28.096 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:28.096 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:28.096 INFO:teuthology.orchestra.run.smithi040.stdout:Standby daemons: 2024-04-27T00:01:28.096 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:28.096 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.j{-1:14379} state up:standby seq 1 addr [v2:172.21.15.40:6834/1384127498,v1:172.21.15.40:6835/1384127498] compat {c=[1],r=[1],i=[1fff]}] 2024-04-27T00:01:28.096 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.a{-1:14457} state up:standby seq 1 addr [v2:172.21.15.40:6840/2265538751,v1:172.21.15.40:6841/2265538751] compat {c=[1],r=[1],i=[1fff]}] 2024-04-27T00:01:28.097 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.k{-1:24452} state up:standby seq 1 addr [v2:172.21.15.80:6832/3975801652,v1:172.21.15.80:6833/3975801652] compat {c=[1],r=[1],i=[1fff]}] 2024-04-27T00:01:28.097 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.l{-1:24454} state up:standby seq 1 addr [v2:172.21.15.103:6832/970790712,v1:172.21.15.103:6833/970790712] compat {c=[1],r=[1],i=[1fff]}] 2024-04-27T00:01:28.097 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.d{-1:24493} state up:standby seq 1 addr [v2:172.21.15.40:6836/3497909120,v1:172.21.15.40:6837/3497909120] compat {c=[1],r=[1],i=[1fff]}] 2024-04-27T00:01:28.097 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.e{-1:24500} state up:standby seq 1 addr [v2:172.21.15.80:6834/1031714719,v1:172.21.15.80:6835/1031714719] compat {c=[1],r=[1],i=[1fff]}] 2024-04-27T00:01:28.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:27 smithi103 ceph-mon[32952]: pgmap v403: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:28.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.091+0000 7fe4adc00700 1 -- 172.21.15.40:0/2882364945 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe48c0a3cd0 msgr2=0x7fe48c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:28.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.091+0000 7fe4adc00700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe48c0a3cd0 0x7fe48c0a6190 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe4900036f0 tx=0x7fe490005fd0 comp rx=0 tx=0).stop 2024-04-27T00:01:28.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.091+0000 7fe4adc00700 1 -- 172.21.15.40:0/2882364945 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4a81056c0 msgr2=0x7fe4a819b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:28.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.091+0000 7fe4adc00700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4a81056c0 0x7fe4a819b7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe49800e970 tx=0x7fe49800ec80 comp rx=0 tx=0).stop 2024-04-27T00:01:28.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.091+0000 7fe4adc00700 1 -- 172.21.15.40:0/2882364945 shutdown_connections 2024-04-27T00:01:28.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.091+0000 7fe4adc00700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe48c0a3cd0 0x7fe48c0a6190 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:28.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.091+0000 7fe4adc00700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4a8107260 0x7fe4a81a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:28.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.091+0000 7fe4adc00700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe4a81056c0 0x7fe4a819b7a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:28.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.091+0000 7fe4adc00700 1 --2- 172.21.15.40:0/2882364945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4a81068c0 0x7fe4a819bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:28.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.091+0000 7fe4adc00700 1 -- 172.21.15.40:0/2882364945 >> 172.21.15.40:0/2882364945 conn(0x7fe4a8100c80 msgr2=0x7fe4a8109bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:28.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.091+0000 7fe4adc00700 1 -- 172.21.15.40:0/2882364945 shutdown_connections 2024-04-27T00:01:28.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:28.091+0000 7fe4adc00700 1 -- 172.21.15.40:0/2882364945 wait complete. 2024-04-27T00:01:28.101 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 23 2024-04-27T00:01:28.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-27T00:01:29.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:28 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2882364945' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T00:01:29.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:28 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2882364945' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T00:01:29.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:28 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2882364945' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-27T00:01:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:01:29.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:29 smithi040 ceph-mon[28095]: pgmap v404: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:30.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:29 smithi080 ceph-mon[32362]: pgmap v404: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:30.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:29 smithi103 ceph-mon[32952]: pgmap v404: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:31.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.750+0000 7f515a400700 1 -- 172.21.15.40:0/3758667405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5154109100 msgr2=0x7f515410b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:31.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.750+0000 7f515a400700 1 --2- 172.21.15.40:0/3758667405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5154109100 0x7f515410b510 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f514c009f90 tx=0x7f514c00b330 comp rx=0 tx=0).stop 2024-04-27T00:01:31.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.750+0000 7f515a400700 1 -- 172.21.15.40:0/3758667405 shutdown_connections 2024-04-27T00:01:31.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.750+0000 7f515a400700 1 --2- 172.21.15.40:0/3758667405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5154109100 0x7f515410b510 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:31.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.750+0000 7f515a400700 1 --2- 172.21.15.40:0/3758667405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5154103ea0 0x7f5154106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:31.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.750+0000 7f515a400700 1 --2- 172.21.15.40:0/3758667405 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f51541067d0 0x7f5154108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:31.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.750+0000 7f515a400700 1 -- 172.21.15.40:0/3758667405 >> 172.21.15.40:0/3758667405 conn(0x7f51540fd830 msgr2=0x7f51540ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:31.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.750+0000 7f515a400700 1 -- 172.21.15.40:0/3758667405 shutdown_connections 2024-04-27T00:01:31.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.750+0000 7f515a400700 1 -- 172.21.15.40:0/3758667405 wait complete. 2024-04-27T00:01:31.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.751+0000 7f515a400700 1 Processor -- start 2024-04-27T00:01:31.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.752+0000 7f515a400700 1 -- start start 2024-04-27T00:01:31.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.752+0000 7f515a400700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5154103ea0 0x7f5154103800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:31.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.752+0000 7f515a400700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51541067d0 0x7f5154101e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:31.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.752+0000 7f515a400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5154109100 0x7f5154102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:31.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.752+0000 7f515a400700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f515410ee70 con 0x7f51541067d0 2024-04-27T00:01:31.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.752+0000 7f515a400700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f515410ecf0 con 0x7f5154103ea0 2024-04-27T00:01:31.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.752+0000 7f515a400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f515410eff0 con 0x7f5154109100 2024-04-27T00:01:31.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.752+0000 7f5159000700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5154103ea0 0x7f5154103800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:31.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.752+0000 7f5153e00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51541067d0 0x7f5154101e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:31.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.752+0000 7f5159000700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5154103ea0 0x7f5154103800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49182/0 (socket says 172.21.15.40:49182) 2024-04-27T00:01:31.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.752+0000 7f5159000700 1 -- 172.21.15.40:0/1050783699 learned_addr learned my addr 172.21.15.40:0/1050783699 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:31.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.752+0000 7f5159a00700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5154109100 0x7f5154102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:31.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.753+0000 7f5159000700 1 -- 172.21.15.40:0/1050783699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5154109100 msgr2=0x7f5154102390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.753+0000 7f5159000700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5154109100 0x7f5154102390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.753+0000 7f5159000700 1 -- 172.21.15.40:0/1050783699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51541067d0 msgr2=0x7f5154101e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.753+0000 7f5159000700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51541067d0 0x7f5154101e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.753+0000 7f5159000700 1 -- 172.21.15.40:0/1050783699 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5154102c40 con 0x7f5154103ea0 2024-04-27T00:01:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.753+0000 7f5159a00700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5154109100 0x7f5154102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.753+0000 7f5153e00700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51541067d0 0x7f5154101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.753+0000 7f5159000700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5154103ea0 0x7f5154103800 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f514800e9c0 tx=0x7f514800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.753+0000 7f5151600700 1 -- 172.21.15.40:0/1050783699 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5148004a30 con 0x7f5154103ea0 2024-04-27T00:01:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.754+0000 7f515a400700 1 -- 172.21.15.40:0/1050783699 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51541a35c0 con 0x7f5154103ea0 2024-04-27T00:01:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.754+0000 7f5151600700 1 -- 172.21.15.40:0/1050783699 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f514800cc70 con 0x7f5154103ea0 2024-04-27T00:01:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.754+0000 7f5151600700 1 -- 172.21.15.40:0/1050783699 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51480054a0 con 0x7f5154103ea0 2024-04-27T00:01:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.754+0000 7f515a400700 1 -- 172.21.15.40:0/1050783699 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51541a3ad0 con 0x7f5154103ea0 2024-04-27T00:01:31.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.755+0000 7f515a400700 1 -- 172.21.15.40:0/1050783699 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f515406a110 con 0x7f5154103ea0 2024-04-27T00:01:31.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.757+0000 7f5151600700 1 -- 172.21.15.40:0/1050783699 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5148004bd0 con 0x7f5154103ea0 2024-04-27T00:01:31.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.758+0000 7f5151600700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f513c0a3cd0 0x7f513c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:31.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.759+0000 7f5151600700 1 -- 172.21.15.40:0/1050783699 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f51480d0a00 con 0x7f5154103ea0 2024-04-27T00:01:31.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.759+0000 7f5153e00700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f513c0a3cd0 0x7f513c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:31.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.759+0000 7f5153e00700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f513c0a3cd0 0x7f513c0a6190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5140011490 tx=0x7f5140009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:31.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.763+0000 7f5151600700 1 -- 172.21.15.40:0/1050783699 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f514809cee0 con 0x7f5154103ea0 2024-04-27T00:01:31.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.996+0000 7f515a400700 1 -- 172.21.15.40:0/1050783699 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f515402e490 con 0x7f5154103ea0 2024-04-27T00:01:31.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:31.997+0000 7f5151600700 1 -- 172.21.15.40:0/1050783699 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v92) ==== 56+0+5565 (secure 0 0 0) 0x7f514809c7e0 con 0x7f5154103ea0 2024-04-27T00:01:32.001 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 92 2024-04-27T00:01:32.001 INFO:teuthology.orchestra.run.smithi040.stdout:fsid 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-27T00:01:32.001 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-04-26T23:49:26.448890+0000 2024-04-27T00:01:32.002 INFO:teuthology.orchestra.run.smithi040.stdout:modified 2024-04-27T00:00:46.365303+0000 2024-04-27T00:01:32.002 INFO:teuthology.orchestra.run.smithi040.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-27T00:01:32.002 INFO:teuthology.orchestra.run.smithi040.stdout:crush_version 27 2024-04-27T00:01:32.002 INFO:teuthology.orchestra.run.smithi040.stdout:full_ratio 0.95 2024-04-27T00:01:32.002 INFO:teuthology.orchestra.run.smithi040.stdout:backfillfull_ratio 0.9 2024-04-27T00:01:32.002 INFO:teuthology.orchestra.run.smithi040.stdout:nearfull_ratio 0.85 2024-04-27T00:01:32.002 INFO:teuthology.orchestra.run.smithi040.stdout:require_min_compat_client luminous 2024-04-27T00:01:32.002 INFO:teuthology.orchestra.run.smithi040.stdout:min_compat_client luminous 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:require_osd_release squid 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:stretch_mode_enabled false 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:max_osd 12 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 up in weight 1 up_from 8 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6802/1203247598,v1:172.21.15.40:6803/1203247598] [v2:172.21.15.40:6804/1203247598,v1:172.21.15.40:6805/1203247598] exists,up 54e9c732-e7a5-45f0-be16-dee18cc70647 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 up in weight 1 up_from 13 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.80:6800/3368601093,v1:172.21.15.80:6801/3368601093] [v2:172.21.15.80:6802/3368601093,v1:172.21.15.80:6803/3368601093] exists,up ee040674-8f99-4406-a7bf-a989815dce57 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 up in weight 1 up_from 18 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6800/3393467187,v1:172.21.15.103:6801/3393467187] [v2:172.21.15.103:6802/3393467187,v1:172.21.15.103:6803/3393467187] exists,up 2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3 up in weight 1 up_from 26 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6810/4161271893,v1:172.21.15.40:6811/4161271893] [v2:172.21.15.40:6812/4161271893,v1:172.21.15.40:6813/4161271893] exists,up 9c08a594-7548-44d3-b6cc-62b25ee078eb 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4 up in weight 1 up_from 34 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.80:6808/3602586790,v1:172.21.15.80:6809/3602586790] [v2:172.21.15.80:6810/3602586790,v1:172.21.15.80:6811/3602586790] exists,up 2e645527-e694-4cd0-99d7-035f5c36bed0 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5 up in weight 1 up_from 39 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6808/63420438,v1:172.21.15.103:6809/63420438] [v2:172.21.15.103:6810/63420438,v1:172.21.15.103:6811/63420438] exists,up 9f6e9855-4856-4a8a-a884-1826a9c29327 2024-04-27T00:01:32.003 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6 up in weight 1 up_from 47 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6818/3383463516,v1:172.21.15.40:6819/3383463516] [v2:172.21.15.40:6820/3383463516,v1:172.21.15.40:6821/3383463516] exists,up 6180405f-9925-491b-aee8-ecaa9fd39d15 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 up in weight 1 up_from 52 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.80:6816/1231184332,v1:172.21.15.80:6817/1231184332] [v2:172.21.15.80:6818/1231184332,v1:172.21.15.80:6819/1231184332] exists,up 9cd5c915-ae45-499a-8910-8619855fec2f 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8 up in weight 1 up_from 60 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6816/4109830511,v1:172.21.15.103:6817/4109830511] [v2:172.21.15.103:6818/4109830511,v1:172.21.15.103:6819/4109830511] exists,up 63176b7b-984c-4aac-be69-3fe6d316ac7e 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9 up in weight 1 up_from 68 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6826/1030365825,v1:172.21.15.40:6827/1030365825] [v2:172.21.15.40:6828/1030365825,v1:172.21.15.40:6829/1030365825] exists,up 0ed158cb-03c5-4930-b529-7d017cf5c31e 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.80:6824/1789667951,v1:172.21.15.80:6825/1789667951] [v2:172.21.15.80:6826/1789667951,v1:172.21.15.80:6827/1789667951] exists,up ba1ee814-e69d-492a-9b3c-64d53ba2e22c 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11 up in weight 1 up_from 81 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6824/2714925397,v1:172.21.15.103:6825/2714925397] [v2:172.21.15.103:6826/2714925397,v1:172.21.15.103:6827/2714925397] exists,up 7a4a8429-ed19-4687-9a13-251dfbf74878 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.0 [5,9] 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.6 [4,7] 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.e [4,6] 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.1e [3,9,5,8] 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.31 [4,8] 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 4.6 [9,11] 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 4.12 [5,10] 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 4.13 [9,8] 2024-04-27T00:01:32.004 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 4.18 [9,8] 2024-04-27T00:01:32.005 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/1959424465 expires 2024-04-27T23:50:44.546469+0000 2024-04-27T00:01:32.005 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6801/1568598447 expires 2024-04-27T23:50:44.546469+0000 2024-04-27T00:01:32.005 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/29889962 expires 2024-04-27T23:50:44.546469+0000 2024-04-27T00:01:32.005 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6800/673764112 expires 2024-04-27T23:50:09.399879+0000 2024-04-27T00:01:32.005 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/3782482730 expires 2024-04-27T23:50:09.399879+0000 2024-04-27T00:01:32.005 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6800/1568598447 expires 2024-04-27T23:50:44.546469+0000 2024-04-27T00:01:32.005 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/1079307273 expires 2024-04-27T23:50:09.399879+0000 2024-04-27T00:01:32.005 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6801/673764112 expires 2024-04-27T23:50:09.399879+0000 2024-04-27T00:01:32.005 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/4238592841 expires 2024-04-27T23:50:44.546469+0000 2024-04-27T00:01:32.005 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/4067589404 expires 2024-04-27T23:50:09.399879+0000 2024-04-27T00:01:32.006 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:31 smithi040 ceph-mon[28095]: pgmap v405: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:01:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:32.001+0000 7f515a400700 1 -- 172.21.15.40:0/1050783699 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f513c0a3cd0 msgr2=0x7f513c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:32.001+0000 7f515a400700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f513c0a3cd0 0x7f513c0a6190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5140011490 tx=0x7f5140009260 comp rx=0 tx=0).stop 2024-04-27T00:01:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:32.001+0000 7f515a400700 1 -- 172.21.15.40:0/1050783699 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5154103ea0 msgr2=0x7f5154103800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:32.001+0000 7f515a400700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5154103ea0 0x7f5154103800 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f514800e9c0 tx=0x7f514800ecd0 comp rx=0 tx=0).stop 2024-04-27T00:01:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:32.001+0000 7f515a400700 1 -- 172.21.15.40:0/1050783699 shutdown_connections 2024-04-27T00:01:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:32.001+0000 7f515a400700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f513c0a3cd0 0x7f513c0a6190 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:32.001+0000 7f515a400700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5154109100 0x7f5154102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:32.001+0000 7f515a400700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5154103ea0 0x7f5154103800 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:32.001+0000 7f515a400700 1 --2- 172.21.15.40:0/1050783699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51541067d0 0x7f5154101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:32.001+0000 7f515a400700 1 -- 172.21.15.40:0/1050783699 >> 172.21.15.40:0/1050783699 conn(0x7f51540fd830 msgr2=0x7f51541047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:32.001+0000 7f515a400700 1 -- 172.21.15.40:0/1050783699 shutdown_connections 2024-04-27T00:01:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:32.002+0000 7f515a400700 1 -- 172.21.15.40:0/1050783699 wait complete. 2024-04-27T00:01:32.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:31 smithi103 ceph-mon[32952]: pgmap v405: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:01:32.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:31 smithi080 ceph-mon[32362]: pgmap v405: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:01:32.854 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-27T00:01:33.011 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T00:01:33.095 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:32 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1050783699' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-27T00:01:33.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:32 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1050783699' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-27T00:01:33.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.236+0000 7fb5f5200640 1 -- 172.21.15.40:0/3568529724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f0108450 msgr2=0x7fb5f010a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:33.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.236+0000 7fb5f5200640 1 --2- 172.21.15.40:0/3568529724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f0108450 0x7fb5f010a8a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009900 tx=0x7fb5e802a3f0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.236+0000 7fb5f5200640 1 -- 172.21.15.40:0/3568529724 shutdown_connections 2024-04-27T00:01:33.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.236+0000 7fb5f5200640 1 --2- 172.21.15.40:0/3568529724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f0108450 0x7fb5f010a8a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.236+0000 7fb5f5200640 1 --2- 172.21.15.40:0/3568529724 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5f0105b20 0x7fb5f0107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.236+0000 7fb5f5200640 1 --2- 172.21.15.40:0/3568529724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f01031f0 0x7fb5f01055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.236+0000 7fb5f5200640 1 -- 172.21.15.40:0/3568529724 >> 172.21.15.40:0/3568529724 conn(0x7fb5f00fcbc0 msgr2=0x7fb5f00ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:33.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.236+0000 7fb5f5200640 1 -- 172.21.15.40:0/3568529724 shutdown_connections 2024-04-27T00:01:33.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.236+0000 7fb5f5200640 1 -- 172.21.15.40:0/3568529724 wait complete. 2024-04-27T00:01:33.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.238+0000 7fb5f5200640 1 Processor -- start 2024-04-27T00:01:33.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.238+0000 7fb5f5200640 1 -- start start 2024-04-27T00:01:33.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.238+0000 7fb5f5200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f01031f0 0x7fb5f0198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:33.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.238+0000 7fb5f5200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5f0105b20 0x7fb5f0198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:33.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.238+0000 7fb5f5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f0108450 0x7fb5f019fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:33.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.238+0000 7fb5f5200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f010e310 con 0x7fb5f0108450 2024-04-27T00:01:33.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.238+0000 7fb5f5200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f010e190 con 0x7fb5f0105b20 2024-04-27T00:01:33.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.238+0000 7fb5f5200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f010e490 con 0x7fb5f01031f0 2024-04-27T00:01:33.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.239+0000 7fb5eea00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5f0105b20 0x7fb5f0198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:33.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.239+0000 7fb5eea00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5f0105b20 0x7fb5f0198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49208/0 (socket says 172.21.15.40:49208) 2024-04-27T00:01:33.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.239+0000 7fb5efe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f0108450 0x7fb5f019fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:33.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.239+0000 7fb5eea00640 1 -- 172.21.15.40:0/3813690109 learned_addr learned my addr 172.21.15.40:0/3813690109 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:33.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.239+0000 7fb5ef400640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f01031f0 0x7fb5f0198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:33.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.239+0000 7fb5eea00640 1 -- 172.21.15.40:0/3813690109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f01031f0 msgr2=0x7fb5f0198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:33.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.239+0000 7fb5eea00640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f01031f0 0x7fb5f0198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.239+0000 7fb5eea00640 1 -- 172.21.15.40:0/3813690109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f0108450 msgr2=0x7fb5f019fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:33.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.239+0000 7fb5eea00640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f0108450 0x7fb5f019fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.239+0000 7fb5eea00640 1 -- 172.21.15.40:0/3813690109 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f01a0540 con 0x7fb5f0105b20 2024-04-27T00:01:33.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.239+0000 7fb5efe00640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f0108450 0x7fb5f019fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:33.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.239+0000 7fb5eea00640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5f0105b20 0x7fb5f0198e60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800e9a0 tx=0x7fb5d800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:33.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.240+0000 7fb5e7e00640 1 -- 172.21.15.40:0/3813690109 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d800cd80 con 0x7fb5f0105b20 2024-04-27T00:01:33.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.240+0000 7fb5e7e00640 1 -- 172.21.15.40:0/3813690109 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5d80035f0 con 0x7fb5f0105b20 2024-04-27T00:01:33.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.240+0000 7fb5e7e00640 1 -- 172.21.15.40:0/3813690109 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8010690 con 0x7fb5f0105b20 2024-04-27T00:01:33.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.240+0000 7fb5f5200640 1 -- 172.21.15.40:0/3813690109 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f01a0830 con 0x7fb5f0105b20 2024-04-27T00:01:33.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.240+0000 7fb5f5200640 1 -- 172.21.15.40:0/3813690109 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f01a0d80 con 0x7fb5f0105b20 2024-04-27T00:01:33.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.241+0000 7fb5f5200640 1 -- 172.21.15.40:0/3813690109 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5f00fe6b0 con 0x7fb5f0105b20 2024-04-27T00:01:33.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.243+0000 7fb5e7e00640 1 -- 172.21.15.40:0/3813690109 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb5d8004000 con 0x7fb5f0105b20 2024-04-27T00:01:33.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.244+0000 7fb5e7e00640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb5d40a3c40 0x7fb5d40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:33.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.244+0000 7fb5e7e00640 1 -- 172.21.15.40:0/3813690109 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fb5d80d0cc0 con 0x7fb5f0105b20 2024-04-27T00:01:33.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.245+0000 7fb5ef400640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb5d40a3c40 0x7fb5d40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:33.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.245+0000 7fb5ef400640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb5d40a3c40 0x7fb5d40a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0005fd0 tx=0x7fb5e0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:33.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.249+0000 7fb5e7e00640 1 -- 172.21.15.40:0/3813690109 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb5d80d7080 con 0x7fb5f0105b20 2024-04-27T00:01:33.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:32 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1050783699' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-27T00:01:33.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.458+0000 7fb5f5200640 1 -- 172.21.15.40:0/3813690109 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5f002dcd0 con 0x7fb5f0105b20 2024-04-27T00:01:33.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.460+0000 7fb5e7e00640 1 -- 172.21.15.40:0/3813690109 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24531 (secure 0 0 0) 0x7fb5d80042f0 con 0x7fb5f0105b20 2024-04-27T00:01:33.462 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:33.462 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-27T00:00:46.365303+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9},{"from":5,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T00:01:33.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.463+0000 7fb5f5200640 1 -- 172.21.15.40:0/3813690109 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb5d40a3c40 msgr2=0x7fb5d40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:33.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.463+0000 7fb5f5200640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb5d40a3c40 0x7fb5d40a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0005fd0 tx=0x7fb5e0009500 comp rx=0 tx=0).stop 2024-04-27T00:01:33.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.463+0000 7fb5f5200640 1 -- 172.21.15.40:0/3813690109 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5f0105b20 msgr2=0x7fb5f0198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:33.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.463+0000 7fb5f5200640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5f0105b20 0x7fb5f0198e60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800e9a0 tx=0x7fb5d800ee60 comp rx=0 tx=0).stop 2024-04-27T00:01:33.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.463+0000 7fb5f5200640 1 -- 172.21.15.40:0/3813690109 shutdown_connections 2024-04-27T00:01:33.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.463+0000 7fb5f5200640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f0108450 0x7fb5f019fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.463+0000 7fb5f5200640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb5f0105b20 0x7fb5f0198e60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.463+0000 7fb5f5200640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb5d40a3c40 0x7fb5d40a6100 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.463+0000 7fb5f5200640 1 --2- 172.21.15.40:0/3813690109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f01031f0 0x7fb5f0198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.463+0000 7fb5f5200640 1 -- 172.21.15.40:0/3813690109 >> 172.21.15.40:0/3813690109 conn(0x7fb5f00fcbc0 msgr2=0x7fb5f0103b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:33.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.463+0000 7fb5f5200640 1 -- 172.21.15.40:0/3813690109 shutdown_connections 2024-04-27T00:01:33.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.463+0000 7fb5f5200640 1 -- 172.21.15.40:0/3813690109 wait complete. 2024-04-27T00:01:33.484 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:53:32.708818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:58:48.506601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:58:49.540141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:58:53.179820+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-27T00:01:33.485 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-27T00:01:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.689+0000 7f6ea1800640 1 -- 172.21.15.40:0/1492966453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e9c105d30 msgr2=0x7f6e9c108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.689+0000 7f6ea1800640 1 --2- 172.21.15.40:0/1492966453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e9c105d30 0x7f6e9c108120 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009900 tx=0x7f6e8802a410 comp rx=0 tx=0).stop 2024-04-27T00:01:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.690+0000 7f6ea1800640 1 -- 172.21.15.40:0/1492966453 shutdown_connections 2024-04-27T00:01:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.690+0000 7f6ea1800640 1 --2- 172.21.15.40:0/1492966453 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6e9c108660 0x7f6e9c10aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.690+0000 7f6ea1800640 1 --2- 172.21.15.40:0/1492966453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e9c105d30 0x7f6e9c108120 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.690+0000 7f6ea1800640 1 --2- 172.21.15.40:0/1492966453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e9c103400 0x7f6e9c1057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.690+0000 7f6ea1800640 1 -- 172.21.15.40:0/1492966453 >> 172.21.15.40:0/1492966453 conn(0x7f6e9c0fcdf0 msgr2=0x7f6e9c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.690+0000 7f6ea1800640 1 -- 172.21.15.40:0/1492966453 shutdown_connections 2024-04-27T00:01:33.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.690+0000 7f6ea1800640 1 -- 172.21.15.40:0/1492966453 wait complete. 2024-04-27T00:01:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.691+0000 7f6ea1800640 1 Processor -- start 2024-04-27T00:01:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.691+0000 7f6ea1800640 1 -- start start 2024-04-27T00:01:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6ea1800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6e9c103400 0x7f6e9c102d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6ea1800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e9c105d30 0x7f6e9c1013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6ea1800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e9c108660 0x7f6e9c1018f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6ea1800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9c10e4f0 con 0x7f6e9c105d30 2024-04-27T00:01:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6ea1800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9c10e370 con 0x7f6e9c103400 2024-04-27T00:01:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6ea1800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9c10e670 con 0x7f6e9c108660 2024-04-27T00:01:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6e9aa00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6e9c103400 0x7f6e9c102d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6e9aa00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6e9c103400 0x7f6e9c102d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49228/0 (socket says 172.21.15.40:49228) 2024-04-27T00:01:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6e9aa00640 1 -- 172.21.15.40:0/503005724 learned_addr learned my addr 172.21.15.40:0/503005724 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6e9a000640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e9c105d30 0x7f6e9c1013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6e9b400640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e9c108660 0x7f6e9c1018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6e9aa00640 1 -- 172.21.15.40:0/503005724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e9c108660 msgr2=0x7f6e9c1018f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6e9aa00640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e9c108660 0x7f6e9c1018f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6e9aa00640 1 -- 172.21.15.40:0/503005724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e9c105d30 msgr2=0x7f6e9c1013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6e9aa00640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e9c105d30 0x7f6e9c1013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.692+0000 7f6e9aa00640 1 -- 172.21.15.40:0/503005724 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e9c102180 con 0x7f6e9c103400 2024-04-27T00:01:33.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.693+0000 7f6e9aa00640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6e9c103400 0x7f6e9c102d70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00e9f0 tx=0x7f6e8c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:33.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.693+0000 7f6e93400640 1 -- 172.21.15.40:0/503005724 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c00cd80 con 0x7f6e9c103400 2024-04-27T00:01:33.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.693+0000 7f6e93400640 1 -- 172.21.15.40:0/503005724 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e8c0108e0 con 0x7f6e9c103400 2024-04-27T00:01:33.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.693+0000 7f6e93400640 1 -- 172.21.15.40:0/503005724 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c018980 con 0x7f6e9c103400 2024-04-27T00:01:33.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.693+0000 7f6ea1800640 1 -- 172.21.15.40:0/503005724 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e9c1a2d20 con 0x7f6e9c103400 2024-04-27T00:01:33.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.694+0000 7f6ea1800640 1 -- 172.21.15.40:0/503005724 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e9c1a3190 con 0x7f6e9c103400 2024-04-27T00:01:33.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.695+0000 7f6ea1800640 1 -- 172.21.15.40:0/503005724 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e9c0fe8e0 con 0x7f6e9c103400 2024-04-27T00:01:33.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.697+0000 7f6e93400640 1 -- 172.21.15.40:0/503005724 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6e8c010430 con 0x7f6e9c103400 2024-04-27T00:01:33.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.698+0000 7f6e93400640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6e800a3c40 0x7f6e800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:33.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.699+0000 7f6e93400640 1 -- 172.21.15.40:0/503005724 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f6e8c0d0c20 con 0x7f6e9c103400 2024-04-27T00:01:33.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.699+0000 7f6e9a000640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6e800a3c40 0x7f6e800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:33.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.700+0000 7f6e9a000640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6e800a3c40 0x7f6e800a6100 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6e88005ab0 tx=0x7f6e88005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:33.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.704+0000 7f6e93400640 1 -- 172.21.15.40:0/503005724 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6e8c09cab0 con 0x7f6e9c103400 2024-04-27T00:01:33.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.918+0000 7f6ea1800640 1 -- 172.21.15.40:0/503005724 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6e9c02dd80 con 0x7f6e9c103400 2024-04-27T00:01:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.919+0000 7f6e93400640 1 -- 172.21.15.40:0/503005724 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f6e8c0a0db0 con 0x7f6e9c103400 2024-04-27T00:01:33.920 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-27T00:01:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.922+0000 7f6ea1800640 1 -- 172.21.15.40:0/503005724 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6e800a3c40 msgr2=0x7f6e800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.922+0000 7f6ea1800640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6e800a3c40 0x7f6e800a6100 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6e88005ab0 tx=0x7f6e88005a20 comp rx=0 tx=0).stop 2024-04-27T00:01:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.922+0000 7f6ea1800640 1 -- 172.21.15.40:0/503005724 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6e9c103400 msgr2=0x7f6e9c102d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.922+0000 7f6ea1800640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6e9c103400 0x7f6e9c102d70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00e9f0 tx=0x7f6e8c00eeb0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.922+0000 7f6ea1800640 1 -- 172.21.15.40:0/503005724 shutdown_connections 2024-04-27T00:01:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.922+0000 7f6ea1800640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6e800a3c40 0x7f6e800a6100 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.922+0000 7f6ea1800640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e9c108660 0x7f6e9c1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.922+0000 7f6ea1800640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e9c105d30 0x7f6e9c1013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.922+0000 7f6ea1800640 1 --2- 172.21.15.40:0/503005724 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6e9c103400 0x7f6e9c102d70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.922+0000 7f6ea1800640 1 -- 172.21.15.40:0/503005724 >> 172.21.15.40:0/503005724 conn(0x7f6e9c0fcdf0 msgr2=0x7f6e9c103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.923+0000 7f6ea1800640 1 -- 172.21.15.40:0/503005724 shutdown_connections 2024-04-27T00:01:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:33.923+0000 7f6ea1800640 1 -- 172.21.15.40:0/503005724 wait complete. 2024-04-27T00:01:33.944 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-27T00:01:34.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:33 smithi103 ceph-mon[32952]: pgmap v406: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:34.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:33 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3813690109' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:34.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:33 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:01:34.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.139+0000 7f6876000640 1 -- 172.21.15.40:0/641254766 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6870076910 msgr2=0x7f6870074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:34.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.139+0000 7f6876000640 1 --2- 172.21.15.40:0/641254766 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6870076910 0x7f6870074d60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6864009900 tx=0x7f686402a480 comp rx=0 tx=0).stop 2024-04-27T00:01:34.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.139+0000 7f6876000640 1 -- 172.21.15.40:0/641254766 shutdown_connections 2024-04-27T00:01:34.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.139+0000 7f6876000640 1 --2- 172.21.15.40:0/641254766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f687010a6f0 0x7f687010cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.139+0000 7f6876000640 1 --2- 172.21.15.40:0/641254766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6870075330 0x7f6870075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.139+0000 7f6876000640 1 --2- 172.21.15.40:0/641254766 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6870076910 0x7f6870074d60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.139+0000 7f6876000640 1 -- 172.21.15.40:0/641254766 >> 172.21.15.40:0/641254766 conn(0x7f68700fddd0 msgr2=0x7f6870100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:34.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.140+0000 7f6876000640 1 -- 172.21.15.40:0/641254766 shutdown_connections 2024-04-27T00:01:34.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.140+0000 7f6876000640 1 -- 172.21.15.40:0/641254766 wait complete. 2024-04-27T00:01:34.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.141+0000 7f6876000640 1 Processor -- start 2024-04-27T00:01:34.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.141+0000 7f6876000640 1 -- start start 2024-04-27T00:01:34.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.141+0000 7f6876000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6870075330 0x7f687019aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:34.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.141+0000 7f6876000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f687010a6f0 0x7f687019b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:34.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.141+0000 7f6876000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68701a0400 0x7f68701a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:34.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.141+0000 7f6876000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f68701105b0 con 0x7f68701a0400 2024-04-27T00:01:34.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.141+0000 7f6876000640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f6870110430 con 0x7f6870075330 2024-04-27T00:01:34.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.141+0000 7f6876000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6870110730 con 0x7f687010a6f0 2024-04-27T00:01:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.142+0000 7f6874c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6870075330 0x7f687019aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.142+0000 7f6874c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6870075330 0x7f687019aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49246/0 (socket says 172.21.15.40:49246) 2024-04-27T00:01:34.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.142+0000 7f6874c00640 1 -- 172.21.15.40:0/2067175480 learned_addr learned my addr 172.21.15.40:0/2067175480 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.142+0000 7f686fe00640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f687010a6f0 0x7f687019b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.142+0000 7f6875600640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68701a0400 0x7f68701a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.142+0000 7f6874c00640 1 -- 172.21.15.40:0/2067175480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f687010a6f0 msgr2=0x7f687019b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.142+0000 7f6874c00640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f687010a6f0 0x7f687019b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.142+0000 7f6874c00640 1 -- 172.21.15.40:0/2067175480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68701a0400 msgr2=0x7f68701a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.142+0000 7f6874c00640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68701a0400 0x7f68701a27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.142+0000 7f6874c00640 1 -- 172.21.15.40:0/2067175480 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68701a2e60 con 0x7f6870075330 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.142+0000 7f6875600640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68701a0400 0x7f68701a27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.142+0000 7f6874c00640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6870075330 0x7f687019aeb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6864005ab0 tx=0x7f6864003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.143+0000 7f686d600640 1 -- 172.21.15.40:0/2067175480 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686403f020 con 0x7f6870075330 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.143+0000 7f686d600640 1 -- 172.21.15.40:0/2067175480 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6864031c50 con 0x7f6870075330 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.143+0000 7f686d600640 1 -- 172.21.15.40:0/2067175480 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864035070 con 0x7f6870075330 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.143+0000 7f6876000640 1 -- 172.21.15.40:0/2067175480 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68701a30f0 con 0x7f6870075330 2024-04-27T00:01:34.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.143+0000 7f6876000640 1 -- 172.21.15.40:0/2067175480 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68701a3590 con 0x7f6870075330 2024-04-27T00:01:34.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.144+0000 7f6876000640 1 -- 172.21.15.40:0/2067175480 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68701094b0 con 0x7f6870075330 2024-04-27T00:01:34.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.146+0000 7f686d600640 1 -- 172.21.15.40:0/2067175480 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6864042020 con 0x7f6870075330 2024-04-27T00:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.147+0000 7f686d600640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f68580a3c40 0x7f68580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.148+0000 7f686d600640 1 -- 172.21.15.40:0/2067175480 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f68640f09b0 con 0x7f6870075330 2024-04-27T00:01:34.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.148+0000 7f686fe00640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f68580a3c40 0x7f68580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:34.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.149+0000 7f686fe00640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f68580a3c40 0x7f68580a6100 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f687019c370 tx=0x7f685c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.150+0000 7f686d600640 1 -- 172.21.15.40:0/2067175480 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f68640bce90 con 0x7f6870075330 2024-04-27T00:01:34.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:33 smithi040 ceph-mon[28095]: pgmap v406: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:34.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:33 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3813690109' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:34.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:33 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:01:34.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:33 smithi080 ceph-mon[32362]: pgmap v406: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:34.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:33 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3813690109' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:34.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:33 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:01:34.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.387+0000 7f6876000640 1 -- 172.21.15.40:0/2067175480 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f687002dc60 con 0x7f6870075330 2024-04-27T00:01:34.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.387+0000 7f686d600640 1 -- 172.21.15.40:0/2067175480 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f68640bc790 con 0x7f6870075330 2024-04-27T00:01:34.389 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-27T00:01:34.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.390+0000 7f6876000640 1 -- 172.21.15.40:0/2067175480 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f68580a3c40 msgr2=0x7f68580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:34.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.390+0000 7f6876000640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f68580a3c40 0x7f68580a6100 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f687019c370 tx=0x7f685c00b410 comp rx=0 tx=0).stop 2024-04-27T00:01:34.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.390+0000 7f6876000640 1 -- 172.21.15.40:0/2067175480 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6870075330 msgr2=0x7f687019aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:34.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.390+0000 7f6876000640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6870075330 0x7f687019aeb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6864005ab0 tx=0x7f6864003c80 comp rx=0 tx=0).stop 2024-04-27T00:01:34.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.390+0000 7f6876000640 1 -- 172.21.15.40:0/2067175480 shutdown_connections 2024-04-27T00:01:34.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.390+0000 7f6876000640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68701a0400 0x7f68701a27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.390+0000 7f6876000640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f68580a3c40 0x7f68580a6100 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.390+0000 7f6876000640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f687010a6f0 0x7f687019b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.390+0000 7f6876000640 1 --2- 172.21.15.40:0/2067175480 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6870075330 0x7f687019aeb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.390+0000 7f6876000640 1 -- 172.21.15.40:0/2067175480 >> 172.21.15.40:0/2067175480 conn(0x7f68700fddd0 msgr2=0x7f6870106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:34.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.390+0000 7f6876000640 1 -- 172.21.15.40:0/2067175480 shutdown_connections 2024-04-27T00:01:34.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.390+0000 7f6876000640 1 -- 172.21.15.40:0/2067175480 wait complete. 2024-04-27T00:01:34.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-27T00:01:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.605+0000 7fae69e00640 1 -- 172.21.15.40:0/4005516890 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fae641033f0 msgr2=0x7fae641057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.605+0000 7fae69e00640 1 --2- 172.21.15.40:0/4005516890 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fae641033f0 0x7fae641057e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fae54009900 tx=0x7fae5402a410 comp rx=0 tx=0).stop 2024-04-27T00:01:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.605+0000 7fae69e00640 1 -- 172.21.15.40:0/4005516890 shutdown_connections 2024-04-27T00:01:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.605+0000 7fae69e00640 1 --2- 172.21.15.40:0/4005516890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae64108650 0x7fae6410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.605+0000 7fae69e00640 1 --2- 172.21.15.40:0/4005516890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae64105d20 0x7fae64108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.605+0000 7fae69e00640 1 --2- 172.21.15.40:0/4005516890 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fae641033f0 0x7fae641057e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.605+0000 7fae69e00640 1 -- 172.21.15.40:0/4005516890 >> 172.21.15.40:0/4005516890 conn(0x7fae640fcde0 msgr2=0x7fae640ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.606+0000 7fae69e00640 1 -- 172.21.15.40:0/4005516890 shutdown_connections 2024-04-27T00:01:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.606+0000 7fae69e00640 1 -- 172.21.15.40:0/4005516890 wait complete. 2024-04-27T00:01:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.606+0000 7fae69e00640 1 Processor -- start 2024-04-27T00:01:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae69e00640 1 -- start start 2024-04-27T00:01:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae69e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae641033f0 0x7fae641969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae69e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fae64105d20 0x7fae64196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae69e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae64108650 0x7fae6419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae69e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fae6410c310 con 0x7fae641033f0 2024-04-27T00:01:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae69e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fae6410c190 con 0x7fae64105d20 2024-04-27T00:01:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae69e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fae6410c490 con 0x7fae64108650 2024-04-27T00:01:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae62a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae641033f0 0x7fae641969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae62000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fae64105d20 0x7fae64196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae62000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fae64105d20 0x7fae64196f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49264/0 (socket says 172.21.15.40:49264) 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae62000640 1 -- 172.21.15.40:0/1915593286 learned_addr learned my addr 172.21.15.40:0/1915593286 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae62a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae641033f0 0x7fae641969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51948/0 (socket says 172.21.15.40:51948) 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.607+0000 7fae63400640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae64108650 0x7fae6419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.608+0000 7fae62000640 1 -- 172.21.15.40:0/1915593286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae64108650 msgr2=0x7fae6419df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.608+0000 7fae62000640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae64108650 0x7fae6419df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.608+0000 7fae62000640 1 -- 172.21.15.40:0/1915593286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae641033f0 msgr2=0x7fae641969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.608+0000 7fae62000640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae641033f0 0x7fae641969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.608+0000 7fae62000640 1 -- 172.21.15.40:0/1915593286 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6419e5f0 con 0x7fae64105d20 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.608+0000 7fae62000640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fae64105d20 0x7fae64196f10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fae50003cb0 tx=0x7fae50009740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.608+0000 7fae68a00640 1 -- 172.21.15.40:0/1915593286 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae50005020 con 0x7fae64105d20 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.608+0000 7fae68a00640 1 -- 172.21.15.40:0/1915593286 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae50005670 con 0x7fae64105d20 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.608+0000 7fae69e00640 1 -- 172.21.15.40:0/1915593286 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae6419e8e0 con 0x7fae64105d20 2024-04-27T00:01:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.608+0000 7fae68a00640 1 -- 172.21.15.40:0/1915593286 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5001c680 con 0x7fae64105d20 2024-04-27T00:01:34.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.609+0000 7fae69e00640 1 -- 172.21.15.40:0/1915593286 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae6419ee30 con 0x7fae64105d20 2024-04-27T00:01:34.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.610+0000 7fae69e00640 1 -- 172.21.15.40:0/1915593286 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae640fe8d0 con 0x7fae64105d20 2024-04-27T00:01:34.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.612+0000 7fae68a00640 1 -- 172.21.15.40:0/1915593286 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fae50023020 con 0x7fae64105d20 2024-04-27T00:01:34.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.613+0000 7fae68a00640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fae480a3cc0 0x7fae480a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:34.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.613+0000 7fae68a00640 1 -- 172.21.15.40:0/1915593286 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fae500d86e0 con 0x7fae64105d20 2024-04-27T00:01:34.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.613+0000 7fae62a00640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fae480a3cc0 0x7fae480a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:34.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.614+0000 7fae62a00640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fae480a3cc0 0x7fae480a6180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fae54031500 tx=0x7fae54005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:34.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.618+0000 7fae68a00640 1 -- 172.21.15.40:0/1915593286 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fae500046c0 con 0x7fae64105d20 2024-04-27T00:01:34.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.822+0000 7fae69e00640 1 -- 172.21.15.40:0/1915593286 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fae6402dce0 con 0x7fae64105d20 2024-04-27T00:01:34.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.823+0000 7fae68a00640 1 -- 172.21.15.40:0/1915593286 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7fae500a4bc0 con 0x7fae64105d20 2024-04-27T00:01:34.825 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-27T00:01:34.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.825+0000 7fae69e00640 1 -- 172.21.15.40:0/1915593286 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fae480a3cc0 msgr2=0x7fae480a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:34.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.825+0000 7fae69e00640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fae480a3cc0 0x7fae480a6180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fae54031500 tx=0x7fae54005a20 comp rx=0 tx=0).stop 2024-04-27T00:01:34.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.826+0000 7fae69e00640 1 -- 172.21.15.40:0/1915593286 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fae64105d20 msgr2=0x7fae64196f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:34.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.826+0000 7fae69e00640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fae64105d20 0x7fae64196f10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fae50003cb0 tx=0x7fae50009740 comp rx=0 tx=0).stop 2024-04-27T00:01:34.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.826+0000 7fae69e00640 1 -- 172.21.15.40:0/1915593286 shutdown_connections 2024-04-27T00:01:34.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.826+0000 7fae69e00640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fae480a3cc0 0x7fae480a6180 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.826+0000 7fae69e00640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae64108650 0x7fae6419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.826+0000 7fae69e00640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fae64105d20 0x7fae64196f10 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.826+0000 7fae69e00640 1 --2- 172.21.15.40:0/1915593286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae641033f0 0x7fae641969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:34.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.826+0000 7fae69e00640 1 -- 172.21.15.40:0/1915593286 >> 172.21.15.40:0/1915593286 conn(0x7fae640fcde0 msgr2=0x7fae64103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:34.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.826+0000 7fae69e00640 1 -- 172.21.15.40:0/1915593286 shutdown_connections 2024-04-27T00:01:34.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:34.826+0000 7fae69e00640 1 -- 172.21.15.40:0/1915593286 wait complete. 2024-04-27T00:01:34.848 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-27T00:01:35.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.040+0000 7f50cec00640 1 -- 172.21.15.40:0/16424456 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50c81028c0 msgr2=0x7f50c8102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.040+0000 7f50cec00640 1 --2- 172.21.15.40:0/16424456 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50c81028c0 0x7f50c8102ce0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f50bc009900 tx=0x7f50bc02a410 comp rx=0 tx=0).stop 2024-04-27T00:01:35.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.040+0000 7f50cec00640 1 -- 172.21.15.40:0/16424456 shutdown_connections 2024-04-27T00:01:35.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.040+0000 7f50cec00640 1 --2- 172.21.15.40:0/16424456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50c8104470 0x7f50c810ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.040+0000 7f50cec00640 1 --2- 172.21.15.40:0/16424456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50c8103ab0 0x7f50c8103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.040+0000 7f50cec00640 1 --2- 172.21.15.40:0/16424456 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50c81028c0 0x7f50c8102ce0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.040+0000 7f50cec00640 1 -- 172.21.15.40:0/16424456 >> 172.21.15.40:0/16424456 conn(0x7f50c80fde30 msgr2=0x7f50c8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:35.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.040+0000 7f50cec00640 1 -- 172.21.15.40:0/16424456 shutdown_connections 2024-04-27T00:01:35.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.040+0000 7f50cec00640 1 -- 172.21.15.40:0/16424456 wait complete. 2024-04-27T00:01:35.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.041+0000 7f50cec00640 1 Processor -- start 2024-04-27T00:01:35.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.041+0000 7f50cec00640 1 -- start start 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.042+0000 7f50cec00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50c81028c0 0x7f50c8076690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.042+0000 7f50cec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50c8103ab0 0x7f50c8074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.042+0000 7f50cec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50c8104470 0x7f50c8075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.042+0000 7f50cec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f50c810e350 con 0x7f50c8103ab0 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.042+0000 7f50cec00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f50c810e1d0 con 0x7f50c81028c0 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.042+0000 7f50cec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f50c810e4d0 con 0x7f50c8104470 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.042+0000 7f50cce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50c8103ab0 0x7f50c8074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.042+0000 7f50cd800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50c81028c0 0x7f50c8076690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.042+0000 7f50ce200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50c8104470 0x7f50c8075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.042+0000 7f50cd800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50c81028c0 0x7f50c8076690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49282/0 (socket says 172.21.15.40:49282) 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.042+0000 7f50cd800640 1 -- 172.21.15.40:0/3317623786 learned_addr learned my addr 172.21.15.40:0/3317623786 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.043+0000 7f50cd800640 1 -- 172.21.15.40:0/3317623786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50c8104470 msgr2=0x7f50c8075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.043+0000 7f50cd800640 1 --2- 172.21.15.40:0/3317623786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50c8104470 0x7f50c8075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.043+0000 7f50cd800640 1 -- 172.21.15.40:0/3317623786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50c8103ab0 msgr2=0x7f50c8074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.043+0000 7f50cd800640 1 --2- 172.21.15.40:0/3317623786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50c8103ab0 0x7f50c8074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.043+0000 7f50cd800640 1 -- 172.21.15.40:0/3317623786 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c8075b20 con 0x7f50c81028c0 2024-04-27T00:01:35.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.043+0000 7f50cd800640 1 --2- 172.21.15.40:0/3317623786 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50c81028c0 0x7f50c8076690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f50bc006010 tx=0x7f50bc0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:35.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.043+0000 7f50c6000640 1 -- 172.21.15.40:0/3317623786 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc03f030 con 0x7f50c81028c0 2024-04-27T00:01:35.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.043+0000 7f50c6000640 1 -- 172.21.15.40:0/3317623786 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50bc031910 con 0x7f50c81028c0 2024-04-27T00:01:35.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.043+0000 7f50cec00640 1 -- 172.21.15.40:0/3317623786 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50c81a2ba0 con 0x7f50c81028c0 2024-04-27T00:01:35.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.043+0000 7f50c6000640 1 -- 172.21.15.40:0/3317623786 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc035070 con 0x7f50c81028c0 2024-04-27T00:01:35.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.043+0000 7f50cec00640 1 -- 172.21.15.40:0/3317623786 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50c81a3060 con 0x7f50c81028c0 2024-04-27T00:01:35.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.045+0000 7f50cec00640 1 -- 172.21.15.40:0/3317623786 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50c8109d00 con 0x7f50c81028c0 2024-04-27T00:01:35.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.047+0000 7f50c6000640 1 -- 172.21.15.40:0/3317623786 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f50bc042030 con 0x7f50c81028c0 2024-04-27T00:01:35.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.048+0000 7f50c6000640 1 --2- 172.21.15.40:0/3317623786 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f50b00a3c40 0x7f50b00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:35.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.048+0000 7f50c6000640 1 -- 172.21.15.40:0/3317623786 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f50bc0f0d20 con 0x7f50c81028c0 2024-04-27T00:01:35.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.048+0000 7f50cce00640 1 --2- 172.21.15.40:0/3317623786 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f50b00a3c40 0x7f50b00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:35.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.049+0000 7f50cce00640 1 --2- 172.21.15.40:0/3317623786 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f50b00a3c40 0x7f50b00a6100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f50c8076400 tx=0x7f50b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:35.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.052+0000 7f50c6000640 1 -- 172.21.15.40:0/3317623786 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f50bc0bd200 con 0x7f50c81028c0 2024-04-27T00:01:35.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:34 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/503005724' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:01:35.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:34 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2067175480' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:35.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:34 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:34 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1915593286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:35.211 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:34 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/503005724' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:01:35.211 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.211 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.211 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.211 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.211 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:34 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2067175480' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:35.212 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.212 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:34 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.212 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:34 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1915593286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:35.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:34 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/503005724' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:01:35.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:34 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2067175480' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:35.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:34 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:35.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:34 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1915593286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:35.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.273+0000 7f50cec00640 1 -- 172.21.15.40:0/3317623786 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f50c802dc80 con 0x7f50c81028c0 2024-04-27T00:01:35.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.274+0000 7f50c6000640 1 -- 172.21.15.40:0/3317623786 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f50bc0bcb00 con 0x7f50c81028c0 2024-04-27T00:01:35.275 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-27T00:01:35.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.276+0000 7f50cec00640 1 -- 172.21.15.40:0/3317623786 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f50b00a3c40 msgr2=0x7f50b00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.276+0000 7f50cec00640 1 --2- 172.21.15.40:0/3317623786 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f50b00a3c40 0x7f50b00a6100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f50c8076400 tx=0x7f50b4005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:35.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.276+0000 7f50cec00640 1 -- 172.21.15.40:0/3317623786 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50c81028c0 msgr2=0x7f50c8076690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.276+0000 7f50cec00640 1 --2- 172.21.15.40:0/3317623786 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50c81028c0 0x7f50c8076690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f50bc006010 tx=0x7f50bc0042f0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.277+0000 7f50cec00640 1 -- 172.21.15.40:0/3317623786 shutdown_connections 2024-04-27T00:01:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.277+0000 7f50cec00640 1 --2- 172.21.15.40:0/3317623786 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f50b00a3c40 0x7f50b00a6100 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.277+0000 7f50cec00640 1 --2- 172.21.15.40:0/3317623786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50c8104470 0x7f50c8075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.277+0000 7f50cec00640 1 --2- 172.21.15.40:0/3317623786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50c8103ab0 0x7f50c8074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.277+0000 7f50cec00640 1 --2- 172.21.15.40:0/3317623786 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f50c81028c0 0x7f50c8076690 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.277+0000 7f50cec00640 1 -- 172.21.15.40:0/3317623786 >> 172.21.15.40:0/3317623786 conn(0x7f50c80fde30 msgr2=0x7f50c8106ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.277+0000 7f50cec00640 1 -- 172.21.15.40:0/3317623786 shutdown_connections 2024-04-27T00:01:35.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.277+0000 7f50cec00640 1 -- 172.21.15.40:0/3317623786 wait complete. 2024-04-27T00:01:35.296 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-27T00:01:35.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.480+0000 7f2020400640 1 -- 172.21.15.40:0/1242282077 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2018078ed0 msgr2=0x7f2018079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.480+0000 7f2020400640 1 --2- 172.21.15.40:0/1242282077 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2018078ed0 0x7f2018079770 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2010009be0 tx=0x7f201002a6d0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.480+0000 7f2020400640 1 -- 172.21.15.40:0/1242282077 shutdown_connections 2024-04-27T00:01:35.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.480+0000 7f2020400640 1 --2- 172.21.15.40:0/1242282077 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2018078ed0 0x7f2018079770 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.480+0000 7f2020400640 1 --2- 172.21.15.40:0/1242282077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2018078530 0x7f2018078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.480+0000 7f2020400640 1 --2- 172.21.15.40:0/1242282077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20180772f0 0x7f2018077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.480+0000 7f2020400640 1 -- 172.21.15.40:0/1242282077 >> 172.21.15.40:0/1242282077 conn(0x7f2018100120 msgr2=0x7f2018102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:35.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.480+0000 7f2020400640 1 -- 172.21.15.40:0/1242282077 shutdown_connections 2024-04-27T00:01:35.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.480+0000 7f2020400640 1 -- 172.21.15.40:0/1242282077 wait complete. 2024-04-27T00:01:35.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.481+0000 7f2020400640 1 Processor -- start 2024-04-27T00:01:35.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.481+0000 7f2020400640 1 -- start start 2024-04-27T00:01:35.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.482+0000 7f2020400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20180772f0 0x7f201819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:35.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.482+0000 7f2020400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2018078530 0x7f201819f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:35.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.482+0000 7f2020400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2018078ed0 0x7f20181a66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:35.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.482+0000 7f2020400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2018114ad0 con 0x7f2018078ed0 2024-04-27T00:01:35.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.482+0000 7f2020400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f2018114950 con 0x7f2018078530 2024-04-27T00:01:35.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.482+0000 7f2020400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2018114c50 con 0x7f20180772f0 2024-04-27T00:01:35.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.482+0000 7f201da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20180772f0 0x7f201819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:35.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.482+0000 7f201d000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2018078530 0x7f201819f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:35.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.482+0000 7f201d000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2018078530 0x7f201819f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49300/0 (socket says 172.21.15.40:49300) 2024-04-27T00:01:35.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.482+0000 7f201d000640 1 -- 172.21.15.40:0/1154249776 learned_addr learned my addr 172.21.15.40:0/1154249776 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:35.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.482+0000 7f201e400640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2018078ed0 0x7f20181a66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:35.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.483+0000 7f201da00640 1 -- 172.21.15.40:0/1154249776 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2018078530 msgr2=0x7f201819f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.483+0000 7f201da00640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2018078530 0x7f201819f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.483+0000 7f201da00640 1 -- 172.21.15.40:0/1154249776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2018078ed0 msgr2=0x7f20181a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.483+0000 7f201da00640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2018078ed0 0x7f20181a66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.483+0000 7f201da00640 1 -- 172.21.15.40:0/1154249776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20181a6d40 con 0x7f20180772f0 2024-04-27T00:01:35.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.483+0000 7f201d000640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2018078530 0x7f201819f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:35.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.483+0000 7f201da00640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20180772f0 0x7f201819f120 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f200c00e9f0 tx=0x7f200c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:35.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.485+0000 7f2016000640 1 -- 172.21.15.40:0/1154249776 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c00ccc0 con 0x7f20180772f0 2024-04-27T00:01:35.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.485+0000 7f2016000640 1 -- 172.21.15.40:0/1154249776 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f200c005020 con 0x7f20180772f0 2024-04-27T00:01:35.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.485+0000 7f2016000640 1 -- 172.21.15.40:0/1154249776 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c004a00 con 0x7f20180772f0 2024-04-27T00:01:35.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.485+0000 7f2020400640 1 -- 172.21.15.40:0/1154249776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20181a6fd0 con 0x7f20180772f0 2024-04-27T00:01:35.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.485+0000 7f2020400640 1 -- 172.21.15.40:0/1154249776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20181a7440 con 0x7f20180772f0 2024-04-27T00:01:35.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.487+0000 7f2020400640 1 -- 172.21.15.40:0/1154249776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2018111690 con 0x7f20180772f0 2024-04-27T00:01:35.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.488+0000 7f2016000640 1 -- 172.21.15.40:0/1154249776 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f200c004ba0 con 0x7f20180772f0 2024-04-27T00:01:35.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.489+0000 7f2016000640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f20000a3cc0 0x7f20000a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:35.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.490+0000 7f2016000640 1 -- 172.21.15.40:0/1154249776 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f200c0d07f0 con 0x7f20180772f0 2024-04-27T00:01:35.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.490+0000 7f201d000640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f20000a3cc0 0x7f20000a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:35.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.490+0000 7f201d000640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f20000a3cc0 0x7f20000a6180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f20080096c0 tx=0x7f2008009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:35.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.494+0000 7f2016000640 1 -- 172.21.15.40:0/1154249776 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f200c09ccd0 con 0x7f20180772f0 2024-04-27T00:01:35.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.736+0000 7f2020400640 1 -- 172.21.15.40:0/1154249776 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f201802dcc0 con 0x7f20180772f0 2024-04-27T00:01:35.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.737+0000 7f2016000640 1 -- 172.21.15.40:0/1154249776 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11872 (secure 0 0 0) 0x7f200c09c5d0 con 0x7f20180772f0 2024-04-27T00:01:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:35.738 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/2265538751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":2265538751},{"type":"v1","addr":"172.21.15.40:6841","nonce":2265538751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:58:50.616677+0000","modified":"2024-04-26T23:59:25.374905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24523":{"gid":24523,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.80:6839/3536638444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6838","nonce":3536638444},{"type":"v1","addr":"172.21.15.80:6839","nonce":3536638444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24559,24560,24590]},"id":1}]} 2024-04-27T00:01:35.739 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 23 2024-04-27T00:01:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.739+0000 7f2020400640 1 -- 172.21.15.40:0/1154249776 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f20000a3cc0 msgr2=0x7f20000a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.739+0000 7f2020400640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f20000a3cc0 0x7f20000a6180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f20080096c0 tx=0x7f2008009260 comp rx=0 tx=0).stop 2024-04-27T00:01:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.739+0000 7f2020400640 1 -- 172.21.15.40:0/1154249776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20180772f0 msgr2=0x7f201819f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.739+0000 7f2020400640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20180772f0 0x7f201819f120 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f200c00e9f0 tx=0x7f200c00eeb0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.739+0000 7f2020400640 1 -- 172.21.15.40:0/1154249776 shutdown_connections 2024-04-27T00:01:35.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.739+0000 7f2020400640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2018078ed0 0x7f20181a66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.739+0000 7f2020400640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2018078530 0x7f201819f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.739+0000 7f2020400640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f20000a3cc0 0x7f20000a6180 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.739+0000 7f2020400640 1 --2- 172.21.15.40:0/1154249776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20180772f0 0x7f201819f120 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.739+0000 7f2020400640 1 -- 172.21.15.40:0/1154249776 >> 172.21.15.40:0/1154249776 conn(0x7f2018100120 msgr2=0x7f20181017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:35.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.740+0000 7f2020400640 1 -- 172.21.15.40:0/1154249776 shutdown_connections 2024-04-27T00:01:35.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.740+0000 7f2020400640 1 -- 172.21.15.40:0/1154249776 wait complete. 2024-04-27T00:01:35.759 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T00:01:35.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:35 smithi040 ceph-mon[28095]: pgmap v407: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:35.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:35 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3317623786' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:35.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:35 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1154249776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:35.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.970+0000 7f600dc00640 1 -- 172.21.15.40:0/3534280966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6008105be0 msgr2=0x7f6008107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.970+0000 7f600dc00640 1 --2- 172.21.15.40:0/3534280966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6008105be0 0x7f6008107fd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4009900 tx=0x7f5ff402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.970+0000 7f600dc00640 1 -- 172.21.15.40:0/3534280966 shutdown_connections 2024-04-27T00:01:35.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.970+0000 7f600dc00640 1 --2- 172.21.15.40:0/3534280966 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6008108510 0x7f600810a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.970+0000 7f600dc00640 1 --2- 172.21.15.40:0/3534280966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6008105be0 0x7f6008107fd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.970+0000 7f600dc00640 1 --2- 172.21.15.40:0/3534280966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60081032b0 0x7f60081056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.970+0000 7f600dc00640 1 -- 172.21.15.40:0/3534280966 >> 172.21.15.40:0/3534280966 conn(0x7f60080fcbc0 msgr2=0x7f60080ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:35.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.970+0000 7f600dc00640 1 -- 172.21.15.40:0/3534280966 shutdown_connections 2024-04-27T00:01:35.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.970+0000 7f600dc00640 1 -- 172.21.15.40:0/3534280966 wait complete. 2024-04-27T00:01:35.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.971+0000 7f600dc00640 1 Processor -- start 2024-04-27T00:01:35.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.971+0000 7f600dc00640 1 -- start start 2024-04-27T00:01:35.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f600dc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60081032b0 0x7f600819ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:35.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f600dc00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6008105be0 0x7f600819b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:35.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f600dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6008108510 0x7f60081a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:35.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f600dc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f600810e2d0 con 0x7f6008108510 2024-04-27T00:01:35.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f600dc00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f600810e150 con 0x7f6008105be0 2024-04-27T00:01:35.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f600dc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f600810e450 con 0x7f60081032b0 2024-04-27T00:01:35.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f6007400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6008105be0 0x7f600819b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:35.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f6007400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6008105be0 0x7f600819b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49320/0 (socket says 172.21.15.40:49320) 2024-04-27T00:01:35.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f6007400640 1 -- 172.21.15.40:0/1568832877 learned_addr learned my addr 172.21.15.40:0/1568832877 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:35.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f600d200640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6008108510 0x7f60081a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:35.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f6007e00640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60081032b0 0x7f600819ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f6007400640 1 -- 172.21.15.40:0/1568832877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60081032b0 msgr2=0x7f600819ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.972+0000 7f6007400640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60081032b0 0x7f600819ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.973+0000 7f6007400640 1 -- 172.21.15.40:0/1568832877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6008108510 msgr2=0x7f60081a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.973+0000 7f6007400640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6008108510 0x7f60081a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.973+0000 7f6007400640 1 -- 172.21.15.40:0/1568832877 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60081a27a0 con 0x7f6008105be0 2024-04-27T00:01:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.973+0000 7f6007e00640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60081032b0 0x7f600819ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:35.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.973+0000 7f6007400640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6008105be0 0x7f600819b0c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4031500 tx=0x7f5ff4003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:35.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.973+0000 7f6004c00640 1 -- 172.21.15.40:0/1568832877 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff403f070 con 0x7f6008105be0 2024-04-27T00:01:35.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.973+0000 7f6004c00640 1 -- 172.21.15.40:0/1568832877 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ff4031c50 con 0x7f6008105be0 2024-04-27T00:01:35.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.973+0000 7f6004c00640 1 -- 172.21.15.40:0/1568832877 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4035030 con 0x7f6008105be0 2024-04-27T00:01:35.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.973+0000 7f600dc00640 1 -- 172.21.15.40:0/1568832877 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60081a2a30 con 0x7f6008105be0 2024-04-27T00:01:35.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.973+0000 7f600dc00640 1 -- 172.21.15.40:0/1568832877 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60081a2ed0 con 0x7f6008105be0 2024-04-27T00:01:35.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.975+0000 7f600dc00640 1 -- 172.21.15.40:0/1568832877 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f600810a000 con 0x7f6008105be0 2024-04-27T00:01:35.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.977+0000 7f6004c00640 1 -- 172.21.15.40:0/1568832877 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5ff4042070 con 0x7f6008105be0 2024-04-27T00:01:35.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.978+0000 7f6004c00640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5fec0a3cc0 0x7f5fec0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:35.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.978+0000 7f6004c00640 1 -- 172.21.15.40:0/1568832877 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f5ff40f0f40 con 0x7f6008105be0 2024-04-27T00:01:35.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.978+0000 7f6007e00640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5fec0a3cc0 0x7f5fec0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:35.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.979+0000 7f6007e00640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5fec0a3cc0 0x7f5fec0a6180 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5ff80095f0 tx=0x7f5ff8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:35.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:35.982+0000 7f6004c00640 1 -- 172.21.15.40:0/1568832877 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5ff40bce30 con 0x7f6008105be0 2024-04-27T00:01:36.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.198+0000 7f600dc00640 1 -- 172.21.15.40:0/1568832877 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f600802dcf0 con 0x7f6008105be0 2024-04-27T00:01:36.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.200+0000 7f6004c00640 1 -- 172.21.15.40:0/1568832877 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24531 (secure 0 0 0) 0x7f5ff40c1130 con 0x7f6008105be0 2024-04-27T00:01:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:36.203 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-27T00:00:46.365303+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9},{"from":5,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T00:01:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.204+0000 7f600dc00640 1 -- 172.21.15.40:0/1568832877 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5fec0a3cc0 msgr2=0x7f5fec0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:36.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.204+0000 7f600dc00640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5fec0a3cc0 0x7f5fec0a6180 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5ff80095f0 tx=0x7f5ff8005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.204+0000 7f600dc00640 1 -- 172.21.15.40:0/1568832877 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6008105be0 msgr2=0x7f600819b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.204+0000 7f600dc00640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6008105be0 0x7f600819b0c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4031500 tx=0x7f5ff4003c10 comp rx=0 tx=0).stop 2024-04-27T00:01:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.204+0000 7f600dc00640 1 -- 172.21.15.40:0/1568832877 shutdown_connections 2024-04-27T00:01:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.204+0000 7f600dc00640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6008108510 0x7f60081a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.204+0000 7f600dc00640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6008105be0 0x7f600819b0c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.204+0000 7f600dc00640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5fec0a3cc0 0x7f5fec0a6180 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.204+0000 7f600dc00640 1 --2- 172.21.15.40:0/1568832877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60081032b0 0x7f600819ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.204+0000 7f600dc00640 1 -- 172.21.15.40:0/1568832877 >> 172.21.15.40:0/1568832877 conn(0x7f60080fcbc0 msgr2=0x7f6008102960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.204+0000 7f600dc00640 1 -- 172.21.15.40:0/1568832877 shutdown_connections 2024-04-27T00:01:36.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.204+0000 7f600dc00640 1 -- 172.21.15.40:0/1568832877 wait complete. 2024-04-27T00:01:36.228 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:53:32.708818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:58:48.506601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:58:49.540141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:58:53.179820+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-27T00:01:36.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-27T00:01:36.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:35 smithi080 ceph-mon[32362]: pgmap v407: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:36.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:35 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3317623786' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:36.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:35 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1154249776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:36.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:35 smithi103 ceph-mon[32952]: pgmap v407: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:36.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:35 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3317623786' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:36.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:35 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1154249776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:36.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.424+0000 7f7249800640 1 -- 172.21.15.40:0/1887081243 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7244106620 msgr2=0x7f724410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:36.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.424+0000 7f7249800640 1 --2- 172.21.15.40:0/1887081243 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7244106620 0x7f724410ceb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f723c009ca0 tx=0x7f723c02a8c0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.425+0000 7f7249800640 1 -- 172.21.15.40:0/1887081243 shutdown_connections 2024-04-27T00:01:36.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.425+0000 7f7249800640 1 --2- 172.21.15.40:0/1887081243 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7244106620 0x7f724410ceb0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.425+0000 7f7249800640 1 --2- 172.21.15.40:0/1887081243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7244105c60 0x7f72441060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.425+0000 7f7249800640 1 --2- 172.21.15.40:0/1887081243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7244104a70 0x7f7244104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.425+0000 7f7249800640 1 -- 172.21.15.40:0/1887081243 >> 172.21.15.40:0/1887081243 conn(0x7f72440fffe0 msgr2=0x7f7244102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:36.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.425+0000 7f7249800640 1 -- 172.21.15.40:0/1887081243 shutdown_connections 2024-04-27T00:01:36.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.425+0000 7f7249800640 1 -- 172.21.15.40:0/1887081243 wait complete. 2024-04-27T00:01:36.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.426+0000 7f7249800640 1 Processor -- start 2024-04-27T00:01:36.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.426+0000 7f7249800640 1 -- start start 2024-04-27T00:01:36.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.426+0000 7f7249800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7244104a70 0x7f724419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:36.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.426+0000 7f7249800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7244105c60 0x7f724419b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:36.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.426+0000 7f7249800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7244106620 0x7f72441a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:36.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.426+0000 7f7249800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72441105f0 con 0x7f7244105c60 2024-04-27T00:01:36.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.426+0000 7f7249800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f7244110470 con 0x7f7244104a70 2024-04-27T00:01:36.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.426+0000 7f7249800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7244110770 con 0x7f7244106620 2024-04-27T00:01:36.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.427+0000 7f7243e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7244104a70 0x7f724419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:36.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.427+0000 7f7243e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7244104a70 0x7f724419ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49336/0 (socket says 172.21.15.40:49336) 2024-04-27T00:01:36.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.427+0000 7f7243e00640 1 -- 172.21.15.40:0/315016694 learned_addr learned my addr 172.21.15.40:0/315016694 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:36.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.427+0000 7f7248e00640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7244106620 0x7f72441a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:36.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.427+0000 7f7243400640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7244105c60 0x7f724419b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:36.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.427+0000 7f7243e00640 1 -- 172.21.15.40:0/315016694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7244106620 msgr2=0x7f72441a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:36.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.427+0000 7f7243e00640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7244106620 0x7f72441a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.427+0000 7f7243e00640 1 -- 172.21.15.40:0/315016694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7244105c60 msgr2=0x7f724419b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:36.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.427+0000 7f7243e00640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7244105c60 0x7f724419b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.427+0000 7f7243e00640 1 -- 172.21.15.40:0/315016694 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72441a2840 con 0x7f7244104a70 2024-04-27T00:01:36.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.427+0000 7f7243400640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7244105c60 0x7f724419b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:36.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.427+0000 7f7243e00640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7244104a70 0x7f724419ac30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f723400e9c0 tx=0x7f723400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:36.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.428+0000 7f7240c00640 1 -- 172.21.15.40:0/315016694 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723400cd80 con 0x7f7244104a70 2024-04-27T00:01:36.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.428+0000 7f7240c00640 1 -- 172.21.15.40:0/315016694 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72340035f0 con 0x7f7244104a70 2024-04-27T00:01:36.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.428+0000 7f7240c00640 1 -- 172.21.15.40:0/315016694 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234010690 con 0x7f7244104a70 2024-04-27T00:01:36.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.428+0000 7f7249800640 1 -- 172.21.15.40:0/315016694 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72441a2b30 con 0x7f7244104a70 2024-04-27T00:01:36.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.428+0000 7f7249800640 1 -- 172.21.15.40:0/315016694 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72441a3030 con 0x7f7244104a70 2024-04-27T00:01:36.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.429+0000 7f7249800640 1 -- 172.21.15.40:0/315016694 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f724410be00 con 0x7f7244104a70 2024-04-27T00:01:36.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.431+0000 7f7240c00640 1 -- 172.21.15.40:0/315016694 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f72340037a0 con 0x7f7244104a70 2024-04-27T00:01:36.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.432+0000 7f7240c00640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f72280a3cc0 0x7f72280a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:36.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.433+0000 7f7240c00640 1 -- 172.21.15.40:0/315016694 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f72340d0d50 con 0x7f7244104a70 2024-04-27T00:01:36.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.433+0000 7f7243400640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f72280a3cc0 0x7f72280a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:36.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.434+0000 7f7243400640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f72280a3cc0 0x7f72280a6180 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7230009c20 tx=0x7f7230005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:36.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.436+0000 7f7240c00640 1 -- 172.21.15.40:0/315016694 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f723409d230 con 0x7f7244104a70 2024-04-27T00:01:36.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.681+0000 7f7249800640 1 -- 172.21.15.40:0/315016694 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f724402dd80 con 0x7f7244104a70 2024-04-27T00:01:36.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.682+0000 7f7240c00640 1 -- 172.21.15.40:0/315016694 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f723409cb30 con 0x7f7244104a70 2024-04-27T00:01:36.683 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-27T00:01:36.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.685+0000 7f7249800640 1 -- 172.21.15.40:0/315016694 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f72280a3cc0 msgr2=0x7f72280a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:36.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.685+0000 7f7249800640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f72280a3cc0 0x7f72280a6180 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7230009c20 tx=0x7f7230005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:36.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.685+0000 7f7249800640 1 -- 172.21.15.40:0/315016694 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7244104a70 msgr2=0x7f724419ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:36.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.685+0000 7f7249800640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7244104a70 0x7f724419ac30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f723400e9c0 tx=0x7f723400ee80 comp rx=0 tx=0).stop 2024-04-27T00:01:36.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.685+0000 7f7249800640 1 -- 172.21.15.40:0/315016694 shutdown_connections 2024-04-27T00:01:36.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.685+0000 7f7249800640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f72280a3cc0 0x7f72280a6180 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.685+0000 7f7249800640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7244106620 0x7f72441a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.685+0000 7f7249800640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7244105c60 0x7f724419b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.686+0000 7f7249800640 1 --2- 172.21.15.40:0/315016694 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7244104a70 0x7f724419ac30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.686+0000 7f7249800640 1 -- 172.21.15.40:0/315016694 >> 172.21.15.40:0/315016694 conn(0x7f72440fffe0 msgr2=0x7f7244108fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:36.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.686+0000 7f7249800640 1 -- 172.21.15.40:0/315016694 shutdown_connections 2024-04-27T00:01:36.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.686+0000 7f7249800640 1 -- 172.21.15.40:0/315016694 wait complete. 2024-04-27T00:01:36.708 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-27T00:01:36.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.900+0000 7f6327600640 1 -- 172.21.15.40:0/1390607757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6320103400 msgr2=0x7f63201057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:36.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.900+0000 7f6327600640 1 --2- 172.21.15.40:0/1390607757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6320103400 0x7f63201057f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6310009900 tx=0x7f631002a400 comp rx=0 tx=0).stop 2024-04-27T00:01:36.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.900+0000 7f6327600640 1 -- 172.21.15.40:0/1390607757 shutdown_connections 2024-04-27T00:01:36.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.900+0000 7f6327600640 1 --2- 172.21.15.40:0/1390607757 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6320108660 0x7f632010aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.900+0000 7f6327600640 1 --2- 172.21.15.40:0/1390607757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6320105d30 0x7f6320108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.900+0000 7f6327600640 1 --2- 172.21.15.40:0/1390607757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6320103400 0x7f63201057f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.900+0000 7f6327600640 1 -- 172.21.15.40:0/1390607757 >> 172.21.15.40:0/1390607757 conn(0x7f63200fcdf0 msgr2=0x7f63200ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.900+0000 7f6327600640 1 -- 172.21.15.40:0/1390607757 shutdown_connections 2024-04-27T00:01:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.900+0000 7f6327600640 1 -- 172.21.15.40:0/1390607757 wait complete. 2024-04-27T00:01:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.901+0000 7f6327600640 1 Processor -- start 2024-04-27T00:01:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.901+0000 7f6327600640 1 -- start start 2024-04-27T00:01:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.902+0000 7f6327600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6320103400 0x7f632019ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.902+0000 7f6327600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6320105d30 0x7f632019b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:36.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.902+0000 7f6327600640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6320108660 0x7f63201a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:36.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.902+0000 7f6327600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f632010e4e0 con 0x7f6320103400 2024-04-27T00:01:36.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.902+0000 7f6327600640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f632010e360 con 0x7f6320108660 2024-04-27T00:01:36.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.902+0000 7f6327600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f632010e660 con 0x7f6320105d30 2024-04-27T00:01:36.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.902+0000 7f6324c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6320103400 0x7f632019ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:36.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.902+0000 7f6324c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6320103400 0x7f632019ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52048/0 (socket says 172.21.15.40:52048) 2024-04-27T00:01:36.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.902+0000 7f6324c00640 1 -- 172.21.15.40:0/3636013559 learned_addr learned my addr 172.21.15.40:0/3636013559 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:36.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.902+0000 7f6325600640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6320108660 0x7f63201a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:36.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.902+0000 7f6317e00640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6320105d30 0x7f632019b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:36.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.903+0000 7f6325600640 1 -- 172.21.15.40:0/3636013559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6320105d30 msgr2=0x7f632019b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:36.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.903+0000 7f6325600640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6320105d30 0x7f632019b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.903+0000 7f6325600640 1 -- 172.21.15.40:0/3636013559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6320103400 msgr2=0x7f632019ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:36.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.903+0000 7f6325600640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6320103400 0x7f632019ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:36.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.903+0000 7f6325600640 1 -- 172.21.15.40:0/3636013559 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63201a2ab0 con 0x7f6320108660 2024-04-27T00:01:36.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.903+0000 7f6317e00640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6320105d30 0x7f632019b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:36.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.903+0000 7f6324c00640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6320103400 0x7f632019ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:36.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.903+0000 7f6325600640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6320108660 0x7f63201a2440 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f631800ec50 tx=0x7f631800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:36.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.903+0000 7f631e000640 1 -- 172.21.15.40:0/3636013559 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6318003b70 con 0x7f6320108660 2024-04-27T00:01:36.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.903+0000 7f6327600640 1 -- 172.21.15.40:0/3636013559 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63201a2da0 con 0x7f6320108660 2024-04-27T00:01:36.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.903+0000 7f631e000640 1 -- 172.21.15.40:0/3636013559 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6318003d10 con 0x7f6320108660 2024-04-27T00:01:36.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.903+0000 7f631e000640 1 -- 172.21.15.40:0/3636013559 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6318010760 con 0x7f6320108660 2024-04-27T00:01:36.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.904+0000 7f6327600640 1 -- 172.21.15.40:0/3636013559 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63201a3260 con 0x7f6320108660 2024-04-27T00:01:36.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.905+0000 7f6327600640 1 -- 172.21.15.40:0/3636013559 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f632010a1b0 con 0x7f6320108660 2024-04-27T00:01:36.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.906+0000 7f631e000640 1 -- 172.21.15.40:0/3636013559 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6318004000 con 0x7f6320108660 2024-04-27T00:01:36.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.908+0000 7f631e000640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f63040a3cc0 0x7f63040a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:36.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.908+0000 7f631e000640 1 -- 172.21.15.40:0/3636013559 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f63180d0e10 con 0x7f6320108660 2024-04-27T00:01:36.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.908+0000 7f6324c00640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f63040a3cc0 0x7f63040a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:36.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.909+0000 7f6324c00640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f63040a3cc0 0x7f63040a6180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6310009230 tx=0x7f63100058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:36.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:36.912+0000 7f631e000640 1 -- 172.21.15.40:0/3636013559 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f63180d4030 con 0x7f6320108660 2024-04-27T00:01:37.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.154+0000 7f6327600640 1 -- 172.21.15.40:0/3636013559 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f63200635d0 con 0x7f6320108660 2024-04-27T00:01:37.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.154+0000 7f631e000640 1 -- 172.21.15.40:0/3636013559 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f631809d2f0 con 0x7f6320108660 2024-04-27T00:01:37.156 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-27T00:01:37.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.157+0000 7f6327600640 1 -- 172.21.15.40:0/3636013559 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f63040a3cc0 msgr2=0x7f63040a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:37.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.157+0000 7f6327600640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f63040a3cc0 0x7f63040a6180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6310009230 tx=0x7f63100058d0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.157+0000 7f6327600640 1 -- 172.21.15.40:0/3636013559 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6320108660 msgr2=0x7f63201a2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:37.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.157+0000 7f6327600640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6320108660 0x7f63201a2440 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f631800ec50 tx=0x7f631800c6a0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.157+0000 7f6327600640 1 -- 172.21.15.40:0/3636013559 shutdown_connections 2024-04-27T00:01:37.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.157+0000 7f6327600640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6320108660 0x7f63201a2440 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.157+0000 7f6327600640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f63040a3cc0 0x7f63040a6180 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.157+0000 7f6327600640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6320105d30 0x7f632019b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.157+0000 7f6327600640 1 --2- 172.21.15.40:0/3636013559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6320103400 0x7f632019ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.157+0000 7f6327600640 1 -- 172.21.15.40:0/3636013559 >> 172.21.15.40:0/3636013559 conn(0x7f63200fcdf0 msgr2=0x7f6320102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:37.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.157+0000 7f6327600640 1 -- 172.21.15.40:0/3636013559 shutdown_connections 2024-04-27T00:01:37.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.158+0000 7f6327600640 1 -- 172.21.15.40:0/3636013559 wait complete. 2024-04-27T00:01:37.179 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-27T00:01:37.237 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:37.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:37.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:37.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:37.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:36 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1568832877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:37.238 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:36 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/315016694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:01:37.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:37.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:37.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:37.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:37.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:36 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1568832877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:37.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:36 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/315016694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:36 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1568832877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:37.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:36 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/315016694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:01:37.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.375+0000 7feb09c00640 1 -- 172.21.15.40:0/51950200 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb041033f0 msgr2=0x7feb041057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:37.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.375+0000 7feb09c00640 1 --2- 172.21.15.40:0/51950200 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb041033f0 0x7feb041057e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7feaf4009900 tx=0x7feaf402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.375+0000 7feb09c00640 1 -- 172.21.15.40:0/51950200 shutdown_connections 2024-04-27T00:01:37.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.375+0000 7feb09c00640 1 --2- 172.21.15.40:0/51950200 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb04108650 0x7feb0410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.375+0000 7feb09c00640 1 --2- 172.21.15.40:0/51950200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb04105d20 0x7feb04108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.375+0000 7feb09c00640 1 --2- 172.21.15.40:0/51950200 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb041033f0 0x7feb041057e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.375+0000 7feb09c00640 1 -- 172.21.15.40:0/51950200 >> 172.21.15.40:0/51950200 conn(0x7feb040fcde0 msgr2=0x7feb040ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:37.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.375+0000 7feb09c00640 1 -- 172.21.15.40:0/51950200 shutdown_connections 2024-04-27T00:01:37.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.375+0000 7feb09c00640 1 -- 172.21.15.40:0/51950200 wait complete. 2024-04-27T00:01:37.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.376+0000 7feb09c00640 1 Processor -- start 2024-04-27T00:01:37.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.377+0000 7feb09c00640 1 -- start start 2024-04-27T00:01:37.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.377+0000 7feb09c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb041033f0 0x7feb04102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:37.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.377+0000 7feb09c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb04105d20 0x7feb041013a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.377+0000 7feb09c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb04108650 0x7feb041018e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.377+0000 7feb09c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb0410e4a0 con 0x7feb04108650 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.377+0000 7feb09c00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7feb0410e320 con 0x7feb04105d20 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.377+0000 7feb09c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb0410e620 con 0x7feb041033f0 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.377+0000 7feb02000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb04105d20 0x7feb041013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.377+0000 7feb02000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb04105d20 0x7feb041013a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49394/0 (socket says 172.21.15.40:49394) 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.377+0000 7feb02000640 1 -- 172.21.15.40:0/3980341815 learned_addr learned my addr 172.21.15.40:0/3980341815 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.377+0000 7feb03400640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb04108650 0x7feb041018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.378+0000 7feb02a00640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb041033f0 0x7feb04102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.378+0000 7feb02000640 1 -- 172.21.15.40:0/3980341815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb041033f0 msgr2=0x7feb04102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.378+0000 7feb02000640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb041033f0 0x7feb04102d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.378+0000 7feb02000640 1 -- 172.21.15.40:0/3980341815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb04108650 msgr2=0x7feb041018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.378+0000 7feb02000640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb04108650 0x7feb041018e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.378+0000 7feb02000640 1 -- 172.21.15.40:0/3980341815 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb04102170 con 0x7feb04105d20 2024-04-27T00:01:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.378+0000 7feb02000640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb04105d20 0x7feb041013a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7feaf000e9c0 tx=0x7feaf000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.379+0000 7feafb400640 1 -- 172.21.15.40:0/3980341815 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf000ccc0 con 0x7feb04105d20 2024-04-27T00:01:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.379+0000 7feb09c00640 1 -- 172.21.15.40:0/3980341815 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb041a2cf0 con 0x7feb04105d20 2024-04-27T00:01:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.379+0000 7feafb400640 1 -- 172.21.15.40:0/3980341815 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaf00054d0 con 0x7feb04105d20 2024-04-27T00:01:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.379+0000 7feafb400640 1 -- 172.21.15.40:0/3980341815 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf0004dc0 con 0x7feb04105d20 2024-04-27T00:01:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.379+0000 7feb09c00640 1 -- 172.21.15.40:0/3980341815 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb041a31d0 con 0x7feb04105d20 2024-04-27T00:01:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.380+0000 7feb09c00640 1 -- 172.21.15.40:0/3980341815 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb040fe8d0 con 0x7feb04105d20 2024-04-27T00:01:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.382+0000 7feafb400640 1 -- 172.21.15.40:0/3980341815 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7feaf0005020 con 0x7feb04105d20 2024-04-27T00:01:37.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.383+0000 7feafb400640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feae80a3cc0 0x7feae80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:37.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.384+0000 7feafb400640 1 -- 172.21.15.40:0/3980341815 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7feaf00d0dd0 con 0x7feb04105d20 2024-04-27T00:01:37.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.384+0000 7feb02a00640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feae80a3cc0 0x7feae80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:37.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.385+0000 7feb02a00640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feae80a3cc0 0x7feae80a6180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feaf4000c00 tx=0x7feaf40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:37.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.388+0000 7feafb400640 1 -- 172.21.15.40:0/3980341815 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7feaf009d2b0 con 0x7feb04105d20 2024-04-27T00:01:37.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.597+0000 7feb09c00640 1 -- 172.21.15.40:0/3980341815 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7feb0402dd20 con 0x7feb04105d20 2024-04-27T00:01:37.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.598+0000 7feafb400640 1 -- 172.21.15.40:0/3980341815 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7feaf009cbb0 con 0x7feb04105d20 2024-04-27T00:01:37.600 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-27T00:01:37.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.602+0000 7feb09c00640 1 -- 172.21.15.40:0/3980341815 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feae80a3cc0 msgr2=0x7feae80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:37.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.602+0000 7feb09c00640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feae80a3cc0 0x7feae80a6180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feaf4000c00 tx=0x7feaf40314a0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.602+0000 7feb09c00640 1 -- 172.21.15.40:0/3980341815 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb04105d20 msgr2=0x7feb041013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:37.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.602+0000 7feb09c00640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb04105d20 0x7feb041013a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7feaf000e9c0 tx=0x7feaf000edc0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.602+0000 7feb09c00640 1 -- 172.21.15.40:0/3980341815 shutdown_connections 2024-04-27T00:01:37.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.602+0000 7feb09c00640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb04108650 0x7feb041018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.602+0000 7feb09c00640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feb04105d20 0x7feb041013a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.602+0000 7feb09c00640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feae80a3cc0 0x7feae80a6180 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.602+0000 7feb09c00640 1 --2- 172.21.15.40:0/3980341815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb041033f0 0x7feb04102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.602+0000 7feb09c00640 1 -- 172.21.15.40:0/3980341815 >> 172.21.15.40:0/3980341815 conn(0x7feb040fcde0 msgr2=0x7feb04103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:37.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.602+0000 7feb09c00640 1 -- 172.21.15.40:0/3980341815 shutdown_connections 2024-04-27T00:01:37.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.602+0000 7feb09c00640 1 -- 172.21.15.40:0/3980341815 wait complete. 2024-04-27T00:01:37.624 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-27T00:01:37.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.832+0000 7f7de2c00640 1 -- 172.21.15.40:0/3721028640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ddc108520 msgr2=0x7f7ddc10a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:37.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.832+0000 7f7de2c00640 1 --2- 172.21.15.40:0/3721028640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ddc108520 0x7f7ddc10a970 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009be0 tx=0x7f7dd402a6d0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.832+0000 7f7de2c00640 1 -- 172.21.15.40:0/3721028640 shutdown_connections 2024-04-27T00:01:37.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.832+0000 7f7de2c00640 1 --2- 172.21.15.40:0/3721028640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ddc108520 0x7f7ddc10a970 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.832+0000 7f7de2c00640 1 --2- 172.21.15.40:0/3721028640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ddc105bf0 0x7f7ddc107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.832+0000 7f7de2c00640 1 --2- 172.21.15.40:0/3721028640 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ddc1032c0 0x7f7ddc1056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.832+0000 7f7de2c00640 1 -- 172.21.15.40:0/3721028640 >> 172.21.15.40:0/3721028640 conn(0x7f7ddc0fccd0 msgr2=0x7f7ddc0ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:37.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.832+0000 7f7de2c00640 1 -- 172.21.15.40:0/3721028640 shutdown_connections 2024-04-27T00:01:37.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.832+0000 7f7de2c00640 1 -- 172.21.15.40:0/3721028640 wait complete. 2024-04-27T00:01:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.833+0000 7f7de2c00640 1 Processor -- start 2024-04-27T00:01:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.833+0000 7f7de2c00640 1 -- start start 2024-04-27T00:01:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de2c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ddc1032c0 0x7f7ddc102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de2c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ddc105bf0 0x7f7ddc101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de2c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ddc108520 0x7f7ddc1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de2c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7ddc10e390 con 0x7f7ddc1032c0 2024-04-27T00:01:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de2c00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f7ddc10e210 con 0x7f7ddc105bf0 2024-04-27T00:01:37.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de2c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ddc10e510 con 0x7f7ddc108520 2024-04-27T00:01:37.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de1800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ddc1032c0 0x7f7ddc102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:37.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de1800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ddc1032c0 0x7f7ddc102c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52082/0 (socket says 172.21.15.40:52082) 2024-04-27T00:01:37.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de1800640 1 -- 172.21.15.40:0/2389049280 learned_addr learned my addr 172.21.15.40:0/2389049280 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:37.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de2200640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ddc108520 0x7f7ddc1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:37.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de0e00640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ddc105bf0 0x7f7ddc101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:37.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de2200640 1 -- 172.21.15.40:0/2389049280 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ddc105bf0 msgr2=0x7f7ddc101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:37.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de2200640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ddc105bf0 0x7f7ddc101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de2200640 1 -- 172.21.15.40:0/2389049280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ddc1032c0 msgr2=0x7f7ddc102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:37.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de2200640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ddc1032c0 0x7f7ddc102c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:37.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.834+0000 7f7de2200640 1 -- 172.21.15.40:0/2389049280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ddc101cf0 con 0x7f7ddc108520 2024-04-27T00:01:37.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.835+0000 7f7de0e00640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ddc105bf0 0x7f7ddc101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:37.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.835+0000 7f7de2200640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ddc108520 0x7f7ddc1017b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4034040 tx=0x7f7dd4030d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:37.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.835+0000 7f7dda000640 1 -- 172.21.15.40:0/2389049280 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4041020 con 0x7f7ddc108520 2024-04-27T00:01:37.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.835+0000 7f7dda000640 1 -- 172.21.15.40:0/2389049280 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dd40318f0 con 0x7f7ddc108520 2024-04-27T00:01:37.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.835+0000 7f7dda000640 1 -- 172.21.15.40:0/2389049280 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4036040 con 0x7f7ddc108520 2024-04-27T00:01:37.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.835+0000 7f7de2c00640 1 -- 172.21.15.40:0/2389049280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ddc101f80 con 0x7f7ddc108520 2024-04-27T00:01:37.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.835+0000 7f7de2c00640 1 -- 172.21.15.40:0/2389049280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ddc1a7440 con 0x7f7ddc108520 2024-04-27T00:01:37.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.836+0000 7f7de2c00640 1 -- 172.21.15.40:0/2389049280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ddc0fe7c0 con 0x7f7ddc108520 2024-04-27T00:01:37.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.839+0000 7f7dda000640 1 -- 172.21.15.40:0/2389049280 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7dd4044020 con 0x7f7ddc108520 2024-04-27T00:01:37.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.840+0000 7f7dda000640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7dc40a3c40 0x7f7dc40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:37.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.840+0000 7f7dda000640 1 -- 172.21.15.40:0/2389049280 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f7dd40f18b0 con 0x7f7ddc108520 2024-04-27T00:01:37.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.840+0000 7f7de1800640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7dc40a3c40 0x7f7dc40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:37.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.841+0000 7f7de1800640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7dc40a3c40 0x7f7dc40a6100 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0009cd0 tx=0x7f7dd0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:37.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:37.842+0000 7f7dda000640 1 -- 172.21.15.40:0/2389049280 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7dd404b020 con 0x7f7ddc108520 2024-04-27T00:01:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.058+0000 7f7de2c00640 1 -- 172.21.15.40:0/2389049280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7ddc02dc60 con 0x7f7ddc108520 2024-04-27T00:01:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.059+0000 7f7dda000640 1 -- 172.21.15.40:0/2389049280 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f7dd40bdea0 con 0x7f7ddc108520 2024-04-27T00:01:38.061 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-27T00:01:38.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.062+0000 7f7de2c00640 1 -- 172.21.15.40:0/2389049280 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7dc40a3c40 msgr2=0x7f7dc40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.062+0000 7f7de2c00640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7dc40a3c40 0x7f7dc40a6100 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0009cd0 tx=0x7f7dd0005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.062+0000 7f7de2c00640 1 -- 172.21.15.40:0/2389049280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ddc108520 msgr2=0x7f7ddc1017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.062+0000 7f7de2c00640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ddc108520 0x7f7ddc1017b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4034040 tx=0x7f7dd4030d50 comp rx=0 tx=0).stop 2024-04-27T00:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.062+0000 7f7de2c00640 1 -- 172.21.15.40:0/2389049280 shutdown_connections 2024-04-27T00:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.062+0000 7f7de2c00640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7dc40a3c40 0x7f7dc40a6100 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.062+0000 7f7de2c00640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ddc108520 0x7f7ddc1017b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.062+0000 7f7de2c00640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7ddc105bf0 0x7f7ddc101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.062+0000 7f7de2c00640 1 --2- 172.21.15.40:0/2389049280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ddc1032c0 0x7f7ddc102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.062+0000 7f7de2c00640 1 -- 172.21.15.40:0/2389049280 >> 172.21.15.40:0/2389049280 conn(0x7f7ddc0fccd0 msgr2=0x7f7ddc103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.062+0000 7f7de2c00640 1 -- 172.21.15.40:0/2389049280 shutdown_connections 2024-04-27T00:01:38.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.062+0000 7f7de2c00640 1 -- 172.21.15.40:0/2389049280 wait complete. 2024-04-27T00:01:38.085 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-27T00:01:38.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.277+0000 7fb738200640 1 -- 172.21.15.40:0/2137611840 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb730076970 msgr2=0x7fb730074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:38.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.277+0000 7fb738200640 1 --2- 172.21.15.40:0/2137611840 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb730076970 0x7fb730074dc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb724009930 tx=0x7fb72402a420 comp rx=0 tx=0).stop 2024-04-27T00:01:38.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.277+0000 7fb738200640 1 -- 172.21.15.40:0/2137611840 shutdown_connections 2024-04-27T00:01:38.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.277+0000 7fb738200640 1 --2- 172.21.15.40:0/2137611840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb73010ef40 0x7fb730111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.277+0000 7fb738200640 1 --2- 172.21.15.40:0/2137611840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb730075390 0x7fb730075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.277+0000 7fb738200640 1 --2- 172.21.15.40:0/2137611840 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb730076970 0x7fb730074dc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.277+0000 7fb738200640 1 -- 172.21.15.40:0/2137611840 >> 172.21.15.40:0/2137611840 conn(0x7fb7300fdee0 msgr2=0x7fb730100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:38.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.277+0000 7fb738200640 1 -- 172.21.15.40:0/2137611840 shutdown_connections 2024-04-27T00:01:38.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.277+0000 7fb738200640 1 -- 172.21.15.40:0/2137611840 wait complete. 2024-04-27T00:01:38.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.278+0000 7fb738200640 1 Processor -- start 2024-04-27T00:01:38.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.278+0000 7fb738200640 1 -- start start 2024-04-27T00:01:38.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.278+0000 7fb738200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb730075390 0x7fb73019f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:38.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.278+0000 7fb738200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb730076970 0x7fb73019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:38.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb738200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb73010ef40 0x7fb7301a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:38.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb738200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb730114af0 con 0x7fb730075390 2024-04-27T00:01:38.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb738200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb730114970 con 0x7fb730076970 2024-04-27T00:01:38.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb738200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb730114c70 con 0x7fb73010ef40 2024-04-27T00:01:38.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb734e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb730076970 0x7fb73019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:38.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb735800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb730075390 0x7fb73019f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:38.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb734e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb730076970 0x7fb73019f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49430/0 (socket says 172.21.15.40:49430) 2024-04-27T00:01:38.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb734e00640 1 -- 172.21.15.40:0/705816707 learned_addr learned my addr 172.21.15.40:0/705816707 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb736200640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb73010ef40 0x7fb7301a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb734e00640 1 -- 172.21.15.40:0/705816707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb73010ef40 msgr2=0x7fb7301a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb734e00640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb73010ef40 0x7fb7301a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb734e00640 1 -- 172.21.15.40:0/705816707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb730075390 msgr2=0x7fb73019f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb734e00640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb730075390 0x7fb73019f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.279+0000 7fb734e00640 1 -- 172.21.15.40:0/705816707 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7301a6db0 con 0x7fb730076970 2024-04-27T00:01:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.280+0000 7fb734e00640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb730076970 0x7fb73019f6d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb72000d950 tx=0x7fb72000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.280+0000 7fb72e000640 1 -- 172.21.15.40:0/705816707 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb720017070 con 0x7fb730076970 2024-04-27T00:01:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.280+0000 7fb72e000640 1 -- 172.21.15.40:0/705816707 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb720009c30 con 0x7fb730076970 2024-04-27T00:01:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.280+0000 7fb72e000640 1 -- 172.21.15.40:0/705816707 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72000f620 con 0x7fb730076970 2024-04-27T00:01:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.280+0000 7fb738200640 1 -- 172.21.15.40:0/705816707 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7301a70a0 con 0x7fb730076970 2024-04-27T00:01:38.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.280+0000 7fb738200640 1 -- 172.21.15.40:0/705816707 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7301a75a0 con 0x7fb730076970 2024-04-27T00:01:38.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.281+0000 7fb738200640 1 -- 172.21.15.40:0/705816707 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb730105110 con 0x7fb730076970 2024-04-27T00:01:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.283+0000 7fb72e000640 1 -- 172.21.15.40:0/705816707 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb720010460 con 0x7fb730076970 2024-04-27T00:01:38.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.286+0000 7fb72e000640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7180a3c40 0x7fb7180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:38.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.286+0000 7fb72e000640 1 -- 172.21.15.40:0/705816707 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fb7200d04b0 con 0x7fb730076970 2024-04-27T00:01:38.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.286+0000 7fb735800640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7180a3c40 0x7fb7180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.287+0000 7fb735800640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7180a3c40 0x7fb7180a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb724005ab0 tx=0x7fb724005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.289+0000 7fb72e000640 1 -- 172.21.15.40:0/705816707 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb720009de0 con 0x7fb730076970 2024-04-27T00:01:38.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:38 smithi080 ceph-mon[32362]: pgmap v408: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:38.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:38 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3636013559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:38.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:38.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:38.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:38 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3980341815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:38.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:38 smithi103 ceph-mon[32952]: pgmap v408: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:38.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:38 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3636013559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:38.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:38.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:38.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:38 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3980341815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:38.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:38 smithi040 ceph-mon[28095]: pgmap v408: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:38.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:38 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3636013559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:38.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:38.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:38.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:38 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3980341815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:38.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.539+0000 7fb738200640 1 -- 172.21.15.40:0/705816707 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb73002dd20 con 0x7fb730076970 2024-04-27T00:01:38.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.546+0000 7fb72e000640 1 -- 172.21.15.40:0/705816707 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11872 (secure 0 0 0) 0x7fb72009c340 con 0x7fb730076970 2024-04-27T00:01:38.548 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:38.548 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/2265538751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":2265538751},{"type":"v1","addr":"172.21.15.40:6841","nonce":2265538751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:58:50.616677+0000","modified":"2024-04-26T23:59:25.374905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24523":{"gid":24523,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.80:6839/3536638444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6838","nonce":3536638444},{"type":"v1","addr":"172.21.15.80:6839","nonce":3536638444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24559,24560,24590]},"id":1}]} 2024-04-27T00:01:38.548 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 23 2024-04-27T00:01:38.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.549+0000 7fb738200640 1 -- 172.21.15.40:0/705816707 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7180a3c40 msgr2=0x7fb7180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.549+0000 7fb738200640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7180a3c40 0x7fb7180a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb724005ab0 tx=0x7fb724005a20 comp rx=0 tx=0).stop 2024-04-27T00:01:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.549+0000 7fb738200640 1 -- 172.21.15.40:0/705816707 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb730076970 msgr2=0x7fb73019f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.549+0000 7fb738200640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb730076970 0x7fb73019f6d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb72000d950 tx=0x7fb72000de10 comp rx=0 tx=0).stop 2024-04-27T00:01:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.549+0000 7fb738200640 1 -- 172.21.15.40:0/705816707 shutdown_connections 2024-04-27T00:01:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.549+0000 7fb738200640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7180a3c40 0x7fb7180a6100 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.549+0000 7fb738200640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb73010ef40 0x7fb7301a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.549+0000 7fb738200640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb730076970 0x7fb73019f6d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.549+0000 7fb738200640 1 --2- 172.21.15.40:0/705816707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb730075390 0x7fb73019f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.549+0000 7fb738200640 1 -- 172.21.15.40:0/705816707 >> 172.21.15.40:0/705816707 conn(0x7fb7300fdee0 msgr2=0x7fb7301022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.549+0000 7fb738200640 1 -- 172.21.15.40:0/705816707 shutdown_connections 2024-04-27T00:01:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.549+0000 7fb738200640 1 -- 172.21.15.40:0/705816707 wait complete. 2024-04-27T00:01:38.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T00:01:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.764+0000 7f9757e00640 1 -- 172.21.15.40:0/3449452395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9758104960 msgr2=0x7f9758104d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.764+0000 7f9757e00640 1 --2- 172.21.15.40:0/3449452395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9758104960 0x7f9758104d80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9748009900 tx=0x7f974802a3f0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.764+0000 7f9757e00640 1 -- 172.21.15.40:0/3449452395 shutdown_connections 2024-04-27T00:01:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.764+0000 7f9757e00640 1 --2- 172.21.15.40:0/3449452395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9758106510 0x7f975810cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.764+0000 7f9757e00640 1 --2- 172.21.15.40:0/3449452395 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9758105b50 0x7f9758105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.764+0000 7f9757e00640 1 --2- 172.21.15.40:0/3449452395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9758104960 0x7f9758104d80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.764+0000 7f9757e00640 1 -- 172.21.15.40:0/3449452395 >> 172.21.15.40:0/3449452395 conn(0x7f97580ffeb0 msgr2=0x7f9758102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.764+0000 7f9757e00640 1 -- 172.21.15.40:0/3449452395 shutdown_connections 2024-04-27T00:01:38.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.764+0000 7f9757e00640 1 -- 172.21.15.40:0/3449452395 wait complete. 2024-04-27T00:01:38.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.765+0000 7f9757e00640 1 Processor -- start 2024-04-27T00:01:38.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.766+0000 7f9757e00640 1 -- start start 2024-04-27T00:01:38.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.766+0000 7f9757e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9758104960 0x7f975819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:38.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.766+0000 7f9757e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9758105b50 0x7f975819b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:38.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.766+0000 7f9757e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9758106510 0x7f97581a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:38.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.766+0000 7f9757e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9758110460 con 0x7f9758106510 2024-04-27T00:01:38.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.766+0000 7f9757e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f97581102e0 con 0x7f9758104960 2024-04-27T00:01:38.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.766+0000 7f9757e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f97581105e0 con 0x7f9758105b50 2024-04-27T00:01:38.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.766+0000 7f9756a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9758104960 0x7f975819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:38.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.766+0000 7f974fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9758105b50 0x7f975819b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:38.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.766+0000 7f974fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9758105b50 0x7f975819b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47936/0 (socket says 172.21.15.40:47936) 2024-04-27T00:01:38.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.766+0000 7f974fe00640 1 -- 172.21.15.40:0/2000984794 learned_addr learned my addr 172.21.15.40:0/2000984794 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:38.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.766+0000 7f9757400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9758106510 0x7f97581a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:38.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.767+0000 7f974fe00640 1 -- 172.21.15.40:0/2000984794 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9758104960 msgr2=0x7f975819abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:38.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.767+0000 7f974fe00640 1 --2- 172.21.15.40:0/2000984794 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9758104960 0x7f975819abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.767+0000 7f974fe00640 1 -- 172.21.15.40:0/2000984794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9758106510 msgr2=0x7f97581a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:38.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.767+0000 7f974fe00640 1 --2- 172.21.15.40:0/2000984794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9758106510 0x7f97581a2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.767+0000 7f974fe00640 1 -- 172.21.15.40:0/2000984794 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97581a27d0 con 0x7f9758105b50 2024-04-27T00:01:38.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.767+0000 7f974fe00640 1 --2- 172.21.15.40:0/2000984794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9758105b50 0x7f975819b0f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f974000e9f0 tx=0x7f974000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:38.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.767+0000 7f974f400640 1 -- 172.21.15.40:0/2000984794 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974000cd80 con 0x7f9758105b50 2024-04-27T00:01:38.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.768+0000 7f974f400640 1 -- 172.21.15.40:0/2000984794 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97400108e0 con 0x7f9758105b50 2024-04-27T00:01:38.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.768+0000 7f9757e00640 1 -- 172.21.15.40:0/2000984794 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97581a2ac0 con 0x7f9758105b50 2024-04-27T00:01:38.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.768+0000 7f974f400640 1 -- 172.21.15.40:0/2000984794 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9740018980 con 0x7f9758105b50 2024-04-27T00:01:38.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.768+0000 7f9757e00640 1 -- 172.21.15.40:0/2000984794 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97581a3010 con 0x7f9758105b50 2024-04-27T00:01:38.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.769+0000 7f9757e00640 1 -- 172.21.15.40:0/2000984794 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f975810bd40 con 0x7f9758105b50 2024-04-27T00:01:38.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.771+0000 7f974f400640 1 -- 172.21.15.40:0/2000984794 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9740010430 con 0x7f9758105b50 2024-04-27T00:01:38.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.772+0000 7f974f400640 1 --2- 172.21.15.40:0/2000984794 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f973c0a3c40 0x7f973c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:38.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.773+0000 7f974f400640 1 -- 172.21.15.40:0/2000984794 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f97400d0e90 con 0x7f9758105b50 2024-04-27T00:01:38.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.773+0000 7f9756a00640 1 --2- 172.21.15.40:0/2000984794 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f973c0a3c40 0x7f973c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:38.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.774+0000 7f9756a00640 1 --2- 172.21.15.40:0/2000984794 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f973c0a3c40 0x7f973c0a6100 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9748009230 tx=0x7f97480314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:38.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.774+0000 7f974f400640 1 -- 172.21.15.40:0/2000984794 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f974009cd20 con 0x7f9758105b50 2024-04-27T00:01:38.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.984+0000 7f9757e00640 1 -- 172.21.15.40:0/2000984794 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97580635d0 con 0x7f9758105b50 2024-04-27T00:01:38.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.986+0000 7f974f400640 1 -- 172.21.15.40:0/2000984794 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24531 (secure 0 0 0) 0x7f97400a1020 con 0x7f9758105b50 2024-04-27T00:01:38.988 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:01:38.988 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":92,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-27T00:00:46.365303+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9},{"from":5,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T00:01:38.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.989+0000 7f9757e00640 1 -- 172.21.15.40:0/2000984794 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f973c0a3c40 msgr2=0x7f973c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:38.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.989+0000 7f9757e00640 1 --2- 172.21.15.40:0/2000984794 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f973c0a3c40 0x7f973c0a6100 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9748009230 tx=0x7f97480314a0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.989+0000 7f9757e00640 1 -- 172.21.15.40:0/2000984794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9758105b50 msgr2=0x7f975819b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:38.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.989+0000 7f9757e00640 1 --2- 172.21.15.40:0/2000984794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9758105b50 0x7f975819b0f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f974000e9f0 tx=0x7f974000eeb0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.989+0000 7f9757e00640 1 -- 172.21.15.40:0/2000984794 shutdown_connections 2024-04-27T00:01:38.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.989+0000 7f9757e00640 1 --2- 172.21.15.40:0/2000984794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9758106510 0x7f97581a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.989+0000 7f9757e00640 1 --2- 172.21.15.40:0/2000984794 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f973c0a3c40 0x7f973c0a6100 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.989+0000 7f9757e00640 1 --2- 172.21.15.40:0/2000984794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9758105b50 0x7f975819b0f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.989+0000 7f9757e00640 1 --2- 172.21.15.40:0/2000984794 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9758104960 0x7f975819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:38.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.989+0000 7f9757e00640 1 -- 172.21.15.40:0/2000984794 >> 172.21.15.40:0/2000984794 conn(0x7f97580ffeb0 msgr2=0x7f9758108f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:38.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.989+0000 7f9757e00640 1 -- 172.21.15.40:0/2000984794 shutdown_connections 2024-04-27T00:01:38.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:38.990+0000 7f9757e00640 1 -- 172.21.15.40:0/2000984794 wait complete. 2024-04-27T00:01:39.012 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T23:58:50.616677+0000', 'modified': '2024-04-26T23:59:25.374905+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24590, 'mds_1': 24560, 'mds_2': 24559}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24523': {'gid': 24523, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.103:6835/2596993377', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6834', 'nonce': 2596993377}, {'type': 'v1', 'addr': '172.21.15.103:6835', 'nonce': 2596993377}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.80:6837/4116866934', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6836', 'nonce': 4116866934}, {'type': 'v1', 'addr': '172.21.15.80:6837', 'nonce': 4116866934}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.103:6837/2206925949', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 2206925949}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 2206925949}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24559': {'gid': 24559, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.103:6839/1527392010', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6838', 'nonce': 1527392010}, {'type': 'v1', 'addr': '172.21.15.103:6839', 'nonce': 1527392010}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.40:6839/2205335602', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 2205335602}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 2205335602}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.80:6839/3536638444', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.80:6838', 'nonce': 3536638444}, {'type': 'v1', 'addr': '172.21.15.80:6839', 'nonce': 3536638444}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24590, 'qdb_cluster': [24559, 24560, 24590]} 2024-04-27T00:01:39.013 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-27T00:01:39.013 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-27T00:01:39.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.231+0000 7f196a600640 1 -- 172.21.15.40:0/23009858 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f196410a1a0 msgr2=0x7f196410a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.231+0000 7f196a600640 1 --2- 172.21.15.40:0/23009858 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f196410a1a0 0x7f196410a620 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f194c009900 tx=0x7f194c02a410 comp rx=0 tx=0).stop 2024-04-27T00:01:39.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.231+0000 7f196a600640 1 -- 172.21.15.40:0/23009858 shutdown_connections 2024-04-27T00:01:39.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.231+0000 7f196a600640 1 --2- 172.21.15.40:0/23009858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f196410ab60 0x7f19641113f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.231+0000 7f196a600640 1 --2- 172.21.15.40:0/23009858 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f196410a1a0 0x7f196410a620 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.231+0000 7f196a600640 1 --2- 172.21.15.40:0/23009858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1964108fb0 0x7f19641093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.231+0000 7f196a600640 1 -- 172.21.15.40:0/23009858 >> 172.21.15.40:0/23009858 conn(0x7f1964077f40 msgr2=0x7f196407a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.231+0000 7f196a600640 1 -- 172.21.15.40:0/23009858 shutdown_connections 2024-04-27T00:01:39.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.231+0000 7f196a600640 1 -- 172.21.15.40:0/23009858 wait complete. 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.232+0000 7f196a600640 1 Processor -- start 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.232+0000 7f196a600640 1 -- start start 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.233+0000 7f196a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1964108fb0 0x7f196419f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.233+0000 7f196a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f196410a1a0 0x7f196419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.233+0000 7f196a600640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f196410ab60 0x7f19641a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.233+0000 7f196a600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1964114b40 con 0x7f1964108fb0 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.233+0000 7f196a600640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f19641149c0 con 0x7f196410ab60 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.233+0000 7f196a600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1964114cc0 con 0x7f196410a1a0 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.233+0000 7f1963e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f196410ab60 0x7f19641a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.233+0000 7f1963400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1964108fb0 0x7f196419f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.233+0000 7f1963e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f196410ab60 0x7f19641a6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49468/0 (socket says 172.21.15.40:49468) 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.233+0000 7f1963e00640 1 -- 172.21.15.40:0/2984412482 learned_addr learned my addr 172.21.15.40:0/2984412482 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.233+0000 7f195be00640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f196410a1a0 0x7f196419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.234+0000 7f1963e00640 1 -- 172.21.15.40:0/2984412482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f196410a1a0 msgr2=0x7f196419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.234+0000 7f1963e00640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f196410a1a0 0x7f196419f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.234+0000 7f1963e00640 1 -- 172.21.15.40:0/2984412482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1964108fb0 msgr2=0x7f196419f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.234+0000 7f1963e00640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1964108fb0 0x7f196419f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.234+0000 7f1963e00640 1 -- 172.21.15.40:0/2984412482 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19641a6df0 con 0x7f196410ab60 2024-04-27T00:01:39.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.234+0000 7f1963e00640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f196410ab60 0x7f19641a6780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f195c00efd0 tx=0x7f195c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.234+0000 7f1960c00640 1 -- 172.21.15.40:0/2984412482 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c003b70 con 0x7f196410ab60 2024-04-27T00:01:39.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.234+0000 7f1960c00640 1 -- 172.21.15.40:0/2984412482 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f195c003d10 con 0x7f196410ab60 2024-04-27T00:01:39.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.234+0000 7f1960c00640 1 -- 172.21.15.40:0/2984412482 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c0108a0 con 0x7f196410ab60 2024-04-27T00:01:39.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.234+0000 7f196a600640 1 -- 172.21.15.40:0/2984412482 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19641a70e0 con 0x7f196410ab60 2024-04-27T00:01:39.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.234+0000 7f196a600640 1 -- 172.21.15.40:0/2984412482 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19641a7650 con 0x7f196410ab60 2024-04-27T00:01:39.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.238+0000 7f196d424740 1 Processor -- start 2024-04-27T00:01:39.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.238+0000 7f1960c00640 1 -- 172.21.15.40:0/2984412482 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f195c004020 con 0x7f196410ab60 2024-04-27T00:01:39.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.238+0000 7f196d424740 1 -- start start 2024-04-27T00:01:39.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.238+0000 7f196d424740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cf0f0 0x55dadd4cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.238+0000 7f196d424740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cfae0 0x55dadd4d45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.238+0000 7f196d424740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4d70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.238+0000 7f196d424740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dadd433600 con 0x55dadd4cf0f0 2024-04-27T00:01:39.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.238+0000 7f196d424740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55dadd3d0100 con 0x55dadd4d4c50 2024-04-27T00:01:39.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.238+0000 7f196d424740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dadd42f5c0 con 0x55dadd4cfae0 2024-04-27T00:01:39.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.239+0000 7f1963400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cfae0 0x55dadd4d45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.239+0000 7f195be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cf0f0 0x55dadd4cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.239+0000 7f1963400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cfae0 0x55dadd4d45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47968/0 (socket says 172.21.15.40:47968) 2024-04-27T00:01:39.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.239+0000 7f1963400640 1 -- 172.21.15.40:0/3014249725 learned_addr learned my addr 172.21.15.40:0/3014249725 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.239+0000 7f195be00640 1 --2- 172.21.15.40:0/3014249725 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4d70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.239+0000 7f1963400640 1 -- 172.21.15.40:0/3014249725 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 msgr2=0x55dadd4d70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.239+0000 7f1963400640 1 --2- 172.21.15.40:0/3014249725 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4d70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.239+0000 7f1963400640 1 -- 172.21.15.40:0/3014249725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cf0f0 msgr2=0x55dadd4cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.239+0000 7f1963400640 1 --2- 172.21.15.40:0/3014249725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cf0f0 0x55dadd4cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.239+0000 7f1963400640 1 -- 172.21.15.40:0/3014249725 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dadd1452d0 con 0x55dadd4cfae0 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.239+0000 7f195be00640 1 --2- 172.21.15.40:0/3014249725 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4d70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.239+0000 7f1963400640 1 --2- 172.21.15.40:0/3014249725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cfae0 0x55dadd4d45e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1954009ca0 tx=0x7f1954011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f873e3c91f1f919 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.240+0000 7f1960c00640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f19440a3cc0 0x7f19440a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.240+0000 7f1958c00640 1 -- 172.21.15.40:0/3014249725 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954004220 con 0x55dadd4cfae0 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.240+0000 7f1958c00640 1 -- 172.21.15.40:0/3014249725 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19540043c0 con 0x55dadd4cfae0 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.240+0000 7f1958c00640 1 -- 172.21.15.40:0/3014249725 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954019810 con 0x55dadd4cfae0 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.240+0000 7f1960c00640 1 -- 172.21.15.40:0/2984412482 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f195c0d0c80 con 0x7f196410ab60 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.240+0000 7f195be00640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f19440a3cc0 0x7f19440a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.240+0000 7f196d424740 1 -- 172.21.15.40:0/3014249725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cfae0 msgr2=0x55dadd4d45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.240+0000 7f196d424740 1 --2- 172.21.15.40:0/3014249725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cfae0 0x55dadd4d45e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1954009ca0 tx=0x7f1954011810 comp rx=0 tx=0).stop 2024-04-27T00:01:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.241+0000 7f196d424740 1 -- 172.21.15.40:0/3014249725 shutdown_connections 2024-04-27T00:01:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.241+0000 7f196d424740 1 --2- 172.21.15.40:0/3014249725 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4d70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.241+0000 7f196d424740 1 --2- 172.21.15.40:0/3014249725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cfae0 0x55dadd4d45e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.241+0000 7f196d424740 1 --2- 172.21.15.40:0/3014249725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cf0f0 0x55dadd4cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.241+0000 7f196d424740 1 -- 172.21.15.40:0/3014249725 >> 172.21.15.40:0/3014249725 conn(0x55dadd463d10 msgr2=0x55dadd4cab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.241+0000 7f196d424740 1 -- 172.21.15.40:0/3014249725 shutdown_connections 2024-04-27T00:01:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.241+0000 7f195be00640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f19440a3cc0 0x7f19440a6180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f194c031500 tx=0x7f194c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.241+0000 7f196d424740 1 -- 172.21.15.40:0/3014249725 wait complete. 2024-04-27T00:01:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.241+0000 7f196d424740 1 -- start start 2024-04-27T00:01:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f196d424740 1 Processor -- start 2024-04-27T00:01:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f196d424740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4d9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f196d424740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4cfae0 0x55dadd4d9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f196d424740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4d4c50 0x55dadd4e0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f196d424740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dadd4d7c00 con 0x55dadd4d4c50 2024-04-27T00:01:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f196d424740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55dadd3baf70 con 0x55dadd4cfae0 2024-04-27T00:01:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f196d424740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dadd4d7d80 con 0x55dadd4cf0f0 2024-04-27T00:01:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f196d424740 10 client.0 _ll_register_callbacks cb 0x55dadd429480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:01:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f195be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4cfae0 0x55dadd4d9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f1963e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4d4c50 0x55dadd4e0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f195be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4cfae0 0x55dadd4d9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49484/0 (socket says 172.21.15.40:49484) 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f1963400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4d9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.242+0000 7f195be00640 1 -- 172.21.15.40:0/2413137386 learned_addr learned my addr 172.21.15.40:0/2413137386 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.243+0000 7f1963400640 1 -- 172.21.15.40:0/2413137386 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4cfae0 msgr2=0x55dadd4d9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.243+0000 7f1963400640 1 --2- 172.21.15.40:0/2413137386 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4cfae0 0x55dadd4d9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.243+0000 7f1963400640 1 -- 172.21.15.40:0/2413137386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4d4c50 msgr2=0x55dadd4e0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.243+0000 7f1963400640 1 --2- 172.21.15.40:0/2413137386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4d4c50 0x55dadd4e0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.243+0000 7f1963400640 1 -- 172.21.15.40:0/2413137386 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dadd20f6b0 con 0x55dadd4cf0f0 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.243+0000 7f1963400640 1 --2- 172.21.15.40:0/2413137386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4d9540 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x55dadd4cba20 tx=0x7f1954003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75f67a3d39ce8f86 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.243+0000 7f1951600640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.243+0000 7f1951600640 1 -- 172.21.15.40:0/2413137386 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954028070 con 0x55dadd4cf0f0 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.243+0000 7f1951600640 1 -- 172.21.15.40:0/2413137386 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f1954005040 con 0x55dadd4cf0f0 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.243+0000 7f196d424740 1 -- 172.21.15.40:0/2413137386 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19540043c0 con 0x55dadd4cf0f0 2024-04-27T00:01:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.244+0000 7f1951600640 1 -- 172.21.15.40:0/2413137386 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1954034050 con 0x55dadd4cf0f0 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.244+0000 7f196d424740 10 client.25345 fetch_fsmap learned FSMap version 23 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.244+0000 7f196d424740 1 -- 172.21.15.40:0/2413137386 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55dadd4e3a10 con 0x55dadd4cf0f0 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.244+0000 7f1951600640 1 -- 172.21.15.40:0/2413137386 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19540043c0 con 0x55dadd4cf0f0 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.244+0000 7f1951600640 1 -- 172.21.15.40:0/2413137386 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f1954004b90 con 0x55dadd4cf0f0 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.244+0000 7f196d424740 10 client.25345 fetch_fsmap finished waiting for FSMap version 23 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.244+0000 7f196d424740 10 client.25345 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.244+0000 7f196d424740 20 client.25345 populate_metadata read hostname 'smithi040' 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.244+0000 7f196d424740 1 --2- 172.21.15.40:0/2413137386 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55dadd4e5e10 0x55dadd4e7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.245+0000 7f196d424740 4 client.25345 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.245+0000 7f196d424740 1 -- 172.21.15.40:0/2413137386 --> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dadd441210 con 0x55dadd4e5e10 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.245+0000 7f195be00640 1 --2- 172.21.15.40:0/2413137386 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55dadd4e5e10 0x55dadd4e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.246+0000 7f195be00640 1 --2- 172.21.15.40:0/2413137386 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55dadd4e5e10 0x55dadd4e7280 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=235b747585087208 server_cookie=d38f91ba1f15b743 in_seq=0 out_seq=0 2024-04-27T00:01:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.246+0000 7f1951600640 10 client.25345 ms_handle_connect on v2:172.21.15.103:6838/1527392010 2024-04-27T00:01:39.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f1951600640 1 -- 172.21.15.40:0/2413137386 <== mds.2 v2:172.21.15.103:6838/1527392010 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dadd441210 con 0x55dadd4e5e10 2024-04-27T00:01:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f1951600640 10 client.25345 handle_command_reply: tid=0 2024-04-27T00:01:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f196d424740 1 client.25345 shutdown 2024-04-27T00:01:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f196d424740 1 -- 172.21.15.40:0/2413137386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 msgr2=0x55dadd4d9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f196d424740 1 --2- 172.21.15.40:0/2413137386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4d9540 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x55dadd4cba20 tx=0x7f1954003820 comp rx=0 tx=0).stop 2024-04-27T00:01:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f196d424740 1 -- 172.21.15.40:0/2413137386 shutdown_connections 2024-04-27T00:01:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f196d424740 1 --2- 172.21.15.40:0/2413137386 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55dadd4e5e10 0x55dadd4e7280 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f196d424740 1 --2- 172.21.15.40:0/2413137386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4d4c50 0x55dadd4e0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f196d424740 1 --2- 172.21.15.40:0/2413137386 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4cfae0 0x55dadd4d9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f196d424740 1 --2- 172.21.15.40:0/2413137386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4d9540 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f1951600640 0 client.25345 ms_handle_reset on v2:172.21.15.103:6838/1527392010 2024-04-27T00:01:39.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f196d424740 1 -- 172.21.15.40:0/2413137386 >> 172.21.15.40:0/2413137386 conn(0x55dadd463d10 msgr2=0x55dadd4c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f196d424740 1 -- 172.21.15.40:0/2413137386 shutdown_connections 2024-04-27T00:01:39.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f196d424740 1 -- 172.21.15.40:0/2413137386 wait complete. 2024-04-27T00:01:39.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.247+0000 7f196d424740 20 client.25345 trim_cache size 0 max 16384 2024-04-27T00:01:39.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.268+0000 7f196d424740 1 Processor -- start 2024-04-27T00:01:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.268+0000 7f196d424740 1 -- start start 2024-04-27T00:01:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.268+0000 7f196d424740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4c9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.268+0000 7f196d424740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cfae0 0x55dadd4e09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.268+0000 7f196d424740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.268+0000 7f196d424740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dadd4e55c0 con 0x55dadd4cfae0 2024-04-27T00:01:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.268+0000 7f196d424740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55dadd4e5c90 con 0x55dadd4d4c50 2024-04-27T00:01:39.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.268+0000 7f196d424740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dadd4e6360 con 0x55dadd4cf0f0 2024-04-27T00:01:39.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.269+0000 7f1963e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.269+0000 7f195be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cfae0 0x55dadd4e09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.269+0000 7f1963e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4c9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49498/0 (socket says 172.21.15.40:49498) 2024-04-27T00:01:39.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.269+0000 7f1963e00640 1 -- 172.21.15.40:0/3731793413 learned_addr learned my addr 172.21.15.40:0/3731793413 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.269+0000 7f1963400640 1 --2- 172.21.15.40:0/3731793413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.269+0000 7f1963e00640 1 -- 172.21.15.40:0/3731793413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 msgr2=0x55dadd4c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.269+0000 7f1963e00640 1 --2- 172.21.15.40:0/3731793413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.269+0000 7f1963e00640 1 -- 172.21.15.40:0/3731793413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cfae0 msgr2=0x55dadd4e09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.269+0000 7f1963e00640 1 --2- 172.21.15.40:0/3731793413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cfae0 0x55dadd4e09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.269+0000 7f1963e00640 1 -- 172.21.15.40:0/3731793413 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dadd4c68a0 con 0x55dadd4d4c50 2024-04-27T00:01:39.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.269+0000 7f1963e00640 1 --2- 172.21.15.40:0/3731793413 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4c9b60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f195c000f80 tx=0x7f195c009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a62022ac77204bd server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.270+0000 7f1953e00640 1 -- 172.21.15.40:0/3731793413 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c0b5020 con 0x55dadd4d4c50 2024-04-27T00:01:39.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.270+0000 7f1953e00640 1 -- 172.21.15.40:0/3731793413 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f195c09ca60 con 0x55dadd4d4c50 2024-04-27T00:01:39.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.270+0000 7f1953e00640 1 -- 172.21.15.40:0/3731793413 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c0990d0 con 0x55dadd4d4c50 2024-04-27T00:01:39.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.270+0000 7f196d424740 1 -- 172.21.15.40:0/3731793413 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 msgr2=0x55dadd4c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.270+0000 7f196d424740 1 --2- 172.21.15.40:0/3731793413 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4c9b60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f195c000f80 tx=0x7f195c009bf0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.270+0000 7f196d424740 1 -- 172.21.15.40:0/3731793413 shutdown_connections 2024-04-27T00:01:39.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.270+0000 7f196d424740 1 --2- 172.21.15.40:0/3731793413 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4c9b60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.270+0000 7f196d424740 1 --2- 172.21.15.40:0/3731793413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cfae0 0x55dadd4e09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.270+0000 7f196d424740 1 --2- 172.21.15.40:0/3731793413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.270+0000 7f196d424740 1 -- 172.21.15.40:0/3731793413 >> 172.21.15.40:0/3731793413 conn(0x55dadd463d10 msgr2=0x55dadd4e4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.270+0000 7f196d424740 1 -- 172.21.15.40:0/3731793413 shutdown_connections 2024-04-27T00:01:39.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.271+0000 7f196d424740 1 -- 172.21.15.40:0/3731793413 wait complete. 2024-04-27T00:01:39.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.271+0000 7f196d424740 1 -- start start 2024-04-27T00:01:39.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.271+0000 7f196d424740 1 Processor -- start 2024-04-27T00:01:39.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f196d424740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4dc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f196d424740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cfae0 0x55dadd4dcc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f196d424740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4dfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f196d424740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dadd4e7100 con 0x55dadd4cfae0 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f196d424740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55dadd4e6a30 con 0x55dadd4d4c50 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f196d424740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dadd4e77d0 con 0x55dadd4cf0f0 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f196d424740 10 client.0 _ll_register_callbacks cb 0x7f1954009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f195be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cfae0 0x55dadd4dcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f195be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cfae0 0x55dadd4dcc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52176/0 (socket says 172.21.15.40:52176) 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f195be00640 1 -- 172.21.15.40:0/2422155049 learned_addr learned my addr 172.21.15.40:0/2422155049 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f1963e00640 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4dfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f1963400640 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4dc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f1963e00640 1 -- 172.21.15.40:0/2422155049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 msgr2=0x55dadd4dc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f1963e00640 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4dc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f1963e00640 1 -- 172.21.15.40:0/2422155049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cfae0 msgr2=0x55dadd4dcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f1963e00640 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cfae0 0x55dadd4dcc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f1963e00640 1 -- 172.21.15.40:0/2422155049 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dadd4e0270 con 0x55dadd4d4c50 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f195be00640 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cfae0 0x55dadd4dcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.272+0000 7f1963400640 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4dc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.273+0000 7f1963e00640 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4dfc80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f195c098ec0 tx=0x7f195c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bd24a28a6bdcfa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.273+0000 7f1951600640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.273+0000 7f1951600640 1 -- 172.21.15.40:0/2422155049 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c0b5030 con 0x55dadd4d4c50 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.273+0000 7f196d424740 1 -- 172.21.15.40:0/2422155049 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f195c09ca60 con 0x55dadd4d4c50 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.273+0000 7f1951600640 1 -- 172.21.15.40:0/2422155049 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f195c0ce4e0 con 0x55dadd4d4c50 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.273+0000 7f1951600640 1 -- 172.21.15.40:0/2422155049 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f195c0990d0 con 0x55dadd4d4c50 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.273+0000 7f196d424740 10 client.25337 fetch_fsmap learned FSMap version 23 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.273+0000 7f196d424740 1 -- 172.21.15.40:0/2422155049 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55dadd54c410 con 0x55dadd4d4c50 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.273+0000 7f1951600640 1 -- 172.21.15.40:0/2422155049 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f195c09ca60 con 0x55dadd4d4c50 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.273+0000 7f1951600640 1 -- 172.21.15.40:0/2422155049 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f195c01f020 con 0x55dadd4d4c50 2024-04-27T00:01:39.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.273+0000 7f196d424740 10 client.25337 fetch_fsmap finished waiting for FSMap version 23 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.274+0000 7f196d424740 10 client.25337 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.274+0000 7f196d424740 20 client.25337 populate_metadata read hostname 'smithi040' 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.274+0000 7f196d424740 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55dadd550a80 0x55dadd54f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.274+0000 7f196d424740 4 client.25337 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.274+0000 7f196d424740 1 -- 172.21.15.40:0/2422155049 --> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] -- command(tid 0: {"prefix": "status"}) -- 0x55dadd43b080 con 0x55dadd550a80 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.274+0000 7f1963400640 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55dadd550a80 0x55dadd54f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f1963400640 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55dadd550a80 0x55dadd54f790 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=94f2c199c74028d8 server_cookie=7a032125dedfcc62 in_seq=0 out_seq=0 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f1951600640 10 client.25337 ms_handle_connect on v2:172.21.15.103:6838/1527392010 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f1951600640 1 -- 172.21.15.40:0/2422155049 <== mds.2 v2:172.21.15.103:6838/1527392010 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55dadd43b080 con 0x55dadd550a80 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f1951600640 10 client.25337 handle_command_reply: tid=0 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f196d424740 1 client.25337 shutdown 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f196d424740 1 -- 172.21.15.40:0/2422155049 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 msgr2=0x55dadd4dfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f196d424740 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4dfc80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f195c098ec0 tx=0x7f195c009f90 comp rx=0 tx=0).stop 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f196d424740 1 -- 172.21.15.40:0/2422155049 shutdown_connections 2024-04-27T00:01:39.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f196d424740 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.103:6838/1527392010,v1:172.21.15.103:6839/1527392010] conn(0x55dadd550a80 0x55dadd54f790 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f196d424740 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dadd4d4c50 0x55dadd4dfc80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f196d424740 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dadd4cfae0 0x55dadd4dcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f196d424740 1 --2- 172.21.15.40:0/2422155049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dadd4cf0f0 0x55dadd4dc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f1951600640 0 client.25337 ms_handle_reset on v2:172.21.15.103:6838/1527392010 2024-04-27T00:01:39.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f196d424740 1 -- 172.21.15.40:0/2422155049 >> 172.21.15.40:0/2422155049 conn(0x55dadd463d10 msgr2=0x55dadd4c9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f196d424740 1 -- 172.21.15.40:0/2422155049 shutdown_connections 2024-04-27T00:01:39.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.275+0000 7f196d424740 1 -- 172.21.15.40:0/2422155049 wait complete. 2024-04-27T00:01:39.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.276+0000 7f196d424740 20 client.25337 trim_cache size 0 max 16384 2024-04-27T00:01:39.282 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-27T00:01:39.283 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "78cff6ea-0427-11ef-bc93-c7b262605968", 2024-04-27T00:01:39.283 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-04-27T00:01:39.283 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24559, 2024-04-27T00:01:39.283 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-27T00:01:39.283 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-27T00:01:39.283 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-27T00:01:39.283 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 134.92181080899999, 2024-04-27T00:01:39.283 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 23, 2024-04-27T00:01:39.283 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 92, 2024-04-27T00:01:39.283 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-27T00:01:39.283 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 185.91595592300001 2024-04-27T00:01:39.283 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-27T00:01:39.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.276+0000 7f196a600640 1 -- 172.21.15.40:0/2984412482 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f19440a3cc0 msgr2=0x7f19440a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.276+0000 7f196a600640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f19440a3cc0 0x7f19440a6180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f194c031500 tx=0x7f194c000bc0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.276+0000 7f196a600640 1 -- 172.21.15.40:0/2984412482 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f196410ab60 msgr2=0x7f19641a6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.276+0000 7f196a600640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f196410ab60 0x7f19641a6780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f195c00efd0 tx=0x7f195c00c6a0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.276+0000 7f196a600640 1 -- 172.21.15.40:0/2984412482 shutdown_connections 2024-04-27T00:01:39.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.276+0000 7f196a600640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f196410ab60 0x7f19641a6780 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.276+0000 7f196a600640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f19440a3cc0 0x7f19440a6180 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.276+0000 7f196a600640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f196410a1a0 0x7f196419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.276+0000 7f196a600640 1 --2- 172.21.15.40:0/2984412482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1964108fb0 0x7f196419f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.276+0000 7f196a600640 1 -- 172.21.15.40:0/2984412482 >> 172.21.15.40:0/2984412482 conn(0x7f1964077f40 msgr2=0x7f196410d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.276+0000 7f196a600640 1 -- 172.21.15.40:0/2984412482 shutdown_connections 2024-04-27T00:01:39.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.277+0000 7f196a600640 1 -- 172.21.15.40:0/2984412482 wait complete. 2024-04-27T00:01:39.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:39 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2389049280' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:39.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:39 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:01:39.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:39 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:01:39.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:39 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/705816707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:39.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:39 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:39.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:39 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2000984794' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:39.296 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-27T00:01:39.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:39 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2389049280' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:39.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:39 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:01:39.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:39 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:01:39.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:39 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/705816707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:39.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:39 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:39.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:39 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2000984794' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:39.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:39 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2389049280' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:39.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:01:39.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:01:39.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:39 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/705816707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:39.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:39 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:01:39.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:39 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2000984794' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:39.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.502+0000 7f90b6200640 1 -- 172.21.15.40:0/3167909566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b0106540 msgr2=0x7f90b010cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.502+0000 7f90b6200640 1 --2- 172.21.15.40:0/3167909566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b0106540 0x7f90b010cdd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009ca0 tx=0x7f90a802a8b0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.502+0000 7f90b6200640 1 -- 172.21.15.40:0/3167909566 shutdown_connections 2024-04-27T00:01:39.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.502+0000 7f90b6200640 1 --2- 172.21.15.40:0/3167909566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b0106540 0x7f90b010cdd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.502+0000 7f90b6200640 1 --2- 172.21.15.40:0/3167909566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b0105b80 0x7f90b0106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.502+0000 7f90b6200640 1 --2- 172.21.15.40:0/3167909566 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f90b0104990 0x7f90b0104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.502+0000 7f90b6200640 1 -- 172.21.15.40:0/3167909566 >> 172.21.15.40:0/3167909566 conn(0x7f90b00fff00 msgr2=0x7f90b0102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.502+0000 7f90b6200640 1 -- 172.21.15.40:0/3167909566 shutdown_connections 2024-04-27T00:01:39.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.502+0000 7f90b6200640 1 -- 172.21.15.40:0/3167909566 wait complete. 2024-04-27T00:01:39.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.503+0000 7f90b6200640 1 Processor -- start 2024-04-27T00:01:39.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.504+0000 7f90b6200640 1 -- start start 2024-04-27T00:01:39.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.504+0000 7f90b6200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b0104990 0x7f90b019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.504+0000 7f90b6200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b0105b80 0x7f90b019b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.504+0000 7f90b6200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f90b0106540 0x7f90b01a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.504+0000 7f90b6200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f90b01104f0 con 0x7f90b0104990 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.504+0000 7f90b6200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f90b0110370 con 0x7f90b0106540 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.504+0000 7f90b6200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f90b0110670 con 0x7f90b0105b80 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.504+0000 7f90b4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b0104990 0x7f90b019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.504+0000 7f90b4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b0104990 0x7f90b019ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52188/0 (socket says 172.21.15.40:52188) 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.504+0000 7f90b4e00640 1 -- 172.21.15.40:0/2310788358 learned_addr learned my addr 172.21.15.40:0/2310788358 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.504+0000 7f90b5800640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f90b0106540 0x7f90b01a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.504+0000 7f90afe00640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b0105b80 0x7f90b019b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.505+0000 7f90b4e00640 1 -- 172.21.15.40:0/2310788358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b0105b80 msgr2=0x7f90b019b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.505+0000 7f90b4e00640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b0105b80 0x7f90b019b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.505+0000 7f90b4e00640 1 -- 172.21.15.40:0/2310788358 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f90b0106540 msgr2=0x7f90b01a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.505+0000 7f90b4e00640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f90b0106540 0x7f90b01a20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.505+0000 7f90b4e00640 1 -- 172.21.15.40:0/2310788358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b01a2720 con 0x7f90b0104990 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.505+0000 7f90b5800640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f90b0106540 0x7f90b01a20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:39.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.505+0000 7f90b4e00640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b0104990 0x7f90b019ab00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f90a400e9c0 tx=0x7f90a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.505+0000 7f90ad600640 1 -- 172.21.15.40:0/2310788358 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a400cd80 con 0x7f90b0104990 2024-04-27T00:01:39.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.505+0000 7f90ad600640 1 -- 172.21.15.40:0/2310788358 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90a40035f0 con 0x7f90b0104990 2024-04-27T00:01:39.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.505+0000 7f90b6200640 1 -- 172.21.15.40:0/2310788358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b01a2a10 con 0x7f90b0104990 2024-04-27T00:01:39.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.505+0000 7f90ad600640 1 -- 172.21.15.40:0/2310788358 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a4010690 con 0x7f90b0104990 2024-04-27T00:01:39.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.506+0000 7f90b6200640 1 -- 172.21.15.40:0/2310788358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b01a2fb0 con 0x7f90b0104990 2024-04-27T00:01:39.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.509+0000 7f90ba56a740 1 Processor -- start 2024-04-27T00:01:39.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.509+0000 7f90ba56a740 1 -- start start 2024-04-27T00:01:39.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.509+0000 7f90ba56a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe62c0 0x55b03cfe66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.509+0000 7f90ba56a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe6cb0 0x55b03cfeb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.509+0000 7f90ba56a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 0x55b03cfee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.509+0000 7f90ba56a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b03cf4a600 con 0x55b03cfe6cb0 2024-04-27T00:01:39.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.509+0000 7f90ba56a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55b03cee7100 con 0x55b03cfebe20 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.509+0000 7f90ba56a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b03cf465c0 con 0x55b03cfe62c0 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90ad600640 1 -- 172.21.15.40:0/2310788358 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f90a40037a0 con 0x7f90b0104990 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90afe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 0x55b03cfee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90afe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 0x55b03cfee2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49524/0 (socket says 172.21.15.40:49524) 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90afe00640 1 -- 172.21.15.40:0/1298127552 learned_addr learned my addr 172.21.15.40:0/1298127552 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90b5800640 1 --2- 172.21.15.40:0/1298127552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe62c0 0x55b03cfe66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90b4e00640 1 --2- 172.21.15.40:0/1298127552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe6cb0 0x55b03cfeb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90afe00640 1 -- 172.21.15.40:0/1298127552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe62c0 msgr2=0x55b03cfe66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90afe00640 1 --2- 172.21.15.40:0/1298127552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe62c0 0x55b03cfe66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90afe00640 1 -- 172.21.15.40:0/1298127552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe6cb0 msgr2=0x55b03cfeb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90afe00640 1 --2- 172.21.15.40:0/1298127552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe6cb0 0x55b03cfeb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90afe00640 1 -- 172.21.15.40:0/1298127552 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b03cc5c2d0 con 0x55b03cfebe20 2024-04-27T00:01:39.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90b4e00640 1 --2- 172.21.15.40:0/1298127552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe6cb0 0x55b03cfeb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T00:01:39.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.510+0000 7f90afe00640 1 --2- 172.21.15.40:0/1298127552 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 0x55b03cfee2b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f909c009c90 tx=0x7f909c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35095ea4a02442c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.511+0000 7f90a2000640 1 -- 172.21.15.40:0/1298127552 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c01e070 con 0x55b03cfebe20 2024-04-27T00:01:39.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.511+0000 7f90a2000640 1 -- 172.21.15.40:0/1298127552 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f909c019ae0 con 0x55b03cfebe20 2024-04-27T00:01:39.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.511+0000 7f90a2000640 1 -- 172.21.15.40:0/1298127552 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c00ea40 con 0x55b03cfebe20 2024-04-27T00:01:39.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.512+0000 7f90ba56a740 1 -- 172.21.15.40:0/1298127552 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 msgr2=0x55b03cfee2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.512+0000 7f90ba56a740 1 --2- 172.21.15.40:0/1298127552 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 0x55b03cfee2b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f909c009c90 tx=0x7f909c011810 comp rx=0 tx=0).stop 2024-04-27T00:01:39.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.512+0000 7f90ba56a740 1 -- 172.21.15.40:0/1298127552 shutdown_connections 2024-04-27T00:01:39.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.512+0000 7f90ba56a740 1 --2- 172.21.15.40:0/1298127552 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 0x55b03cfee2b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.512+0000 7f90ba56a740 1 --2- 172.21.15.40:0/1298127552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe6cb0 0x55b03cfeb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.512+0000 7f90ba56a740 1 --2- 172.21.15.40:0/1298127552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe62c0 0x55b03cfe66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.512+0000 7f90ba56a740 1 -- 172.21.15.40:0/1298127552 >> 172.21.15.40:0/1298127552 conn(0x55b03cf7ad10 msgr2=0x55b03cfe1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.512+0000 7f90ba56a740 1 -- 172.21.15.40:0/1298127552 shutdown_connections 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.512+0000 7f90ad600640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f90980a3cc0 0x7f90980a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.512+0000 7f90ba56a740 1 -- 172.21.15.40:0/1298127552 wait complete. 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.512+0000 7f90ad600640 1 -- 172.21.15.40:0/2310788358 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f90a40d0c10 con 0x7f90b0104990 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.512+0000 7f90afe00640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f90980a3cc0 0x7f90980a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.513+0000 7f90ba56a740 1 -- start start 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.513+0000 7f90ba56a740 1 Processor -- start 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.513+0000 7f90afe00640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f90980a3cc0 0x7f90980a6180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f909c00e4d0 tx=0x7f909c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.513+0000 7f90ba56a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe62c0 0x55b03cff06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.513+0000 7f90ba56a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe6cb0 0x55b03cff0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.513+0000 7f90ba56a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 0x55b03cff7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.513+0000 7f90ba56a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b03cfeedd0 con 0x55b03cfe6cb0 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.513+0000 7f90ba56a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55b03ced1f70 con 0x55b03cfebe20 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.513+0000 7f90ba56a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b03cfeef50 con 0x55b03cfe62c0 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.514+0000 7f90ba56a740 10 client.0 _ll_register_callbacks cb 0x55b03cf40480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.514+0000 7f90afe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe6cb0 0x55b03cff0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.514+0000 7f90b5800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 0x55b03cff7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.514+0000 7f90b5800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 0x55b03cff7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49540/0 (socket says 172.21.15.40:49540) 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.514+0000 7f90afe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe6cb0 0x55b03cff0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52208/0 (socket says 172.21.15.40:52208) 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.514+0000 7f90b5800640 1 -- 172.21.15.40:0/3437552759 learned_addr learned my addr 172.21.15.40:0/3437552759 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.514+0000 7f90b4e00640 1 --2- 172.21.15.40:0/3437552759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe62c0 0x55b03cff06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.514+0000 7f90b5800640 1 -- 172.21.15.40:0/3437552759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe62c0 msgr2=0x55b03cff06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.514+0000 7f90b5800640 1 --2- 172.21.15.40:0/3437552759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe62c0 0x55b03cff06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.514+0000 7f90b5800640 1 -- 172.21.15.40:0/3437552759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe6cb0 msgr2=0x55b03cff0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.514+0000 7f90b5800640 1 --2- 172.21.15.40:0/3437552759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe6cb0 0x55b03cff0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.514+0000 7f90b5800640 1 -- 172.21.15.40:0/3437552759 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b03cd266b0 con 0x55b03cfebe20 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.515+0000 7f90b5800640 1 --2- 172.21.15.40:0/3437552759 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 0x55b03cff7c40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009ca0 tx=0x7f90a8030df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6f07110cb5ed8b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.515+0000 7f9096a00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.515+0000 7f9096a00640 1 -- 172.21.15.40:0/3437552759 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8003cc0 con 0x55b03cfebe20 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.515+0000 7f9096a00640 1 -- 172.21.15.40:0/3437552759 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f90a8004040 con 0x55b03cfebe20 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.515+0000 7f90ba56a740 1 -- 172.21.15.40:0/3437552759 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f909c019ae0 con 0x55b03cfebe20 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.515+0000 7f9096a00640 1 -- 172.21.15.40:0/3437552759 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f90a8003e60 con 0x55b03cfebe20 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.515+0000 7f90ba56a740 10 client.25349 fetch_fsmap learned FSMap version 23 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.515+0000 7f90ba56a740 1 -- 172.21.15.40:0/3437552759 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b03cffac00 con 0x55b03cfebe20 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.516+0000 7f9096a00640 1 -- 172.21.15.40:0/3437552759 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f909c019ae0 con 0x55b03cfebe20 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.516+0000 7f9096a00640 1 -- 172.21.15.40:0/3437552759 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f90a8005140 con 0x55b03cfebe20 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.516+0000 7f90ba56a740 10 client.25349 fetch_fsmap finished waiting for FSMap version 23 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.516+0000 7f90ba56a740 10 client.25349 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-27T00:01:39.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.516+0000 7f90ba56a740 20 client.25349 populate_metadata read hostname 'smithi040' 2024-04-27T00:01:39.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.516+0000 7f90ba56a740 1 --2- 172.21.15.40:0/3437552759 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x55b03cffd340 0x55b03cffe7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.516+0000 7f90ba56a740 4 client.25349 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:01:39.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.516+0000 7f90ba56a740 1 -- 172.21.15.40:0/3437552759 --> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b03cf58210 con 0x55b03cffd340 2024-04-27T00:01:39.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.516+0000 7f90b4e00640 1 --2- 172.21.15.40:0/3437552759 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x55b03cffd340 0x55b03cffe7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.517+0000 7f90b4e00640 1 --2- 172.21.15.40:0/3437552759 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x55b03cffd340 0x55b03cffe7b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fb5b01f8ba54d0f1 server_cookie=c92834b1b89a33a in_seq=0 out_seq=0 2024-04-27T00:01:39.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.517+0000 7f9096a00640 10 client.25349 ms_handle_connect on v2:172.21.15.40:6838/2205335602 2024-04-27T00:01:39.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f9096a00640 1 -- 172.21.15.40:0/3437552759 <== mds.1 v2:172.21.15.40:6838/2205335602 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b03cf58210 con 0x55b03cffd340 2024-04-27T00:01:39.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f9096a00640 10 client.25349 handle_command_reply: tid=0 2024-04-27T00:01:39.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f90ba56a740 1 client.25349 shutdown 2024-04-27T00:01:39.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f90ba56a740 1 -- 172.21.15.40:0/3437552759 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 msgr2=0x55b03cff7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f90ba56a740 1 --2- 172.21.15.40:0/3437552759 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 0x55b03cff7c40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009ca0 tx=0x7f90a8030df0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f90ba56a740 1 -- 172.21.15.40:0/3437552759 shutdown_connections 2024-04-27T00:01:39.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f90ba56a740 1 --2- 172.21.15.40:0/3437552759 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x55b03cffd340 0x55b03cffe7b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f90ba56a740 1 --2- 172.21.15.40:0/3437552759 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfebe20 0x55b03cff7c40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f90ba56a740 1 --2- 172.21.15.40:0/3437552759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe6cb0 0x55b03cff0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f90ba56a740 1 --2- 172.21.15.40:0/3437552759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe62c0 0x55b03cff06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f9096a00640 0 client.25349 ms_handle_reset on v2:172.21.15.40:6838/2205335602 2024-04-27T00:01:39.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f90ba56a740 1 -- 172.21.15.40:0/3437552759 >> 172.21.15.40:0/3437552759 conn(0x55b03cf7ad10 msgr2=0x55b03cfe0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f90ba56a740 1 -- 172.21.15.40:0/3437552759 shutdown_connections 2024-04-27T00:01:39.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.518+0000 7f90ba56a740 1 -- 172.21.15.40:0/3437552759 wait complete. 2024-04-27T00:01:39.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.519+0000 7f90ba56a740 20 client.25349 trim_cache size 0 max 16384 2024-04-27T00:01:39.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.538+0000 7f90ba56a740 1 Processor -- start 2024-04-27T00:01:39.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.539+0000 7f90ba56a740 1 -- start start 2024-04-27T00:01:39.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.539+0000 7f90ba56a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe62c0 0x55b03cfdcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.539+0000 7f90ba56a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe6cb0 0x55b03cff7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.539+0000 7f90ba56a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfebe20 0x55b03cffc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.539+0000 7f90ba56a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b03cffcaf0 con 0x55b03cfe62c0 2024-04-27T00:01:39.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.539+0000 7f90ba56a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55b03cffd1c0 con 0x55b03cfe6cb0 2024-04-27T00:01:39.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.539+0000 7f90ba56a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b03cffd890 con 0x55b03cfebe20 2024-04-27T00:01:39.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.539+0000 7f90afe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe6cb0 0x55b03cff7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.539+0000 7f90afe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe6cb0 0x55b03cff7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49544/0 (socket says 172.21.15.40:49544) 2024-04-27T00:01:39.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.539+0000 7f90afe00640 1 -- 172.21.15.40:0/3158018991 learned_addr learned my addr 172.21.15.40:0/3158018991 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.539+0000 7f90b4e00640 1 --2- 172.21.15.40:0/3158018991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe62c0 0x55b03cfdcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.539+0000 7f90b5800640 1 --2- 172.21.15.40:0/3158018991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfebe20 0x55b03cffc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.540+0000 7f90afe00640 1 -- 172.21.15.40:0/3158018991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfebe20 msgr2=0x55b03cffc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.540+0000 7f90afe00640 1 --2- 172.21.15.40:0/3158018991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfebe20 0x55b03cffc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.540+0000 7f90afe00640 1 -- 172.21.15.40:0/3158018991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe62c0 msgr2=0x55b03cfdcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.540+0000 7f90afe00640 1 --2- 172.21.15.40:0/3158018991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe62c0 0x55b03cfdcdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.540+0000 7f90afe00640 1 -- 172.21.15.40:0/3158018991 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b03cfe7090 con 0x55b03cfe6cb0 2024-04-27T00:01:39.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.540+0000 7f90b4e00640 1 --2- 172.21.15.40:0/3158018991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe62c0 0x55b03cfdcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:39.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.540+0000 7f90afe00640 1 --2- 172.21.15.40:0/3158018991 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe6cb0 0x55b03cff7c00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f909c009820 tx=0x7f909c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adf3b6c17efa980e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.540+0000 7f90a1600640 1 -- 172.21.15.40:0/3158018991 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c01e040 con 0x55b03cfe6cb0 2024-04-27T00:01:39.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.540+0000 7f90a1600640 1 -- 172.21.15.40:0/3158018991 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f909c029040 con 0x55b03cfe6cb0 2024-04-27T00:01:39.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.540+0000 7f90a1600640 1 -- 172.21.15.40:0/3158018991 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c024470 con 0x55b03cfe6cb0 2024-04-27T00:01:39.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.541+0000 7f90ba56a740 1 -- 172.21.15.40:0/3158018991 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe6cb0 msgr2=0x55b03cff7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.541+0000 7f90ba56a740 1 --2- 172.21.15.40:0/3158018991 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe6cb0 0x55b03cff7c00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f909c009820 tx=0x7f909c005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:39.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.541+0000 7f90ba56a740 1 -- 172.21.15.40:0/3158018991 shutdown_connections 2024-04-27T00:01:39.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.541+0000 7f90ba56a740 1 --2- 172.21.15.40:0/3158018991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfebe20 0x55b03cffc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.541+0000 7f90ba56a740 1 --2- 172.21.15.40:0/3158018991 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe6cb0 0x55b03cff7c00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.541+0000 7f90ba56a740 1 --2- 172.21.15.40:0/3158018991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfe62c0 0x55b03cfdcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.541+0000 7f90ba56a740 1 -- 172.21.15.40:0/3158018991 >> 172.21.15.40:0/3158018991 conn(0x55b03cf7ad10 msgr2=0x55b03cfefce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.541+0000 7f90ba56a740 1 -- 172.21.15.40:0/3158018991 shutdown_connections 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.541+0000 7f90ba56a740 1 -- 172.21.15.40:0/3158018991 wait complete. 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.542+0000 7f90ba56a740 1 -- start start 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.542+0000 7f90ba56a740 1 Processor -- start 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.542+0000 7f90ba56a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe62c0 0x55b03cff5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.542+0000 7f90ba56a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe6cb0 0x55b03d05f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.542+0000 7f90ba56a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfebe20 0x55b03d061820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.542+0000 7f90ba56a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b03cffe630 con 0x55b03cfebe20 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.542+0000 7f90ba56a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55b03cffdf60 con 0x55b03cfe62c0 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.542+0000 7f90ba56a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b03cffed00 con 0x55b03cfe6cb0 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.542+0000 7f90ba56a740 10 client.0 _ll_register_callbacks cb 0x7f90a8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:01:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f90b4e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe62c0 0x55b03cff5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f90b5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfebe20 0x55b03d061820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f90b4e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe62c0 0x55b03cff5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49546/0 (socket says 172.21.15.40:49546) 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f90b4e00640 1 -- 172.21.15.40:0/1891783187 learned_addr learned my addr 172.21.15.40:0/1891783187 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f90afe00640 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe6cb0 0x55b03d05f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f90b4e00640 1 -- 172.21.15.40:0/1891783187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe6cb0 msgr2=0x55b03d05f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f90b4e00640 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe6cb0 0x55b03d05f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f90b4e00640 1 -- 172.21.15.40:0/1891783187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfebe20 msgr2=0x55b03d061820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f90b4e00640 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfebe20 0x55b03d061820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f90b4e00640 1 -- 172.21.15.40:0/1891783187 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b03d061ec0 con 0x55b03cfe62c0 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f90b5800640 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfebe20 0x55b03d061820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f90b4e00640 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe62c0 0x55b03cff5b50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f90a400cfd0 tx=0x7f90a40bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89a956c7725d84eb server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.543+0000 7f9096a00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:01:39.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.544+0000 7f9096a00640 1 -- 172.21.15.40:0/1891783187 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a40a0da0 con 0x55b03cfe62c0 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.544+0000 7f9096a00640 1 -- 172.21.15.40:0/1891783187 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f90a4099060 con 0x55b03cfe62c0 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.544+0000 7f90ba56a740 1 -- 172.21.15.40:0/1891783187 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f909c029040 con 0x55b03cfe62c0 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.544+0000 7f9096a00640 1 -- 172.21.15.40:0/1891783187 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f90a40a5660 con 0x55b03cfe62c0 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.544+0000 7f90ba56a740 10 client.25355 fetch_fsmap learned FSMap version 23 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.544+0000 7f90ba56a740 1 -- 172.21.15.40:0/1891783187 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b03d063290 con 0x55b03cfe62c0 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.544+0000 7f9096a00640 1 -- 172.21.15.40:0/1891783187 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f909c029040 con 0x55b03cfe62c0 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.544+0000 7f9096a00640 1 -- 172.21.15.40:0/1891783187 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f90a40bbbb0 con 0x55b03cfe62c0 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.544+0000 7f90ba56a740 10 client.25355 fetch_fsmap finished waiting for FSMap version 23 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.545+0000 7f90ba56a740 10 client.25355 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.545+0000 7f90ba56a740 20 client.25355 populate_metadata read hostname 'smithi040' 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.545+0000 7f90ba56a740 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x55b03d067e60 0x55b03d0642a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.545+0000 7f90ba56a740 4 client.25355 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.545+0000 7f90ba56a740 1 -- 172.21.15.40:0/1891783187 --> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] -- command(tid 0: {"prefix": "status"}) -- 0x55b03cf52080 con 0x55b03d067e60 2024-04-27T00:01:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.545+0000 7f90afe00640 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x55b03d067e60 0x55b03d0642a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f90afe00640 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x55b03d067e60 0x55b03d0642a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b8c29a017d1213a4 server_cookie=ce071e9932bb8db in_seq=0 out_seq=0 2024-04-27T00:01:39.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f9096a00640 10 client.25355 ms_handle_connect on v2:172.21.15.40:6838/2205335602 2024-04-27T00:01:39.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f9096a00640 1 -- 172.21.15.40:0/1891783187 <== mds.1 v2:172.21.15.40:6838/2205335602 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b03cf52080 con 0x55b03d067e60 2024-04-27T00:01:39.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f9096a00640 10 client.25355 handle_command_reply: tid=0 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f90ba56a740 1 client.25355 shutdown 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f90ba56a740 1 -- 172.21.15.40:0/1891783187 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe62c0 msgr2=0x55b03cff5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f90ba56a740 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe62c0 0x55b03cff5b50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f90a400cfd0 tx=0x7f90a40bd040 comp rx=0 tx=0).stop 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f90ba56a740 1 -- 172.21.15.40:0/1891783187 shutdown_connections 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f90ba56a740 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.40:6838/2205335602,v1:172.21.15.40:6839/2205335602] conn(0x55b03d067e60 0x55b03d0642a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f90ba56a740 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b03cfebe20 0x55b03d061820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f90ba56a740 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b03cfe6cb0 0x55b03d05f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f9096a00640 0 client.25355 ms_handle_reset on v2:172.21.15.40:6838/2205335602 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f90ba56a740 1 --2- 172.21.15.40:0/1891783187 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b03cfe62c0 0x55b03cff5b50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f90ba56a740 1 -- 172.21.15.40:0/1891783187 >> 172.21.15.40:0/1891783187 conn(0x55b03cf7ad10 msgr2=0x55b03cfe8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f90ba56a740 1 -- 172.21.15.40:0/1891783187 shutdown_connections 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.546+0000 7f90ba56a740 1 -- 172.21.15.40:0/1891783187 wait complete. 2024-04-27T00:01:39.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.547+0000 7f90ba56a740 20 client.25355 trim_cache size 0 max 16384 2024-04-27T00:01:39.553 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-27T00:01:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "78cff6ea-0427-11ef-bc93-c7b262605968", 2024-04-27T00:01:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-04-27T00:01:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24560, 2024-04-27T00:01:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-27T00:01:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-27T00:01:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-27T00:01:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 136.20706438600001, 2024-04-27T00:01:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 23, 2024-04-27T00:01:39.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 92, 2024-04-27T00:01:39.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-27T00:01:39.554 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 182.15490579300001 2024-04-27T00:01:39.554 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-27T00:01:39.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.547+0000 7f90b6200640 1 -- 172.21.15.40:0/2310788358 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f90980a3cc0 msgr2=0x7f90980a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.547+0000 7f90b6200640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f90980a3cc0 0x7f90980a6180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f909c00e4d0 tx=0x7f909c000bc0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.547+0000 7f90b6200640 1 -- 172.21.15.40:0/2310788358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b0104990 msgr2=0x7f90b019ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.547+0000 7f90b6200640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b0104990 0x7f90b019ab00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f90a400e9c0 tx=0x7f90a400ee80 comp rx=0 tx=0).stop 2024-04-27T00:01:39.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.547+0000 7f90b6200640 1 -- 172.21.15.40:0/2310788358 shutdown_connections 2024-04-27T00:01:39.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.547+0000 7f90b6200640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f90b0106540 0x7f90b01a20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.547+0000 7f90b6200640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f90980a3cc0 0x7f90980a6180 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.547+0000 7f90b6200640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b0105b80 0x7f90b019b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.547+0000 7f90b6200640 1 --2- 172.21.15.40:0/2310788358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b0104990 0x7f90b019ab00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.547+0000 7f90b6200640 1 -- 172.21.15.40:0/2310788358 >> 172.21.15.40:0/2310788358 conn(0x7f90b00fff00 msgr2=0x7f90b0108e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.548+0000 7f90b6200640 1 -- 172.21.15.40:0/2310788358 shutdown_connections 2024-04-27T00:01:39.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.548+0000 7f90b6200640 1 -- 172.21.15.40:0/2310788358 wait complete. 2024-04-27T00:01:39.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-27T00:01:39.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.764+0000 7fc649400640 1 -- 172.21.15.40:0/1220563272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc644076970 msgr2=0x7fc644074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.764+0000 7fc649400640 1 --2- 172.21.15.40:0/1220563272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc644076970 0x7fc644074dc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc634009930 tx=0x7fc63402a420 comp rx=0 tx=0).stop 2024-04-27T00:01:39.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.764+0000 7fc649400640 1 -- 172.21.15.40:0/1220563272 shutdown_connections 2024-04-27T00:01:39.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.764+0000 7fc649400640 1 --2- 172.21.15.40:0/1220563272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc64410ed40 0x7fc644111130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.764+0000 7fc649400640 1 --2- 172.21.15.40:0/1220563272 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc644075390 0x7fc644075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.764+0000 7fc649400640 1 --2- 172.21.15.40:0/1220563272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc644076970 0x7fc644074dc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.764+0000 7fc649400640 1 -- 172.21.15.40:0/1220563272 >> 172.21.15.40:0/1220563272 conn(0x7fc6440fdce0 msgr2=0x7fc644100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.765+0000 7fc649400640 1 -- 172.21.15.40:0/1220563272 shutdown_connections 2024-04-27T00:01:39.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.765+0000 7fc649400640 1 -- 172.21.15.40:0/1220563272 wait complete. 2024-04-27T00:01:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.766+0000 7fc649400640 1 Processor -- start 2024-04-27T00:01:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.766+0000 7fc649400640 1 -- start start 2024-04-27T00:01:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.766+0000 7fc649400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc644075390 0x7fc64419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.766+0000 7fc649400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc644076970 0x7fc64419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.766+0000 7fc649400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64410ed40 0x7fc6441a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.766+0000 7fc649400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc644114890 con 0x7fc64410ed40 2024-04-27T00:01:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.766+0000 7fc649400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fc644114710 con 0x7fc644076970 2024-04-27T00:01:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.766+0000 7fc649400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc644114a10 con 0x7fc644075390 2024-04-27T00:01:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.766+0000 7fc643e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc644075390 0x7fc64419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.766+0000 7fc63be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc644076970 0x7fc64419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.766+0000 7fc643e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc644075390 0x7fc64419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48048/0 (socket says 172.21.15.40:48048) 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.766+0000 7fc643e00640 1 -- 172.21.15.40:0/1858329380 learned_addr learned my addr 172.21.15.40:0/1858329380 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.767+0000 7fc648a00640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64410ed40 0x7fc6441a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.767+0000 7fc63be00640 1 -- 172.21.15.40:0/1858329380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc644075390 msgr2=0x7fc64419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.767+0000 7fc63be00640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc644075390 0x7fc64419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.767+0000 7fc63be00640 1 -- 172.21.15.40:0/1858329380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64410ed40 msgr2=0x7fc6441a6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.767+0000 7fc63be00640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64410ed40 0x7fc6441a6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.767+0000 7fc63be00640 1 -- 172.21.15.40:0/1858329380 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6441a6bd0 con 0x7fc644076970 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.767+0000 7fc643e00640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc644075390 0x7fc64419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.767+0000 7fc63be00640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc644076970 0x7fc64419f4f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00e9f0 tx=0x7fc62c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.767+0000 7fc641600640 1 -- 172.21.15.40:0/1858329380 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c00ccc0 con 0x7fc644076970 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.768+0000 7fc641600640 1 -- 172.21.15.40:0/1858329380 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc62c0108e0 con 0x7fc644076970 2024-04-27T00:01:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.768+0000 7fc641600640 1 -- 172.21.15.40:0/1858329380 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c018880 con 0x7fc644076970 2024-04-27T00:01:39.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.768+0000 7fc649400640 1 -- 172.21.15.40:0/1858329380 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6441a6ec0 con 0x7fc644076970 2024-04-27T00:01:39.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.768+0000 7fc649400640 1 -- 172.21.15.40:0/1858329380 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6441a7330 con 0x7fc644076970 2024-04-27T00:01:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.771+0000 7fc64d812740 1 Processor -- start 2024-04-27T00:01:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.771+0000 7fc641600640 1 -- 172.21.15.40:0/1858329380 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc62c010430 con 0x7fc644076970 2024-04-27T00:01:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.771+0000 7fc64d812740 1 -- start start 2024-04-27T00:01:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.771+0000 7fc64d812740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 0x557377b9a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.771+0000 7fc64d812740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377b9f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.771+0000 7fc64d812740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9fe20 0x557377ba22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.771+0000 7fc64d812740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557377afe600 con 0x557377b9a2c0 2024-04-27T00:01:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.771+0000 7fc64d812740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x557377a9b100 con 0x557377b9fe20 2024-04-27T00:01:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.771+0000 7fc64d812740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557377afa5c0 con 0x557377b9acb0 2024-04-27T00:01:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.772+0000 7fc648a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 0x557377b9a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.772+0000 7fc648a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 0x557377b9a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52256/0 (socket says 172.21.15.40:52256) 2024-04-27T00:01:39.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.772+0000 7fc648a00640 1 -- 172.21.15.40:0/1821773243 learned_addr learned my addr 172.21.15.40:0/1821773243 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.772+0000 7fc643e00640 1 --2- 172.21.15.40:0/1821773243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377b9f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.772+0000 7fc63be00640 1 --2- 172.21.15.40:0/1821773243 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9fe20 0x557377ba22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.772+0000 7fc643e00640 1 -- 172.21.15.40:0/1821773243 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9fe20 msgr2=0x557377ba22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.772+0000 7fc643e00640 1 --2- 172.21.15.40:0/1821773243 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9fe20 0x557377ba22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.772+0000 7fc643e00640 1 -- 172.21.15.40:0/1821773243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 msgr2=0x557377b9a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.772+0000 7fc643e00640 1 --2- 172.21.15.40:0/1821773243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 0x557377b9a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.772+0000 7fc643e00640 1 -- 172.21.15.40:0/1821773243 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573778102d0 con 0x557377b9acb0 2024-04-27T00:01:39.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.772+0000 7fc648a00640 1 --2- 172.21.15.40:0/1821773243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 0x557377b9a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:39.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.773+0000 7fc643e00640 1 --2- 172.21.15.40:0/1821773243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377b9f7b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc634000c00 tx=0x7fc634005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d3c8ec2d0906331 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.773+0000 7fc641600640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6280a3cc0 0x7fc6280a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.773+0000 7fc639200640 1 -- 172.21.15.40:0/1821773243 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634035070 con 0x557377b9acb0 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.773+0000 7fc641600640 1 -- 172.21.15.40:0/1858329380 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fc62c0d0b70 con 0x7fc644076970 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.773+0000 7fc639200640 1 -- 172.21.15.40:0/1821773243 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6340318a0 con 0x557377b9acb0 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.773+0000 7fc639200640 1 -- 172.21.15.40:0/1821773243 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634003e40 con 0x557377b9acb0 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.773+0000 7fc648a00640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6280a3cc0 0x7fc6280a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.774+0000 7fc64d812740 1 -- 172.21.15.40:0/1821773243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 msgr2=0x557377b9f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.774+0000 7fc648a00640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6280a3cc0 0x7fc6280a6180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x557377b9b510 tx=0x7fc63c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.774+0000 7fc64d812740 1 --2- 172.21.15.40:0/1821773243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377b9f7b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc634000c00 tx=0x7fc634005d90 comp rx=0 tx=0).stop 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.774+0000 7fc64d812740 1 -- 172.21.15.40:0/1821773243 shutdown_connections 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.774+0000 7fc64d812740 1 --2- 172.21.15.40:0/1821773243 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9fe20 0x557377ba22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.774+0000 7fc64d812740 1 --2- 172.21.15.40:0/1821773243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377b9f7b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.774+0000 7fc64d812740 1 --2- 172.21.15.40:0/1821773243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 0x557377b9a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.774+0000 7fc64d812740 1 -- 172.21.15.40:0/1821773243 >> 172.21.15.40:0/1821773243 conn(0x557377b2ed10 msgr2=0x557377b95d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.774+0000 7fc64d812740 1 -- 172.21.15.40:0/1821773243 shutdown_connections 2024-04-27T00:01:39.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.774+0000 7fc64d812740 1 -- 172.21.15.40:0/1821773243 wait complete. 2024-04-27T00:01:39.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.774+0000 7fc64d812740 1 -- start start 2024-04-27T00:01:39.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc64d812740 1 Processor -- start 2024-04-27T00:01:39.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc64d812740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 0x557377ba46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc64d812740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377ba4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc64d812740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9fe20 0x557377babc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc64d812740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557377ba2d70 con 0x557377b9a2c0 2024-04-27T00:01:39.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc64d812740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x557377a85f70 con 0x557377b9fe20 2024-04-27T00:01:39.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc64d812740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557377ba2ef0 con 0x557377b9acb0 2024-04-27T00:01:39.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc64d812740 10 client.0 _ll_register_callbacks cb 0x557377af4480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc643e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 0x557377ba46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc648a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9fe20 0x557377babc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc63be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377ba4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc643e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 0x557377ba46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52264/0 (socket says 172.21.15.40:52264) 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.775+0000 7fc643e00640 1 -- 172.21.15.40:0/4244479086 learned_addr learned my addr 172.21.15.40:0/4244479086 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.776+0000 7fc63be00640 1 -- 172.21.15.40:0/4244479086 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9fe20 msgr2=0x557377babc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.776+0000 7fc63be00640 1 --2- 172.21.15.40:0/4244479086 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9fe20 0x557377babc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.776+0000 7fc63be00640 1 -- 172.21.15.40:0/4244479086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 msgr2=0x557377ba46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.776+0000 7fc63be00640 1 --2- 172.21.15.40:0/4244479086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 0x557377ba46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.776+0000 7fc63be00640 1 -- 172.21.15.40:0/4244479086 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573778da6b0 con 0x557377b9acb0 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.776+0000 7fc648a00640 1 --2- 172.21.15.40:0/4244479086 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9fe20 0x557377babc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.776+0000 7fc63be00640 1 --2- 172.21.15.40:0/4244479086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377ba4bf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0ce180 tx=0x7fc62c00ce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e005e5dbc656f8a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.776+0000 7fc631600640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.776+0000 7fc631600640 1 -- 172.21.15.40:0/4244479086 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0a0d00 con 0x557377b9acb0 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.776+0000 7fc631600640 1 -- 172.21.15.40:0/4244479086 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fc62c010a90 con 0x557377b9acb0 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.776+0000 7fc64d812740 1 -- 172.21.15.40:0/4244479086 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6340318a0 con 0x557377b9acb0 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.777+0000 7fc631600640 1 -- 172.21.15.40:0/4244479086 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc62c0a55c0 con 0x557377b9acb0 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.777+0000 7fc64d812740 10 client.25381 fetch_fsmap learned FSMap version 23 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.777+0000 7fc64d812740 1 -- 172.21.15.40:0/4244479086 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557377baec80 con 0x557377b9acb0 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.777+0000 7fc631600640 1 -- 172.21.15.40:0/4244479086 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6340318a0 con 0x557377b9acb0 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.777+0000 7fc631600640 1 -- 172.21.15.40:0/4244479086 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fc62c0bbc80 con 0x557377b9acb0 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.777+0000 7fc64d812740 10 client.25381 fetch_fsmap finished waiting for FSMap version 23 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.777+0000 7fc64d812740 10 client.25381 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.777+0000 7fc64d812740 20 client.25381 populate_metadata read hostname 'smithi040' 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.777+0000 7fc64d812740 1 --2- 172.21.15.40:0/4244479086 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x557377bb12c0 0x557377bb2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.777+0000 7fc64d812740 4 client.25381 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.777+0000 7fc64d812740 1 -- 172.21.15.40:0/4244479086 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557377b0c210 con 0x557377bb12c0 2024-04-27T00:01:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.778+0000 7fc643e00640 1 --2- 172.21.15.40:0/4244479086 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x557377bb12c0 0x557377bb2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.778+0000 7fc643e00640 1 --2- 172.21.15.40:0/4244479086 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x557377bb12c0 0x557377bb2730 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e6aadfae1df2300 server_cookie=3f0b2594ed75e6b in_seq=0 out_seq=0 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.778+0000 7fc631600640 10 client.25381 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.779+0000 7fc631600640 1 -- 172.21.15.40:0/4244479086 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557377b0c210 con 0x557377bb12c0 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.779+0000 7fc631600640 10 client.25381 handle_command_reply: tid=0 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.779+0000 7fc64d812740 1 client.25381 shutdown 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.779+0000 7fc64d812740 1 -- 172.21.15.40:0/4244479086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 msgr2=0x557377ba4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.779+0000 7fc64d812740 1 --2- 172.21.15.40:0/4244479086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377ba4bf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0ce180 tx=0x7fc62c00ce60 comp rx=0 tx=0).stop 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.779+0000 7fc64d812740 1 -- 172.21.15.40:0/4244479086 shutdown_connections 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.779+0000 7fc64d812740 1 --2- 172.21.15.40:0/4244479086 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x557377bb12c0 0x557377bb2730 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.779+0000 7fc64d812740 1 --2- 172.21.15.40:0/4244479086 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9fe20 0x557377babc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.779+0000 7fc631600640 0 client.25381 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.779+0000 7fc64d812740 1 --2- 172.21.15.40:0/4244479086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377ba4bf0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.779+0000 7fc64d812740 1 --2- 172.21.15.40:0/4244479086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9a2c0 0x557377ba46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.779+0000 7fc64d812740 1 -- 172.21.15.40:0/4244479086 >> 172.21.15.40:0/4244479086 conn(0x557377b2ed10 msgr2=0x557377b94f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.780+0000 7fc64d812740 1 -- 172.21.15.40:0/4244479086 shutdown_connections 2024-04-27T00:01:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.780+0000 7fc64d812740 1 -- 172.21.15.40:0/4244479086 wait complete. 2024-04-27T00:01:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.780+0000 7fc64d812740 20 client.25381 trim_cache size 0 max 16384 2024-04-27T00:01:39.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.800+0000 7fc64d812740 1 Processor -- start 2024-04-27T00:01:39.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.800+0000 7fc64d812740 1 -- start start 2024-04-27T00:01:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.800+0000 7fc64d812740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 0x557377b9c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.800+0000 7fc64d812740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377babbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.800+0000 7fc64d812740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9fe20 0x557377b9c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.800+0000 7fc64d812740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557377bb0a70 con 0x557377b9fe20 2024-04-27T00:01:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.800+0000 7fc64d812740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x557377bb1140 con 0x557377b9a2c0 2024-04-27T00:01:39.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.800+0000 7fc64d812740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557377bb1810 con 0x557377b9acb0 2024-04-27T00:01:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.801+0000 7fc648a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9fe20 0x557377b9c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.801+0000 7fc643e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 0x557377b9c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.801+0000 7fc643e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 0x557377b9c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49598/0 (socket says 172.21.15.40:49598) 2024-04-27T00:01:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.801+0000 7fc643e00640 1 -- 172.21.15.40:0/2801798000 learned_addr learned my addr 172.21.15.40:0/2801798000 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.801+0000 7fc63be00640 1 --2- 172.21.15.40:0/2801798000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377babbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.801+0000 7fc643e00640 1 -- 172.21.15.40:0/2801798000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 msgr2=0x557377babbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.801+0000 7fc643e00640 1 --2- 172.21.15.40:0/2801798000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377babbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.801+0000 7fc643e00640 1 -- 172.21.15.40:0/2801798000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9fe20 msgr2=0x557377b9c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.801+0000 7fc643e00640 1 --2- 172.21.15.40:0/2801798000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9fe20 0x557377b9c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.801+0000 7fc643e00640 1 -- 172.21.15.40:0/2801798000 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557377ba5470 con 0x557377b9a2c0 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.801+0000 7fc648a00640 1 --2- 172.21.15.40:0/2801798000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9fe20 0x557377b9c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.801+0000 7fc643e00640 1 --2- 172.21.15.40:0/2801798000 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 0x557377b9c070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc634009a50 tx=0x7fc63403e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f01bc4119d5f69fc server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.802+0000 7fc633e00640 1 -- 172.21.15.40:0/2801798000 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634058020 con 0x557377b9a2c0 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.802+0000 7fc633e00640 1 -- 172.21.15.40:0/2801798000 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc634054e50 con 0x557377b9a2c0 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.802+0000 7fc633e00640 1 -- 172.21.15.40:0/2801798000 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634035040 con 0x557377b9a2c0 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.802+0000 7fc64d812740 1 -- 172.21.15.40:0/2801798000 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 msgr2=0x557377b9c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.802+0000 7fc64d812740 1 --2- 172.21.15.40:0/2801798000 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 0x557377b9c070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc634009a50 tx=0x7fc63403e040 comp rx=0 tx=0).stop 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.803+0000 7fc64d812740 1 -- 172.21.15.40:0/2801798000 shutdown_connections 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.803+0000 7fc64d812740 1 --2- 172.21.15.40:0/2801798000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9fe20 0x557377b9c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.803+0000 7fc64d812740 1 --2- 172.21.15.40:0/2801798000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9acb0 0x557377babbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.803+0000 7fc64d812740 1 --2- 172.21.15.40:0/2801798000 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 0x557377b9c070 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.803+0000 7fc64d812740 1 -- 172.21.15.40:0/2801798000 >> 172.21.15.40:0/2801798000 conn(0x557377b2ed10 msgr2=0x557377ba46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.803+0000 7fc64d812740 1 -- 172.21.15.40:0/2801798000 shutdown_connections 2024-04-27T00:01:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.803+0000 7fc64d812740 1 -- 172.21.15.40:0/2801798000 wait complete. 2024-04-27T00:01:39.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.803+0000 7fc64d812740 1 -- start start 2024-04-27T00:01:39.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.803+0000 7fc64d812740 1 Processor -- start 2024-04-27T00:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc64d812740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 0x557377ba7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc64d812740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9acb0 0x557377ba7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc64d812740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9fe20 0x557377baae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc64d812740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557377bb25b0 con 0x557377b9acb0 2024-04-27T00:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc64d812740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x557377bb1ee0 con 0x557377b9a2c0 2024-04-27T00:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc64d812740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557377bb2c80 con 0x557377b9fe20 2024-04-27T00:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc64d812740 10 client.0 _ll_register_callbacks cb 0x7fc62c0d4150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc63be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9acb0 0x557377ba7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc643e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 0x557377ba7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc63be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9acb0 0x557377ba7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52278/0 (socket says 172.21.15.40:52278) 2024-04-27T00:01:39.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc63be00640 1 -- 172.21.15.40:0/389381477 learned_addr learned my addr 172.21.15.40:0/389381477 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc643e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 0x557377ba7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49606/0 (socket says 172.21.15.40:49606) 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc648a00640 1 --2- 172.21.15.40:0/389381477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9fe20 0x557377baae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc643e00640 1 -- 172.21.15.40:0/389381477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9fe20 msgr2=0x557377baae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc643e00640 1 --2- 172.21.15.40:0/389381477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9fe20 0x557377baae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc643e00640 1 -- 172.21.15.40:0/389381477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9acb0 msgr2=0x557377ba7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc643e00640 1 --2- 172.21.15.40:0/389381477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9acb0 0x557377ba7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.804+0000 7fc643e00640 1 -- 172.21.15.40:0/389381477 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557377bab480 con 0x557377b9a2c0 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.805+0000 7fc643e00640 1 --2- 172.21.15.40:0/389381477 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 0x557377ba7a10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc6340092a0 tx=0x7fc634005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbc9b68aa0bb050f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.805+0000 7fc631600640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.805+0000 7fc631600640 1 -- 172.21.15.40:0/389381477 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634058030 con 0x557377b9a2c0 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.805+0000 7fc631600640 1 -- 172.21.15.40:0/389381477 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fc634004240 con 0x557377b9a2c0 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.805+0000 7fc64d812740 1 -- 172.21.15.40:0/389381477 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc634054e50 con 0x557377b9a2c0 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.805+0000 7fc631600640 1 -- 172.21.15.40:0/389381477 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc63406d050 con 0x557377b9a2c0 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.805+0000 7fc64d812740 10 client.25373 fetch_fsmap learned FSMap version 23 2024-04-27T00:01:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.805+0000 7fc64d812740 1 -- 172.21.15.40:0/389381477 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557377c17870 con 0x557377b9a2c0 2024-04-27T00:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.806+0000 7fc631600640 1 -- 172.21.15.40:0/389381477 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc634054e50 con 0x557377b9a2c0 2024-04-27T00:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.806+0000 7fc631600640 1 -- 172.21.15.40:0/389381477 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fc634054290 con 0x557377b9a2c0 2024-04-27T00:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.806+0000 7fc64d812740 10 client.25373 fetch_fsmap finished waiting for FSMap version 23 2024-04-27T00:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.806+0000 7fc64d812740 10 client.25373 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-27T00:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.806+0000 7fc64d812740 20 client.25373 populate_metadata read hostname 'smithi040' 2024-04-27T00:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.806+0000 7fc64d812740 1 --2- 172.21.15.40:0/389381477 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x557377c1bdb0 0x557377c1aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.806+0000 7fc64d812740 4 client.25373 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-27T00:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.806+0000 7fc64d812740 1 -- 172.21.15.40:0/389381477 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "status"}) -- 0x557377b06080 con 0x557377c1bdb0 2024-04-27T00:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.806+0000 7fc63be00640 1 --2- 172.21.15.40:0/389381477 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x557377c1bdb0 0x557377c1aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc63be00640 1 --2- 172.21.15.40:0/389381477 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x557377c1bdb0 0x557377c1aac0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec28cdd8a5250e75 server_cookie=dfcffd0aa09f3c3f in_seq=0 out_seq=0 2024-04-27T00:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc631600640 10 client.25373 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:01:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc631600640 1 -- 172.21.15.40:0/389381477 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557377b06080 con 0x557377c1bdb0 2024-04-27T00:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc631600640 10 client.25373 handle_command_reply: tid=0 2024-04-27T00:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc64d812740 1 client.25373 shutdown 2024-04-27T00:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc64d812740 1 -- 172.21.15.40:0/389381477 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 msgr2=0x557377ba7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc64d812740 1 --2- 172.21.15.40:0/389381477 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 0x557377ba7a10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc6340092a0 tx=0x7fc634005d90 comp rx=0 tx=0).stop 2024-04-27T00:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc64d812740 1 -- 172.21.15.40:0/389381477 shutdown_connections 2024-04-27T00:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc64d812740 1 --2- 172.21.15.40:0/389381477 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x557377c1bdb0 0x557377c1aac0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc64d812740 1 --2- 172.21.15.40:0/389381477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557377b9fe20 0x557377baae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc64d812740 1 --2- 172.21.15.40:0/389381477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557377b9acb0 0x557377ba7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc631600640 0 client.25373 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc64d812740 1 --2- 172.21.15.40:0/389381477 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x557377b9a2c0 0x557377ba7a10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc64d812740 1 -- 172.21.15.40:0/389381477 >> 172.21.15.40:0/389381477 conn(0x557377b2ed10 msgr2=0x557377b9c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc64d812740 1 -- 172.21.15.40:0/389381477 shutdown_connections 2024-04-27T00:01:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.807+0000 7fc64d812740 1 -- 172.21.15.40:0/389381477 wait complete. 2024-04-27T00:01:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.808+0000 7fc64d812740 20 client.25373 trim_cache size 0 max 16384 2024-04-27T00:01:39.814 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-27T00:01:39.814 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "78cff6ea-0427-11ef-bc93-c7b262605968", 2024-04-27T00:01:39.814 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-04-27T00:01:39.814 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24590, 2024-04-27T00:01:39.814 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-04-27T00:01:39.814 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-04-27T00:01:39.814 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-04-27T00:01:39.815 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 168.693466693, 2024-04-27T00:01:39.815 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 23, 2024-04-27T00:01:39.815 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 92, 2024-04-27T00:01:39.815 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 90, 2024-04-27T00:01:39.815 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 175.10145572299999 2024-04-27T00:01:39.815 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-27T00:01:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.808+0000 7fc649400640 1 -- 172.21.15.40:0/1858329380 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6280a3cc0 msgr2=0x7fc6280a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.808+0000 7fc649400640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6280a3cc0 0x7fc6280a6180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x557377b9b510 tx=0x7fc63c009500 comp rx=0 tx=0).stop 2024-04-27T00:01:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.808+0000 7fc649400640 1 -- 172.21.15.40:0/1858329380 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc644076970 msgr2=0x7fc64419f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.808+0000 7fc649400640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc644076970 0x7fc64419f4f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00e9f0 tx=0x7fc62c00eeb0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.808+0000 7fc649400640 1 -- 172.21.15.40:0/1858329380 shutdown_connections 2024-04-27T00:01:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.808+0000 7fc649400640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64410ed40 0x7fc6441a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.808+0000 7fc649400640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc644076970 0x7fc64419f4f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.808+0000 7fc649400640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6280a3cc0 0x7fc6280a6180 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.808+0000 7fc649400640 1 --2- 172.21.15.40:0/1858329380 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc644075390 0x7fc64419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:39.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.808+0000 7fc649400640 1 -- 172.21.15.40:0/1858329380 >> 172.21.15.40:0/1858329380 conn(0x7fc6440fdce0 msgr2=0x7fc6441020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:39.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.809+0000 7fc649400640 1 -- 172.21.15.40:0/1858329380 shutdown_connections 2024-04-27T00:01:39.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:39.809+0000 7fc649400640 1 -- 172.21.15.40:0/1858329380 wait complete. 2024-04-27T00:01:39.829 INFO:teuthology.run_tasks:Running task sequential... 2024-04-27T00:01:39.881 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-27T00:01:39.882 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-27T00:01:39.882 INFO:teuthology.task.print:Enabling mgr modules 2024-04-27T00:01:39.882 INFO:teuthology.run_tasks:Running task kclient... 2024-04-27T00:01:39.897 INFO:tasks.kclient:Mounting kernel clients... 2024-04-27T00:01:39.897 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-27T00:01:39.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:39.898 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-27T00:01:39.953 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-27T00:01:39.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:39.955 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-27T00:01:40.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:40.039 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-04-27T00:01:40.162 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-27T00:01:40.162 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-27T00:01:40.162 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-27T00:01:40.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-27T00:01:40.600 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:01:40.802 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:40 smithi080 ceph-mon[32362]: pgmap v409: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:40.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:40 smithi103 ceph-mon[32952]: pgmap v409: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:40.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:40 smithi040 ceph-mon[28095]: pgmap v409: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:42.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:42 smithi040 ceph-mon[28095]: pgmap v410: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:01:42.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:42 smithi103 ceph-mon[32952]: pgmap v410: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:01:43.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:42 smithi080 ceph-mon[32362]: pgmap v410: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:01:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.081+0000 7f1b9be00700 1 -- 172.21.15.40:0/1003545591 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1b9c076170 msgr2=0x7f1b9c10f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.081+0000 7f1b9be00700 1 --2- 172.21.15.40:0/1003545591 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1b9c076170 0x7f1b9c10f650 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1b880098f0 tx=0x7f1b88009c00 comp rx=0 tx=0).stop 2024-04-27T00:01:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.081+0000 7f1b9be00700 1 -- 172.21.15.40:0/1003545591 shutdown_connections 2024-04-27T00:01:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.081+0000 7f1b9be00700 1 --2- 172.21.15.40:0/1003545591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b9c10fb90 0x7f1b9c112000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.081+0000 7f1b9be00700 1 --2- 172.21.15.40:0/1003545591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b9c075850 0x7f1b9c075c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.081+0000 7f1b9be00700 1 --2- 172.21.15.40:0/1003545591 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1b9c076170 0x7f1b9c10f650 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:43.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.081+0000 7f1b9be00700 1 -- 172.21.15.40:0/1003545591 >> 172.21.15.40:0/1003545591 conn(0x7f1b9c0feea0 msgr2=0x7f1b9c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:43.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.081+0000 7f1b9be00700 1 -- 172.21.15.40:0/1003545591 shutdown_connections 2024-04-27T00:01:43.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.081+0000 7f1b9be00700 1 -- 172.21.15.40:0/1003545591 wait complete. 2024-04-27T00:01:43.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.082+0000 7f1b9be00700 1 Processor -- start 2024-04-27T00:01:43.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.082+0000 7f1b9be00700 1 -- start start 2024-04-27T00:01:43.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.083+0000 7f1b9be00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b9c075850 0x7f1b9c1a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:43.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.083+0000 7f1b9be00700 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1b9c076170 0x7f1b9c1a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:43.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.083+0000 7f1b9be00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b9c10fb90 0x7f1b9c19bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:43.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.083+0000 7f1b9be00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9c1157a0 con 0x7f1b9c10fb90 2024-04-27T00:01:43.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.083+0000 7f1b9be00700 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9c115620 con 0x7f1b9c076170 2024-04-27T00:01:43.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.083+0000 7f1b9be00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9c115920 con 0x7f1b9c075850 2024-04-27T00:01:43.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.083+0000 7f1b9aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b9c075850 0x7f1b9c1a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:43.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.083+0000 7f1b9aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b9c075850 0x7f1b9c1a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:57496/0 (socket says 172.21.15.40:57496) 2024-04-27T00:01:43.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.083+0000 7f1b9aa00700 1 -- 172.21.15.40:0/196125620 learned_addr learned my addr 172.21.15.40:0/196125620 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:43.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.083+0000 7f1b9a000700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1b9c076170 0x7f1b9c1a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:43.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.084+0000 7f1b9b400700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b9c10fb90 0x7f1b9c19bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:43.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.084+0000 7f1b9aa00700 1 -- 172.21.15.40:0/196125620 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1b9c076170 msgr2=0x7f1b9c1a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:43.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.084+0000 7f1b9aa00700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1b9c076170 0x7f1b9c1a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:43.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.084+0000 7f1b9aa00700 1 -- 172.21.15.40:0/196125620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b9c10fb90 msgr2=0x7f1b9c19bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:43.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.084+0000 7f1b9aa00700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b9c10fb90 0x7f1b9c19bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:43.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.084+0000 7f1b9aa00700 1 -- 172.21.15.40:0/196125620 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b9c19c2a0 con 0x7f1b9c075850 2024-04-27T00:01:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.084+0000 7f1b9aa00700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b9c075850 0x7f1b9c1a1930 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00c0c0 tx=0x7f1b8c00c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.085+0000 7f1b93400700 1 -- 172.21.15.40:0/196125620 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c00cd10 con 0x7f1b9c075850 2024-04-27T00:01:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.085+0000 7f1b9be00700 1 -- 172.21.15.40:0/196125620 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b9c19c530 con 0x7f1b9c075850 2024-04-27T00:01:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.085+0000 7f1b93400700 1 -- 172.21.15.40:0/196125620 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b8c0075c0 con 0x7f1b9c075850 2024-04-27T00:01:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.085+0000 7f1b93400700 1 -- 172.21.15.40:0/196125620 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c0146e0 con 0x7f1b9c075850 2024-04-27T00:01:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.085+0000 7f1b9be00700 1 -- 172.21.15.40:0/196125620 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b9c1a8800 con 0x7f1b9c075850 2024-04-27T00:01:43.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.087+0000 7f1b9be00700 1 -- 172.21.15.40:0/196125620 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b9c06a110 con 0x7f1b9c075850 2024-04-27T00:01:43.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.088+0000 7f1b93400700 1 -- 172.21.15.40:0/196125620 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1b8c01b020 con 0x7f1b9c075850 2024-04-27T00:01:43.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.090+0000 7f1b93400700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1b800a3d50 0x7f1b800a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:43.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.090+0000 7f1b93400700 1 -- 172.21.15.40:0/196125620 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f1b8c0d07f0 con 0x7f1b9c075850 2024-04-27T00:01:43.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.090+0000 7f1b9a000700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1b800a3d50 0x7f1b800a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:43.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.091+0000 7f1b9a000700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1b800a3d50 0x7f1b800a6210 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1b880098c0 tx=0x7f1b88019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:43.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.095+0000 7f1b93400700 1 -- 172.21.15.40:0/196125620 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1b8c010040 con 0x7f1b9c075850 2024-04-27T00:01:43.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.384+0000 7f1b9be00700 1 -- 172.21.15.40:0/196125620 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1b9c19cfe0 con 0x7f1b9c075850 2024-04-27T00:01:43.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.385+0000 7f1b93400700 1 -- 172.21.15.40:0/196125620 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+88 (secure 0 0 0) 0x7f1b8c09ccd0 con 0x7f1b9c075850 2024-04-27T00:01:43.387 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-27T00:01:43.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.388+0000 7f1b9be00700 1 -- 172.21.15.40:0/196125620 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1b800a3d50 msgr2=0x7f1b800a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:43.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.388+0000 7f1b9be00700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1b800a3d50 0x7f1b800a6210 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1b880098c0 tx=0x7f1b88019040 comp rx=0 tx=0).stop 2024-04-27T00:01:43.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.388+0000 7f1b9be00700 1 -- 172.21.15.40:0/196125620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b9c075850 msgr2=0x7f1b9c1a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:43.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.388+0000 7f1b9be00700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b9c075850 0x7f1b9c1a1930 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00c0c0 tx=0x7f1b8c00c3d0 comp rx=0 tx=0).stop 2024-04-27T00:01:43.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.388+0000 7f1b9be00700 1 -- 172.21.15.40:0/196125620 shutdown_connections 2024-04-27T00:01:43.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.388+0000 7f1b9be00700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1b800a3d50 0x7f1b800a6210 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:43.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.388+0000 7f1b9be00700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b9c10fb90 0x7f1b9c19bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:43.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.388+0000 7f1b9be00700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b9c075850 0x7f1b9c1a1930 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:43.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.388+0000 7f1b9be00700 1 --2- 172.21.15.40:0/196125620 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1b9c076170 0x7f1b9c1a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:43.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.388+0000 7f1b9be00700 1 -- 172.21.15.40:0/196125620 >> 172.21.15.40:0/196125620 conn(0x7f1b9c0feea0 msgr2=0x7f1b9c0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:43.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.389+0000 7f1b9be00700 1 -- 172.21.15.40:0/196125620 shutdown_connections 2024-04-27T00:01:43.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:43.389+0000 7f1b9be00700 1 -- 172.21.15.40:0/196125620 wait complete. 2024-04-27T00:01:43.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:43 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/196125620' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-27T00:01:43.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:43 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/196125620' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-27T00:01:44.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:43 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/196125620' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-27T00:01:44.368 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-27T00:01:44.368 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-27T00:01:44.368 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-04-27T00:01:44.368 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-27T00:01:44.368 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-27T00:01:44.368 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-27T00:01:44.368 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-27T00:01:44.368 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-27T00:01:44.368 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-27T00:01:44.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:44.369 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-04-27T00:01:44.429 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-27T00:01:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-27T00:01:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-04-27T00:01:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-27T00:01:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-04-27T00:01:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-27T00:01:44.429 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T00:01:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-04-27T00:01:44.429 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 41942sec preferred_lft 41942sec 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-04-27T00:01:44.430 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-04-27T00:01:44.431 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-27T00:01:44.431 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T00:01:44.432 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-27T00:01:44.432 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add name ceph-brx type bridge 2024-04-27T00:01:44.432 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr flush dev ceph-brx 2024-04-27T00:01:44.432 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set ceph-brx up 2024-04-27T00:01:44.432 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-27T00:01:44.432 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-27T00:01:44.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T00:01:44.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T00:01:44.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:44.682 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-27T00:01:44.759 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-04-27T00:01:44.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:44.763 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-04-27T00:01:44.813 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-04-27T00:01:44.813 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-27T00:01:44.813 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-27T00:01:44.813 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-27T00:01:44.813 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-27T00:01:44.814 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T00:01:44.814 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-27T00:01:44.814 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-27T00:01:44.815 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-27T00:01:44.815 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-27T00:01:44.815 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-27T00:01:44.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:44 smithi103 ceph-mon[32952]: pgmap v411: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:44.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T00:01:44.953 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:44 smithi040 ceph-mon[28095]: pgmap v411: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:45.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T00:01:45.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:45.046 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-27T00:01:45.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:44 smithi080 ceph-mon[32362]: pgmap v411: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:45.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:45.099 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list-id 2024-04-27T00:01:45.153 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T00:01:45.153 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-27T00:01:45.153 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-27T00:01:45.153 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-27T00:01:45.153 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-27T00:01:45.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T00:01:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T00:01:45.346 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-27T00:01:45.346 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T00:01:45.346 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-27T00:01:45.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-27T00:01:45.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-27T00:01:45.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-27T00:01:45.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-27T00:01:45.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-27T00:01:45.347 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-27T00:01:45.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T00:01:45.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T00:01:45.632 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-27T00:01:45.632 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-04-27T00:01:45.632 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-04-27T00:01:45.632 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set dev brx.0 master ceph-brx 2024-04-27T00:01:45.632 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-04-27T00:01:45.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T00:01:45.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-27T00:01:45.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-27T00:01:45.803 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:45 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-27T00:01:45.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T00:01:45.837 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-27T00:01:45.837 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-27T00:01:45.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-27T00:01:45.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-27T00:01:45.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:45 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-27T00:01:45.884 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-27T00:01:45.885 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-27T00:01:45.885 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-27T00:01:45.938 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-27T00:01:45.938 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-27T00:01:45.938 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/44d4f898-2f89-4f34-8673-5d8bb96f7f18 2024-04-27T00:01:45.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:45.938 DEBUG:teuthology.orchestra.run.smithi040:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/44d4f898-2f89-4f34-8673-5d8bb96f7f18 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-27T00:01:46.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-27T00:01:46.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-27T00:01:46.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:45 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-27T00:01:46.073 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-27T00:01:46.073 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-04-27T00:01:46.073 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-27T00:01:46.789 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:46 smithi040 ceph-mon[28095]: pgmap v412: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:46.790 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-27T00:01:46.790 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-27T00:01:46.790 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-27T00:01:46.790 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:46 smithi040 ceph-mon[28095]: osdmap e93: 12 total, 12 up, 12 in 2024-04-27T00:01:46.822 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-27T00:01:46.822 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-04-27T00:01:46.822 INFO:teuthology.orchestra.run.smithi040.stdout:invalid new device string format 2024-04-27T00:01:46.822 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.80:3300,172.21.15.103:3300" 2024-04-27T00:01:46.822 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:3300,172.21.15.80:3300,172.21.15.103:3300:/volumes/qa/sv_1/44d4f898-2f89-4f34-8673-5d8bb96f7f18 2024-04-27T00:01:46.823 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=78cff6ea-0427-11ef-bc93-c7b262605968" will pass to kernel 2024-04-27T00:01:46.827 INFO:teuthology.orchestra.run.smithi040.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-27T00:01:46.827 INFO:teuthology.orchestra.run.smithi040.stdout: You just mounted a file system that supports labels which does not 2024-04-27T00:01:46.827 INFO:teuthology.orchestra.run.smithi040.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-27T00:01:46.827 INFO:teuthology.orchestra.run.smithi040.stdout: applications will generate AVC messages and not be allowed access to 2024-04-27T00:01:46.827 INFO:teuthology.orchestra.run.smithi040.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-27T00:01:46.833 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-27T00:01:46.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:46.833 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-27T00:01:46.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:46 smithi103 ceph-mon[32952]: pgmap v412: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:46.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-27T00:01:46.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-27T00:01:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-27T00:01:46.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:46 smithi103 ceph-mon[32952]: osdmap e93: 12 total, 12 up, 12 in 2024-04-27T00:01:46.923 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T00:01:47.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T00:01:47.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T00:01:47.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:46 smithi080 ceph-mon[32362]: pgmap v412: 129 pgs: 129 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:47.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-27T00:01:47.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-27T00:01:47.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-27T00:01:47.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:46 smithi080 ceph-mon[32362]: osdmap e93: 12 total, 12 up, 12 in 2024-04-27T00:01:47.051 INFO:teuthology.orchestra.run.smithi040.stdout:client25396 2024-04-27T00:01:47.052 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-27T00:01:47.052 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T00:01:47.106 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T00:01:47.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T00:01:47.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T00:01:47.234 INFO:teuthology.orchestra.run.smithi040.stdout:client25396 2024-04-27T00:01:47.234 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-27T00:01:47.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-27T00:01:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T00:01:47.358 INFO:teuthology.orchestra.run.smithi040.stdout:client25396 2024-04-27T00:01:47.359 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/78cff6ea-0427-11ef-bc93-c7b262605968.client25396/status') 2024-04-27T00:01:47.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-27T00:01:47.494 INFO:teuthology.orchestra.run.smithi040.stdout:instance: client.25396 (3)172.21.15.40:0/1194833493 2024-04-27T00:01:47.494 INFO:teuthology.orchestra.run.smithi040.stdout:blocklisted: false 2024-04-27T00:01:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:0+1 records in 2024-04-27T00:01:47.494 INFO:teuthology.orchestra.run.smithi040.stderr:0+1 records out 2024-04-27T00:01:47.495 INFO:teuthology.orchestra.run.smithi040.stderr:71 bytes copied, 9.1122e-05 s, 779 kB/s 2024-04-27T00:01:47.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-27T00:01:47.508 INFO:teuthology.run_tasks:Running task exec... 2024-04-27T00:01:47.563 INFO:teuthology.task.exec:Executing custom commands... 2024-04-27T00:01:47.563 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-04-27T00:01:47.563 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-27T00:01:47.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.780+0000 7f4c4b200640 1 -- 172.21.15.40:0/1990130468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c4410a990 msgr2=0x7f4c44111220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:47.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.780+0000 7f4c4b200640 1 --2- 172.21.15.40:0/1990130468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c4410a990 0x7f4c44111220 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009ca0 tx=0x7f4c4002a8b0 comp rx=0 tx=0).stop 2024-04-27T00:01:47.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.780+0000 7f4c4b200640 1 -- 172.21.15.40:0/1990130468 shutdown_connections 2024-04-27T00:01:47.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.780+0000 7f4c4b200640 1 --2- 172.21.15.40:0/1990130468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c4410a990 0x7f4c44111220 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:47.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.780+0000 7f4c4b200640 1 --2- 172.21.15.40:0/1990130468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44109fd0 0x7f4c4410a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:47.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.780+0000 7f4c4b200640 1 --2- 172.21.15.40:0/1990130468 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c44108de0 0x7f4c44109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:47.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.780+0000 7f4c4b200640 1 -- 172.21.15.40:0/1990130468 >> 172.21.15.40:0/1990130468 conn(0x7f4c44077dc0 msgr2=0x7f4c4407a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:47.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.780+0000 7f4c4b200640 1 -- 172.21.15.40:0/1990130468 shutdown_connections 2024-04-27T00:01:47.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.780+0000 7f4c4b200640 1 -- 172.21.15.40:0/1990130468 wait complete. 2024-04-27T00:01:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.781+0000 7f4c4b200640 1 Processor -- start 2024-04-27T00:01:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.782+0000 7f4c4b200640 1 -- start start 2024-04-27T00:01:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.782+0000 7f4c4b200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c44108de0 0x7f4c4419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.782+0000 7f4c4b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44109fd0 0x7f4c4419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.782+0000 7f4c4b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c4410a990 0x7f4c441a6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.782+0000 7f4c4b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4c44114940 con 0x7f4c4410a990 2024-04-27T00:01:47.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.782+0000 7f4c4b200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441147c0 con 0x7f4c44108de0 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.782+0000 7f4c4b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c44114ac0 con 0x7f4c44109fd0 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.782+0000 7f4c4a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c4410a990 0x7f4c441a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.782+0000 7f4c4a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c4410a990 0x7f4c441a6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60654/0 (socket says 172.21.15.40:60654) 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.782+0000 7f4c4a800640 1 -- 172.21.15.40:0/416729966 learned_addr learned my addr 172.21.15.40:0/416729966 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.782+0000 7f4c49400640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44109fd0 0x7f4c4419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.782+0000 7f4c49e00640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c44108de0 0x7f4c4419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.783+0000 7f4c4a800640 1 -- 172.21.15.40:0/416729966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44109fd0 msgr2=0x7f4c4419f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.783+0000 7f4c4a800640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44109fd0 0x7f4c4419f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.783+0000 7f4c4a800640 1 -- 172.21.15.40:0/416729966 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c44108de0 msgr2=0x7f4c4419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.783+0000 7f4c4a800640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c44108de0 0x7f4c4419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.783+0000 7f4c4a800640 1 -- 172.21.15.40:0/416729966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c441a6bb0 con 0x7f4c4410a990 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.783+0000 7f4c49400640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44109fd0 0x7f4c4419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.783+0000 7f4c4a800640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c4410a990 0x7f4c441a6540 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4c40003bd0 tx=0x7f4c40003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.783+0000 7f4c3ea00640 1 -- 172.21.15.40:0/416729966 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c40004450 con 0x7f4c4410a990 2024-04-27T00:01:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.784+0000 7f4c3ea00640 1 -- 172.21.15.40:0/416729966 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c40040070 con 0x7f4c4410a990 2024-04-27T00:01:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.784+0000 7f4c4b200640 1 -- 172.21.15.40:0/416729966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c441a6e40 con 0x7f4c4410a990 2024-04-27T00:01:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.784+0000 7f4c3ea00640 1 -- 172.21.15.40:0/416729966 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c40031c60 con 0x7f4c4410a990 2024-04-27T00:01:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.784+0000 7f4c4b200640 1 -- 172.21.15.40:0/416729966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c441a7350 con 0x7f4c4410a990 2024-04-27T00:01:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.785+0000 7f4c4b200640 1 -- 172.21.15.40:0/416729966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c44110180 con 0x7f4c4410a990 2024-04-27T00:01:47.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.787+0000 7f4c3ea00640 1 -- 172.21.15.40:0/416729966 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4c40031440 con 0x7f4c4410a990 2024-04-27T00:01:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.788+0000 7f4c3ea00640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4c2c0a3cb0 0x7f4c2c0a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:47.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.788+0000 7f4c3ea00640 1 -- 172.21.15.40:0/416729966 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f4c400f1a30 con 0x7f4c4410a990 2024-04-27T00:01:47.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.789+0000 7f4c49e00640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4c2c0a3cb0 0x7f4c2c0a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:47.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.789+0000 7f4c49e00640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4c2c0a3cb0 0x7f4c2c0a6170 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4c44109e30 tx=0x7f4c3800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:47.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:47.793+0000 7f4c3ea00640 1 -- 172.21.15.40:0/416729966 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4c400bdfc0 con 0x7f4c4410a990 2024-04-27T00:01:47.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:47 smithi103 ceph-mon[32952]: osdmap e94: 12 total, 12 up, 12 in 2024-04-27T00:01:47.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:47 smithi040 ceph-mon[28095]: osdmap e94: 12 total, 12 up, 12 in 2024-04-27T00:01:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.041+0000 7f4c4b200640 1 -- 172.21.15.40:0/416729966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f4c4402dca0 con 0x7f4c4410a990 2024-04-27T00:01:48.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:47 smithi080 ceph-mon[32362]: osdmap e94: 12 total, 12 up, 12 in 2024-04-27T00:01:48.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.606+0000 7f4c3ea00640 1 -- 172.21.15.40:0/416729966 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v24) ==== 117+0+0 (secure 0 0 0) 0x7f4c400bd8c0 con 0x7f4c4410a990 2024-04-27T00:01:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.610+0000 7f4c4b200640 1 -- 172.21.15.40:0/416729966 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4c2c0a3cb0 msgr2=0x7f4c2c0a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.610+0000 7f4c4b200640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4c2c0a3cb0 0x7f4c2c0a6170 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4c44109e30 tx=0x7f4c3800b390 comp rx=0 tx=0).stop 2024-04-27T00:01:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.610+0000 7f4c4b200640 1 -- 172.21.15.40:0/416729966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c4410a990 msgr2=0x7f4c441a6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.610+0000 7f4c4b200640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c4410a990 0x7f4c441a6540 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4c40003bd0 tx=0x7f4c40003c00 comp rx=0 tx=0).stop 2024-04-27T00:01:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.610+0000 7f4c4b200640 1 -- 172.21.15.40:0/416729966 shutdown_connections 2024-04-27T00:01:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.610+0000 7f4c4b200640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c4410a990 0x7f4c441a6540 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.610+0000 7f4c4b200640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f4c2c0a3cb0 0x7f4c2c0a6170 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.610+0000 7f4c4b200640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44109fd0 0x7f4c4419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.610+0000 7f4c4b200640 1 --2- 172.21.15.40:0/416729966 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4c44108de0 0x7f4c4419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:48.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.610+0000 7f4c4b200640 1 -- 172.21.15.40:0/416729966 >> 172.21.15.40:0/416729966 conn(0x7f4c44077dc0 msgr2=0x7f4c4410d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.611+0000 7f4c4b200640 1 -- 172.21.15.40:0/416729966 shutdown_connections 2024-04-27T00:01:48.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:01:48.611+0000 7f4c4b200640 1 -- 172.21.15.40:0/416729966 wait complete. 2024-04-27T00:01:48.636 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-27T00:01:48.660 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-27T00:01:48.669 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-04-27T00:01:48.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-27T00:01:48.675 INFO:teuthology.run_tasks:Running task workunit... 2024-04-27T00:01:48.716 INFO:tasks.workunit:Pulling workunits from ref bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-27T00:01:48.716 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-27T00:01:48.716 INFO:tasks.workunit:timeout=3h 2024-04-27T00:01:48.716 INFO:tasks.workunit:cleanup=True 2024-04-27T00:01:48.716 DEBUG:teuthology.orchestra.run.smithi040:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-27T00:01:48.770 INFO:teuthology.orchestra.run.smithi040.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-27T00:01:48.770 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-27T00:01:48.770 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 72h/114d Inode: 1099511628283 Links: 2 2024-04-27T00:01:48.771 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T00:01:48.771 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-27T00:01:48.771 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-04-26 23:59:27.417694437 +0000 2024-04-27T00:01:48.771 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-04-26 23:59:27.417694437 +0000 2024-04-27T00:01:48.771 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-04-27 00:01:46.915776205 +0000 2024-04-27T00:01:48.771 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-04-26 23:59:27.417694437 +0000 2024-04-27T00:01:48.771 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-27T00:01:48.771 DEBUG:teuthology.orchestra.run.smithi040:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-27T00:01:48.850 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:48 smithi040 ceph-mon[28095]: pgmap v415: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:48.850 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:48 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/416729966' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-27T00:01:48.853 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-27T00:01:48.894 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:48 smithi080 ceph-mon[32362]: pgmap v415: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:48.894 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:48 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/416729966' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-27T00:01:48.908 INFO:tasks.workunit.client.0.smithi040.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-27T00:01:49.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:48 smithi103 ceph-mon[32952]: pgmap v415: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:49.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:48 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/416729966' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-27T00:01:49.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:49 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/416729966' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-27T00:01:49.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:49 smithi080 ceph-mon[32362]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-27T00:01:49.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:49 smithi080 ceph-mon[32362]: pgmap v416: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:49.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:49 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/416729966' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-27T00:01:49.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:49 smithi040 ceph-mon[28095]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-27T00:01:49.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:49 smithi040 ceph-mon[28095]: pgmap v416: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:50.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:49 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/416729966' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-27T00:01:50.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:49 smithi103 ceph-mon[32952]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-27T00:01:50.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:49 smithi103 ceph-mon[32952]: pgmap v416: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:52.098 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:51 smithi103 ceph-mon[32952]: pgmap v417: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 127 B/s wr, 4 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:52.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:51 smithi040 ceph-mon[28095]: pgmap v417: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 127 B/s wr, 4 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:52.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:51 smithi080 ceph-mon[32362]: pgmap v417: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 127 B/s wr, 4 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:54.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:53 smithi103 ceph-mon[32952]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.1 KiB/s wr, 8 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:54.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:53 smithi040 ceph-mon[28095]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.1 KiB/s wr, 8 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:54.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:53 smithi080 ceph-mon[32362]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.1 KiB/s wr, 8 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:56.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:55 smithi040 ceph-mon[28095]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.8 KiB/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:56.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:55 smithi080 ceph-mon[32362]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.8 KiB/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:56.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:55 smithi103 ceph-mon[32952]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.8 KiB/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:58.150 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:57 smithi103 ceph-mon[32952]: pgmap v420: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:58.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:57 smithi040 ceph-mon[28095]: pgmap v420: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:01:58.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:57 smithi080 ceph-mon[32362]: pgmap v420: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s; 2 B/s, 0 objects/s recovering 2024-04-27T00:02:00.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:01:59 smithi040 ceph-mon[28095]: pgmap v421: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.4 KiB/s wr, 5 op/s; 1 B/s, 0 objects/s recovering 2024-04-27T00:02:00.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:01:59 smithi080 ceph-mon[32362]: pgmap v421: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.4 KiB/s wr, 5 op/s; 1 B/s, 0 objects/s recovering 2024-04-27T00:02:00.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:01:59 smithi103 ceph-mon[32952]: pgmap v421: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.4 KiB/s wr, 5 op/s; 1 B/s, 0 objects/s recovering 2024-04-27T00:02:01.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.937+0000 7f72d1a00640 1 -- 172.21.15.40:0/3631589075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72cc1028a0 msgr2=0x7f72cc102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:01.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.937+0000 7f72d1a00640 1 --2- 172.21.15.40:0/3631589075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72cc1028a0 0x7f72cc102d20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009900 tx=0x7f72b402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.937+0000 7f72d1a00640 1 -- 172.21.15.40:0/3631589075 shutdown_connections 2024-04-27T00:02:01.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.937+0000 7f72d1a00640 1 --2- 172.21.15.40:0/3631589075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72cc103260 0x7f72cc109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.937+0000 7f72d1a00640 1 --2- 172.21.15.40:0/3631589075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72cc1028a0 0x7f72cc102d20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.937+0000 7f72d1a00640 1 --2- 172.21.15.40:0/3631589075 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f72cc1016b0 0x7f72cc101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.937+0000 7f72d1a00640 1 -- 172.21.15.40:0/3631589075 >> 172.21.15.40:0/3631589075 conn(0x7f72cc0fcc20 msgr2=0x7f72cc0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:01.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.937+0000 7f72d1a00640 1 -- 172.21.15.40:0/3631589075 shutdown_connections 2024-04-27T00:02:01.940 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.937+0000 7f72d1a00640 1 -- 172.21.15.40:0/3631589075 wait complete. 2024-04-27T00:02:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.938+0000 7f72d1a00640 1 Processor -- start 2024-04-27T00:02:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.938+0000 7f72d1a00640 1 -- start start 2024-04-27T00:02:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.938+0000 7f72d1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72cc1016b0 0x7f72cc19acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.938+0000 7f72d1a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f72cc1028a0 0x7f72cc19b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.938+0000 7f72d1a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72cc103260 0x7f72cc1a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.938+0000 7f72d1a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc10d230 con 0x7f72cc1016b0 2024-04-27T00:02:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.938+0000 7f72d1a00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc10d0b0 con 0x7f72cc1028a0 2024-04-27T00:02:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.938+0000 7f72d1a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc10d3b0 con 0x7f72cc103260 2024-04-27T00:02:01.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.939+0000 7f72cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72cc1016b0 0x7f72cc19acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.939+0000 7f72cb400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f72cc1028a0 0x7f72cc19b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.939+0000 7f72cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72cc1016b0 0x7f72cc19acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39344/0 (socket says 172.21.15.40:39344) 2024-04-27T00:02:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.939+0000 7f72cbe00640 1 -- 172.21.15.40:0/60288084 learned_addr learned my addr 172.21.15.40:0/60288084 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.939+0000 7f72d1000640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72cc103260 0x7f72cc1a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.939+0000 7f72cbe00640 1 -- 172.21.15.40:0/60288084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72cc103260 msgr2=0x7f72cc1a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.939+0000 7f72cbe00640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72cc103260 0x7f72cc1a21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.939+0000 7f72cbe00640 1 -- 172.21.15.40:0/60288084 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f72cc1028a0 msgr2=0x7f72cc19b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.939+0000 7f72cbe00640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f72cc1028a0 0x7f72cc19b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.943 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.939+0000 7f72cbe00640 1 -- 172.21.15.40:0/60288084 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72cc1a2880 con 0x7f72cc1016b0 2024-04-27T00:02:01.944 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.939+0000 7f72cbe00640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72cc1016b0 0x7f72cc19acc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00e9c0 tx=0x7f72bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:01.944 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.940+0000 7f72c8c00640 1 -- 172.21.15.40:0/60288084 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc00cd80 con 0x7f72cc1016b0 2024-04-27T00:02:01.944 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.940+0000 7f72c8c00640 1 -- 172.21.15.40:0/60288084 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72bc0035f0 con 0x7f72cc1016b0 2024-04-27T00:02:01.944 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.940+0000 7f72d1a00640 1 -- 172.21.15.40:0/60288084 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72cc1a2b70 con 0x7f72cc1016b0 2024-04-27T00:02:01.944 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.940+0000 7f72c8c00640 1 -- 172.21.15.40:0/60288084 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc010690 con 0x7f72cc1016b0 2024-04-27T00:02:01.944 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.940+0000 7f72d1a00640 1 -- 172.21.15.40:0/60288084 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72cc1a30c0 con 0x7f72cc1016b0 2024-04-27T00:02:01.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.943+0000 7f72d5f8a740 1 Processor -- start 2024-04-27T00:02:01.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.943+0000 7f72c8c00640 1 -- 172.21.15.40:0/60288084 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f72bc0037a0 con 0x7f72cc1016b0 2024-04-27T00:02:01.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.943+0000 7f72d5f8a740 1 -- start start 2024-04-27T00:02:01.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.944+0000 7f72d5f8a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 0x558195c07200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.944+0000 7f72d5f8a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 0x558195c0c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.944+0000 7f72d5f8a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c0c940 0x558195c0edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.944+0000 7f72d5f8a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558195b6b380 con 0x558195c077d0 2024-04-27T00:02:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.944+0000 7f72d5f8a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x558195b07c30 con 0x558195c06de0 2024-04-27T00:02:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.944+0000 7f72d5f8a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558195b67340 con 0x558195c0c940 2024-04-27T00:02:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.944+0000 7f72cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 0x558195c0c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.944+0000 7f72cb400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c0c940 0x558195c0edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.944+0000 7f72cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 0x558195c0c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39360/0 (socket says 172.21.15.40:39360) 2024-04-27T00:02:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.944+0000 7f72cbe00640 1 -- 172.21.15.40:0/3726118158 learned_addr learned my addr 172.21.15.40:0/3726118158 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.944+0000 7f72d1000640 1 --2- 172.21.15.40:0/3726118158 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 0x558195c07200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.945+0000 7f72cb400640 1 -- 172.21.15.40:0/3726118158 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 msgr2=0x558195c07200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.945+0000 7f72cb400640 1 --2- 172.21.15.40:0/3726118158 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 0x558195c07200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.945+0000 7f72cb400640 1 -- 172.21.15.40:0/3726118158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 msgr2=0x558195c0c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.945+0000 7f72cb400640 1 --2- 172.21.15.40:0/3726118158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 0x558195c0c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.945+0000 7f72cb400640 1 -- 172.21.15.40:0/3726118158 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558195b29940 con 0x558195c0c940 2024-04-27T00:02:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.945+0000 7f72d1000640 1 --2- 172.21.15.40:0/3726118158 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 0x558195c07200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:02:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.945+0000 7f72cb400640 1 --2- 172.21.15.40:0/3726118158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c0c940 0x558195c0edd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f72b402a9e0 tx=0x7f72b4005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c04fc628e97956f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.945+0000 7f72c8c00640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f72b00a3c30 0x7f72b00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.945+0000 7f72c1600640 1 -- 172.21.15.40:0/3726118158 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4035030 con 0x558195c0c940 2024-04-27T00:02:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.945+0000 7f72c1600640 1 -- 172.21.15.40:0/3726118158 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72b4040070 con 0x558195c0c940 2024-04-27T00:02:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.945+0000 7f72c1600640 1 -- 172.21.15.40:0/3726118158 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b40318d0 con 0x558195c0c940 2024-04-27T00:02:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.945+0000 7f72c8c00640 1 -- 172.21.15.40:0/60288084 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f72bc0d0d60 con 0x7f72cc1016b0 2024-04-27T00:02:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.946+0000 7f72d1000640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f72b00a3c30 0x7f72b00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.946+0000 7f72d5f8a740 1 -- 172.21.15.40:0/3726118158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c0c940 msgr2=0x558195c0edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.946+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/3726118158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c0c940 0x558195c0edd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f72b402a9e0 tx=0x7f72b4005c00 comp rx=0 tx=0).stop 2024-04-27T00:02:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.946+0000 7f72d5f8a740 1 -- 172.21.15.40:0/3726118158 shutdown_connections 2024-04-27T00:02:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.946+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/3726118158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c0c940 0x558195c0edd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.946+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/3726118158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 0x558195c0c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.946+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/3726118158 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 0x558195c07200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.946+0000 7f72d5f8a740 1 -- 172.21.15.40:0/3726118158 >> 172.21.15.40:0/3726118158 conn(0x558195c02450 msgr2=0x558195c02850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.946+0000 7f72d1000640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f72b00a3c30 0x7f72b00a60f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x558195c08030 tx=0x7f72c4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.946+0000 7f72d5f8a740 1 -- 172.21.15.40:0/3726118158 shutdown_connections 2024-04-27T00:02:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.946+0000 7f72d5f8a740 1 -- 172.21.15.40:0/3726118158 wait complete. 2024-04-27T00:02:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.947+0000 7f72d5f8a740 1 -- start start 2024-04-27T00:02:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.947+0000 7f72d5f8a740 1 Processor -- start 2024-04-27T00:02:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.947+0000 7f72d5f8a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 0x558195c11000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.947+0000 7f72d5f8a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c077d0 0x558195c11540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.947+0000 7f72d5f8a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c0c940 0x558195c18510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.960 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.947+0000 7f72d5f8a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558195c0f890 con 0x558195c0c940 2024-04-27T00:02:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.947+0000 7f72d5f8a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x558195af2c60 con 0x558195c06de0 2024-04-27T00:02:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.947+0000 7f72d5f8a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558195c0fa10 con 0x558195c077d0 2024-04-27T00:02:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72d5f8a740 10 client.0 _ll_register_callbacks cb 0x558195b1ee50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:02:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72cbe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 0x558195c11000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72cbe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 0x558195c11000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:54676/0 (socket says 172.21.15.40:54676) 2024-04-27T00:02:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72cbe00640 1 -- 172.21.15.40:0/4248310844 learned_addr learned my addr 172.21.15.40:0/4248310844 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72d1000640 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c0c940 0x558195c18510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72cb400640 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c077d0 0x558195c11540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72cbe00640 1 -- 172.21.15.40:0/4248310844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c077d0 msgr2=0x558195c11540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72cbe00640 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c077d0 0x558195c11540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72cbe00640 1 -- 172.21.15.40:0/4248310844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c0c940 msgr2=0x558195c18510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72cbe00640 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c0c940 0x558195c18510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72cbe00640 1 -- 172.21.15.40:0/4248310844 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558195b01510 con 0x558195c06de0 2024-04-27T00:02:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72d1000640 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c0c940 0x558195c18510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:02:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.948+0000 7f72cbe00640 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 0x558195c11000 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f72bc098eb0 tx=0x7f72bc0985d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cb7172f3dcbca67 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.949+0000 7f72ba000640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:02:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.949+0000 7f72ba000640 1 -- 172.21.15.40:0/4248310844 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc09c620 con 0x558195c06de0 2024-04-27T00:02:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.949+0000 7f72ba000640 1 -- 172.21.15.40:0/4248310844 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f72bc0a52c0 con 0x558195c06de0 2024-04-27T00:02:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.949+0000 7f72d5f8a740 1 -- 172.21.15.40:0/4248310844 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72b4040070 con 0x558195c06de0 2024-04-27T00:02:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.949+0000 7f72ba000640 1 -- 172.21.15.40:0/4248310844 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f72bc09ac20 con 0x558195c06de0 2024-04-27T00:02:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.949+0000 7f72d5f8a740 10 client.25397 fetch_fsmap learned FSMap version 24 2024-04-27T00:02:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.949+0000 7f72d5f8a740 1 -- 172.21.15.40:0/4248310844 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55819587d210 con 0x558195c06de0 2024-04-27T00:02:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.949+0000 7f72ba000640 1 -- 172.21.15.40:0/4248310844 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72b4040070 con 0x558195c06de0 2024-04-27T00:02:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.949+0000 7f72ba000640 1 -- 172.21.15.40:0/4248310844 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f72bc0987a0 con 0x558195c06de0 2024-04-27T00:02:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.950+0000 7f72d5f8a740 10 client.25397 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:02:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.950+0000 7f72d5f8a740 10 client.25397 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:02:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.950+0000 7f72d5f8a740 20 client.25397 populate_metadata read hostname 'smithi040' 2024-04-27T00:02:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.950+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558195c1cac0 0x558195c1ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.950+0000 7f72d5f8a740 4 client.25397 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:02:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.950+0000 7f72d5f8a740 1 -- 172.21.15.40:0/4248310844 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558195b78f90 con 0x558195c1cac0 2024-04-27T00:02:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.950+0000 7f72cb400640 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558195c1cac0 0x558195c1ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.951+0000 7f72cb400640 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558195c1cac0 0x558195c1ef10 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a67de8960e6cb554 server_cookie=bba0b9b4fb801af8 in_seq=0 out_seq=0 2024-04-27T00:02:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.951+0000 7f72ba000640 10 client.25397 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.951+0000 7f72ba000640 1 -- 172.21.15.40:0/4248310844 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558195b78f90 con 0x558195c1cac0 2024-04-27T00:02:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.951+0000 7f72ba000640 10 client.25397 handle_command_reply: tid=0 2024-04-27T00:02:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.951+0000 7f72d5f8a740 1 client.25397 shutdown 2024-04-27T00:02:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.952+0000 7f72d5f8a740 1 -- 172.21.15.40:0/4248310844 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 msgr2=0x558195c11000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.952+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 0x558195c11000 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f72bc098eb0 tx=0x7f72bc0985d0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.952+0000 7f72d5f8a740 1 -- 172.21.15.40:0/4248310844 shutdown_connections 2024-04-27T00:02:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.952+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558195c1cac0 0x558195c1ef10 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.952+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c0c940 0x558195c18510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.952+0000 7f72ba000640 0 client.25397 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.952+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c077d0 0x558195c11540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.952+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/4248310844 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c06de0 0x558195c11000 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.952+0000 7f72d5f8a740 1 -- 172.21.15.40:0/4248310844 >> 172.21.15.40:0/4248310844 conn(0x558195c02450 msgr2=0x558195c01650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.952+0000 7f72d5f8a740 1 -- 172.21.15.40:0/4248310844 shutdown_connections 2024-04-27T00:02:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.952+0000 7f72d5f8a740 1 -- 172.21.15.40:0/4248310844 wait complete. 2024-04-27T00:02:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.952+0000 7f72d5f8a740 20 client.25397 trim_cache size 0 max 16384 2024-04-27T00:02:01.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.968+0000 7f72d5f8a740 1 Processor -- start 2024-04-27T00:02:01.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.968+0000 7f72d5f8a740 1 -- start start 2024-04-27T00:02:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72d5f8a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c06de0 0x558195c05400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72d5f8a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 0x558195c184d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72d5f8a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c0c940 0x558195c05940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72d5f8a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc1afe50 con 0x558195c077d0 2024-04-27T00:02:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72d5f8a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc1ad720 con 0x558195c0c940 2024-04-27T00:02:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72d5f8a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc1ab570 con 0x558195c06de0 2024-04-27T00:02:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72cb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 0x558195c184d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72cb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 0x558195c184d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39376/0 (socket says 172.21.15.40:39376) 2024-04-27T00:02:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72cb400640 1 -- 172.21.15.40:0/3765859534 learned_addr learned my addr 172.21.15.40:0/3765859534 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72d1000640 1 --2- 172.21.15.40:0/3765859534 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c0c940 0x558195c05940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72cbe00640 1 --2- 172.21.15.40:0/3765859534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c06de0 0x558195c05400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72cb400640 1 -- 172.21.15.40:0/3765859534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c06de0 msgr2=0x558195c05400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72cb400640 1 --2- 172.21.15.40:0/3765859534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c06de0 0x558195c05400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72cb400640 1 -- 172.21.15.40:0/3765859534 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c0c940 msgr2=0x558195c05940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72cb400640 1 --2- 172.21.15.40:0/3765859534 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c0c940 0x558195c05940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72cb400640 1 -- 172.21.15.40:0/3765859534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558195949620 con 0x558195c077d0 2024-04-27T00:02:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.969+0000 7f72d1000640 1 --2- 172.21.15.40:0/3765859534 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c0c940 0x558195c05940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:02:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.970+0000 7f72cb400640 1 --2- 172.21.15.40:0/3765859534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 0x558195c184d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009900 tx=0x7f72b4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8151a7962c5878 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.970+0000 7f72c0c00640 1 -- 172.21.15.40:0/3765859534 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4057070 con 0x558195c077d0 2024-04-27T00:02:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.970+0000 7f72c0c00640 1 -- 172.21.15.40:0/3765859534 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72b4040040 con 0x558195c077d0 2024-04-27T00:02:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.970+0000 7f72c0c00640 1 -- 172.21.15.40:0/3765859534 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4035040 con 0x558195c077d0 2024-04-27T00:02:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.971+0000 7f72d5f8a740 1 -- 172.21.15.40:0/3765859534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 msgr2=0x558195c184d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.971+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/3765859534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 0x558195c184d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009900 tx=0x7f72b4009260 comp rx=0 tx=0).stop 2024-04-27T00:02:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.971+0000 7f72d5f8a740 1 -- 172.21.15.40:0/3765859534 shutdown_connections 2024-04-27T00:02:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.971+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/3765859534 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c0c940 0x558195c05940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.971+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/3765859534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c077d0 0x558195c184d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.971+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/3765859534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c06de0 0x558195c05400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.971+0000 7f72d5f8a740 1 -- 172.21.15.40:0/3765859534 >> 172.21.15.40:0/3765859534 conn(0x558195c02450 msgr2=0x558195c02830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.971+0000 7f72d5f8a740 1 -- 172.21.15.40:0/3765859534 shutdown_connections 2024-04-27T00:02:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.971+0000 7f72d5f8a740 1 -- 172.21.15.40:0/3765859534 wait complete. 2024-04-27T00:02:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.972+0000 7f72d5f8a740 1 -- start start 2024-04-27T00:02:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.972+0000 7f72d5f8a740 1 Processor -- start 2024-04-27T00:02:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.972+0000 7f72d5f8a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c06de0 0x558195c155d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.972+0000 7f72d5f8a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c077d0 0x558195c15b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.972+0000 7f72d5f8a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c0c940 0x558195c16b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.972+0000 7f72d5f8a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc1abbe0 con 0x558195c06de0 2024-04-27T00:02:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.972+0000 7f72d5f8a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc1ad050 con 0x558195c077d0 2024-04-27T00:02:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.972+0000 7f72d5f8a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc1ac2b0 con 0x558195c0c940 2024-04-27T00:02:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.972+0000 7f72d5f8a740 10 client.0 _ll_register_callbacks cb 0x7f72bc0d4500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:02:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.972+0000 7f72cb400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c077d0 0x558195c15b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.972+0000 7f72cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c06de0 0x558195c155d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.973+0000 7f72cb400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c077d0 0x558195c15b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:54692/0 (socket says 172.21.15.40:54692) 2024-04-27T00:02:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.973+0000 7f72cb400640 1 -- 172.21.15.40:0/2711180330 learned_addr learned my addr 172.21.15.40:0/2711180330 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.973+0000 7f72d1000640 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c0c940 0x558195c16b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.973+0000 7f72cb400640 1 -- 172.21.15.40:0/2711180330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c0c940 msgr2=0x558195c16b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.973+0000 7f72cb400640 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c0c940 0x558195c16b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.973+0000 7f72cb400640 1 -- 172.21.15.40:0/2711180330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c06de0 msgr2=0x558195c155d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.973+0000 7f72cb400640 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c06de0 0x558195c155d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.973+0000 7f72cb400640 1 -- 172.21.15.40:0/2711180330 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558195b25cb0 con 0x558195c077d0 2024-04-27T00:02:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.973+0000 7f72d1000640 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c0c940 0x558195c16b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:02:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.973+0000 7f72cbe00640 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c06de0 0x558195c155d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:02:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.974+0000 7f72cb400640 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c077d0 0x558195c15b10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f72b4005c40 tx=0x7f72b4031710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f64d7816d6f5ada server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.974+0000 7f72ba000640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:02:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.974+0000 7f72ba000640 1 -- 172.21.15.40:0/2711180330 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4035070 con 0x558195c077d0 2024-04-27T00:02:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.974+0000 7f72ba000640 1 -- 172.21.15.40:0/2711180330 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f72b4052370 con 0x558195c077d0 2024-04-27T00:02:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.974+0000 7f72d5f8a740 1 -- 172.21.15.40:0/2711180330 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72b4040040 con 0x558195c077d0 2024-04-27T00:02:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.974+0000 7f72ba000640 1 -- 172.21.15.40:0/2711180330 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f72b4057070 con 0x558195c077d0 2024-04-27T00:02:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.974+0000 7f72d5f8a740 10 client.25403 fetch_fsmap learned FSMap version 24 2024-04-27T00:02:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.974+0000 7f72d5f8a740 1 -- 172.21.15.40:0/2711180330 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558195c18250 con 0x558195c077d0 2024-04-27T00:02:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.975+0000 7f72ba000640 1 -- 172.21.15.40:0/2711180330 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72b4040040 con 0x558195c077d0 2024-04-27T00:02:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.975+0000 7f72ba000640 1 -- 172.21.15.40:0/2711180330 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f72b404e040 con 0x558195c077d0 2024-04-27T00:02:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.975+0000 7f72d5f8a740 10 client.25403 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:02:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.975+0000 7f72d5f8a740 10 client.25403 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:02:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.975+0000 7f72d5f8a740 20 client.25403 populate_metadata read hostname 'smithi040' 2024-04-27T00:02:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.975+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558195c83090 0x558195c85480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.975+0000 7f72d5f8a740 4 client.25403 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:02:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.975+0000 7f72d5f8a740 1 -- 172.21.15.40:0/2711180330 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558195b72e00 con 0x558195c83090 2024-04-27T00:02:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.976+0000 7f72cbe00640 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558195c83090 0x558195c85480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.976+0000 7f72cbe00640 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558195c83090 0x558195c85480 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3339dfb5ec358474 server_cookie=80c0d21e63446648 in_seq=0 out_seq=0 2024-04-27T00:02:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.976+0000 7f72ba000640 10 client.25403 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.990+0000 7f72ba000640 1 -- 172.21.15.40:0/2711180330 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558195b72e00 con 0x558195c83090 2024-04-27T00:02:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.990+0000 7f72ba000640 10 client.25403 handle_command_reply: tid=0 2024-04-27T00:02:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.990+0000 7f72d5f8a740 1 client.25403 shutdown 2024-04-27T00:02:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.990+0000 7f72d5f8a740 1 -- 172.21.15.40:0/2711180330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c077d0 msgr2=0x558195c15b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.990+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c077d0 0x558195c15b10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f72b4005c40 tx=0x7f72b4031710 comp rx=0 tx=0).stop 2024-04-27T00:02:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.990+0000 7f72d5f8a740 1 -- 172.21.15.40:0/2711180330 shutdown_connections 2024-04-27T00:02:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.990+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558195c83090 0x558195c85480 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.990+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558195c0c940 0x558195c16b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.990+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558195c077d0 0x558195c15b10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.990+0000 7f72d5f8a740 1 --2- 172.21.15.40:0/2711180330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558195c06de0 0x558195c155d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.990+0000 7f72ba000640 0 client.25403 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.990+0000 7f72d5f8a740 1 -- 172.21.15.40:0/2711180330 >> 172.21.15.40:0/2711180330 conn(0x558195c02450 msgr2=0x558195c05400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.991+0000 7f72d5f8a740 1 -- 172.21.15.40:0/2711180330 shutdown_connections 2024-04-27T00:02:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.991+0000 7f72d5f8a740 1 -- 172.21.15.40:0/2711180330 wait complete. 2024-04-27T00:02:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.991+0000 7f72d5f8a740 20 client.25403 trim_cache size 0 max 16384 2024-04-27T00:02:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:02:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:02:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:02:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:02:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.991+0000 7f72d1a00640 1 -- 172.21.15.40:0/60288084 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f72b00a3c30 msgr2=0x7f72b00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.991+0000 7f72d1a00640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f72b00a3c30 0x7f72b00a60f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x558195c08030 tx=0x7f72c4009500 comp rx=0 tx=0).stop 2024-04-27T00:02:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.991+0000 7f72d1a00640 1 -- 172.21.15.40:0/60288084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72cc1016b0 msgr2=0x7f72cc19acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.991+0000 7f72d1a00640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72cc1016b0 0x7f72cc19acc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00e9c0 tx=0x7f72bc00ee80 comp rx=0 tx=0).stop 2024-04-27T00:02:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.992+0000 7f72d1a00640 1 -- 172.21.15.40:0/60288084 shutdown_connections 2024-04-27T00:02:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.992+0000 7f72d1a00640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f72b00a3c30 0x7f72b00a60f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.992+0000 7f72d1a00640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72cc103260 0x7f72cc1a21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.992+0000 7f72d1a00640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f72cc1028a0 0x7f72cc19b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.992+0000 7f72d1a00640 1 --2- 172.21.15.40:0/60288084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72cc1016b0 0x7f72cc19acc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.992+0000 7f72d1a00640 1 -- 172.21.15.40:0/60288084 >> 172.21.15.40:0/60288084 conn(0x7f72cc0fcc20 msgr2=0x7f72cc0fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.992+0000 7f72d1a00640 1 -- 172.21.15.40:0/60288084 shutdown_connections 2024-04-27T00:02:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:01.992+0000 7f72d1a00640 1 -- 172.21.15.40:0/60288084 wait complete. 2024-04-27T00:02:02.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:01 smithi040 ceph-mon[28095]: pgmap v422: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-27T00:02:02.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:01 smithi080 ceph-mon[32362]: pgmap v422: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-27T00:02:02.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:01 smithi103 ceph-mon[32952]: pgmap v422: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-27T00:02:04.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:03 smithi040 ceph-mon[28095]: pgmap v423: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-27T00:02:04.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:03 smithi080 ceph-mon[32362]: pgmap v423: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-27T00:02:04.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:03 smithi103 ceph-mon[32952]: pgmap v423: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-27T00:02:06.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:05 smithi040 ceph-mon[28095]: pgmap v424: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-27T00:02:06.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:05 smithi080 ceph-mon[32362]: pgmap v424: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-27T00:02:06.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:05 smithi103 ceph-mon[32952]: pgmap v424: 129 pgs: 129 active+clean; 647 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-27T00:02:08.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:07 smithi040 ceph-mon[28095]: pgmap v425: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-27T00:02:08.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:07 smithi080 ceph-mon[32362]: pgmap v425: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-27T00:02:08.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:07 smithi103 ceph-mon[32952]: pgmap v425: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-27T00:02:10.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:09 smithi040 ceph-mon[28095]: pgmap v426: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-27T00:02:10.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:09 smithi080 ceph-mon[32362]: pgmap v426: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-27T00:02:10.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:09 smithi103 ceph-mon[32952]: pgmap v426: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-27T00:02:12.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:11 smithi040 ceph-mon[28095]: pgmap v427: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-27T00:02:12.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:11 smithi080 ceph-mon[32362]: pgmap v427: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-27T00:02:12.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:11 smithi103 ceph-mon[32952]: pgmap v427: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-27T00:02:14.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:13 smithi040 ceph-mon[28095]: pgmap v428: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-04-27T00:02:14.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:13 smithi080 ceph-mon[32362]: pgmap v428: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-04-27T00:02:14.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:13 smithi103 ceph-mon[32952]: pgmap v428: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-04-27T00:02:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.138+0000 7fc68a000640 1 -- 172.21.15.40:0/3994807817 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc684075390 msgr2=0x7fc684075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.138+0000 7fc68a000640 1 --2- 172.21.15.40:0/3994807817 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc684075390 0x7fc684075810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc670009930 tx=0x7fc67002a420 comp rx=0 tx=0).stop 2024-04-27T00:02:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.138+0000 7fc68a000640 1 -- 172.21.15.40:0/3994807817 shutdown_connections 2024-04-27T00:02:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.138+0000 7fc68a000640 1 --2- 172.21.15.40:0/3994807817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc68410ef60 0x7fc684111350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.138+0000 7fc68a000640 1 --2- 172.21.15.40:0/3994807817 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc684075390 0x7fc684075810 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.138+0000 7fc68a000640 1 --2- 172.21.15.40:0/3994807817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc684076970 0x7fc684074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.138+0000 7fc68a000640 1 -- 172.21.15.40:0/3994807817 >> 172.21.15.40:0/3994807817 conn(0x7fc6840fdf20 msgr2=0x7fc684100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.138+0000 7fc68a000640 1 -- 172.21.15.40:0/3994807817 shutdown_connections 2024-04-27T00:02:15.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.139+0000 7fc68a000640 1 -- 172.21.15.40:0/3994807817 wait complete. 2024-04-27T00:02:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.140+0000 7fc68a000640 1 Processor -- start 2024-04-27T00:02:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.140+0000 7fc68a000640 1 -- start start 2024-04-27T00:02:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.140+0000 7fc68a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc684075390 0x7fc68419f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.140+0000 7fc68a000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc684076970 0x7fc68419f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.143 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.140+0000 7fc68a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc68410ef60 0x7fc6841a66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.140+0000 7fc68a000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc684114b00 con 0x7fc684075390 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.140+0000 7fc68a000640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fc684114980 con 0x7fc684076970 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.140+0000 7fc68a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc684114c80 con 0x7fc68410ef60 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.141+0000 7fc682a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc684076970 0x7fc68419f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.141+0000 7fc683400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc684075390 0x7fc68419f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.141+0000 7fc682a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc684076970 0x7fc68419f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:57518/0 (socket says 172.21.15.40:57518) 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.141+0000 7fc682a00640 1 -- 172.21.15.40:0/1707655048 learned_addr learned my addr 172.21.15.40:0/1707655048 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.141+0000 7fc683e00640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc68410ef60 0x7fc6841a66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.141+0000 7fc682a00640 1 -- 172.21.15.40:0/1707655048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc68410ef60 msgr2=0x7fc6841a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.141+0000 7fc682a00640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc68410ef60 0x7fc6841a66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.141+0000 7fc682a00640 1 -- 172.21.15.40:0/1707655048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc684075390 msgr2=0x7fc68419f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.141+0000 7fc682a00640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc684075390 0x7fc68419f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.141+0000 7fc682a00640 1 -- 172.21.15.40:0/1707655048 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6841a6d40 con 0x7fc684076970 2024-04-27T00:02:15.144 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.142+0000 7fc682a00640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc684076970 0x7fc68419f660 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc670031500 tx=0x7fc670003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.142+0000 7fc67be00640 1 -- 172.21.15.40:0/1707655048 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67003f070 con 0x7fc684076970 2024-04-27T00:02:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.142+0000 7fc67be00640 1 -- 172.21.15.40:0/1707655048 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc670031c50 con 0x7fc684076970 2024-04-27T00:02:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.142+0000 7fc68a000640 1 -- 172.21.15.40:0/1707655048 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6841a6fd0 con 0x7fc684076970 2024-04-27T00:02:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.142+0000 7fc67be00640 1 -- 172.21.15.40:0/1707655048 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670035030 con 0x7fc684076970 2024-04-27T00:02:15.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.142+0000 7fc68a000640 1 -- 172.21.15.40:0/1707655048 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6841a7430 con 0x7fc684076970 2024-04-27T00:02:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.145+0000 7fc68cd95740 1 Processor -- start 2024-04-27T00:02:15.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.146+0000 7fc68cd95740 1 -- start start 2024-04-27T00:02:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.146+0000 7fc68cd95740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d70bd0 0x55e558d70ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.146+0000 7fc68cd95740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d715c0 0x55e558d760c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.146+0000 7fc68cd95740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d76730 0x55e558d78bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.146+0000 7fc68cd95740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e558cd5380 con 0x55e558d70bd0 2024-04-27T00:02:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.146+0000 7fc68cd95740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55e558c71c30 con 0x55e558d76730 2024-04-27T00:02:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.146+0000 7fc68cd95740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e558cd1340 con 0x55e558d715c0 2024-04-27T00:02:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.146+0000 7fc683e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d70bd0 0x55e558d70ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.146+0000 7fc683400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d715c0 0x55e558d760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.146+0000 7fc683e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d70bd0 0x55e558d70ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45416/0 (socket says 172.21.15.40:45416) 2024-04-27T00:02:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.146+0000 7fc683e00640 1 -- 172.21.15.40:0/3832429375 learned_addr learned my addr 172.21.15.40:0/3832429375 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.147+0000 7fc683400640 1 -- 172.21.15.40:0/3832429375 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d76730 msgr2=0x55e558d78bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-27T00:02:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.147+0000 7fc683400640 1 --2- 172.21.15.40:0/3832429375 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d76730 0x55e558d78bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.147+0000 7fc683400640 1 -- 172.21.15.40:0/3832429375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d70bd0 msgr2=0x55e558d70ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.147+0000 7fc683400640 1 --2- 172.21.15.40:0/3832429375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d70bd0 0x55e558d70ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.147+0000 7fc683400640 1 -- 172.21.15.40:0/3832429375 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e558c93940 con 0x55e558d715c0 2024-04-27T00:02:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.147+0000 7fc683e00640 1 --2- 172.21.15.40:0/3832429375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d70bd0 0x55e558d70ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:02:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.147+0000 7fc67be00640 1 -- 172.21.15.40:0/1707655048 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc670042070 con 0x7fc684076970 2024-04-27T00:02:15.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.147+0000 7fc683400640 1 --2- 172.21.15.40:0/3832429375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d715c0 0x55e558d760c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc674009c00 tx=0x7fc674011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e33c49438913e3f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.147+0000 7fc66fe00640 1 -- 172.21.15.40:0/3832429375 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674004230 con 0x55e558d715c0 2024-04-27T00:02:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.147+0000 7fc66fe00640 1 -- 172.21.15.40:0/3832429375 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6740043d0 con 0x55e558d715c0 2024-04-27T00:02:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.147+0000 7fc66fe00640 1 -- 172.21.15.40:0/3832429375 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674004c70 con 0x55e558d715c0 2024-04-27T00:02:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.148+0000 7fc68cd95740 1 -- 172.21.15.40:0/3832429375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d715c0 msgr2=0x55e558d760c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.148+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3832429375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d715c0 0x55e558d760c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc674009c00 tx=0x7fc674011810 comp rx=0 tx=0).stop 2024-04-27T00:02:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.148+0000 7fc68cd95740 1 -- 172.21.15.40:0/3832429375 shutdown_connections 2024-04-27T00:02:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.148+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3832429375 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d76730 0x55e558d78bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.148+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3832429375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d715c0 0x55e558d760c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.148+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3832429375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d70bd0 0x55e558d70ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.148+0000 7fc68cd95740 1 -- 172.21.15.40:0/3832429375 >> 172.21.15.40:0/3832429375 conn(0x55e558d6c240 msgr2=0x55e558d6c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.148+0000 7fc68cd95740 1 -- 172.21.15.40:0/3832429375 shutdown_connections 2024-04-27T00:02:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.148+0000 7fc68cd95740 1 -- 172.21.15.40:0/3832429375 wait complete. 2024-04-27T00:02:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.148+0000 7fc67be00640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6680a3c00 0x7fc6680a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.149+0000 7fc67be00640 1 -- 172.21.15.40:0/1707655048 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fc6700f10e0 con 0x7fc684076970 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.149+0000 7fc683400640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6680a3c00 0x7fc6680a60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.149+0000 7fc68cd95740 1 -- start start 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.149+0000 7fc68cd95740 1 Processor -- start 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc68cd95740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d70bd0 0x55e558d7ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc683400640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6680a3c00 0x7fc6680a60c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc674009bd0 tx=0x7fc674009890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc68cd95740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d7b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc68cd95740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d76730 0x55e558d82390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc68cd95740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e558d796e0 con 0x55e558d715c0 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc68cd95740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55e558c5cc60 con 0x55e558d70bd0 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc68cd95740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e558d79860 con 0x55e558d76730 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc68cd95740 10 client.0 _ll_register_callbacks cb 0x55e558c88e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc683400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d70bd0 0x55e558d7ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc682a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d7b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc682a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d7b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45428/0 (socket says 172.21.15.40:45428) 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc682a00640 1 -- 172.21.15.40:0/3000781050 learned_addr learned my addr 172.21.15.40:0/3000781050 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc683e00640 1 --2- 172.21.15.40:0/3000781050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d76730 0x55e558d82390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc682a00640 1 -- 172.21.15.40:0/3000781050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d76730 msgr2=0x55e558d82390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc682a00640 1 --2- 172.21.15.40:0/3000781050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d76730 0x55e558d82390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.150+0000 7fc682a00640 1 -- 172.21.15.40:0/3000781050 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d70bd0 msgr2=0x55e558d7ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.151+0000 7fc682a00640 1 --2- 172.21.15.40:0/3000781050 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d70bd0 0x55e558d7ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.151+0000 7fc682a00640 1 -- 172.21.15.40:0/3000781050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e558c6b510 con 0x55e558d715c0 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.151+0000 7fc682a00640 1 --2- 172.21.15.40:0/3000781050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d7b350 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc6700bcc10 tx=0x7fc670031f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d63541162bb6c038 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.151+0000 7fc66cc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.151+0000 7fc66cc00640 1 -- 172.21.15.40:0/3000781050 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670049040 con 0x55e558d715c0 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.151+0000 7fc66cc00640 1 -- 172.21.15.40:0/3000781050 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fc6700c5640 con 0x55e558d715c0 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.151+0000 7fc68cd95740 1 -- 172.21.15.40:0/3000781050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6740043d0 con 0x55e558d715c0 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.151+0000 7fc66cc00640 1 -- 172.21.15.40:0/3000781050 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc6700c5840 con 0x55e558d715c0 2024-04-27T00:02:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.151+0000 7fc68cd95740 10 client.15288 fetch_fsmap learned FSMap version 24 2024-04-27T00:02:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.151+0000 7fc68cd95740 1 -- 172.21.15.40:0/3000781050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e5589e7210 con 0x55e558d715c0 2024-04-27T00:02:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.152+0000 7fc66cc00640 1 -- 172.21.15.40:0/3000781050 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6740043d0 con 0x55e558d715c0 2024-04-27T00:02:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.152+0000 7fc66cc00640 1 -- 172.21.15.40:0/3000781050 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc6700bc360 con 0x55e558d715c0 2024-04-27T00:02:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.152+0000 7fc68cd95740 10 client.15288 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:02:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.152+0000 7fc68cd95740 10 client.15288 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:02:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.152+0000 7fc68cd95740 20 client.15288 populate_metadata read hostname 'smithi040' 2024-04-27T00:02:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.152+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3000781050 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e558d86880 0x55e558d88c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.152+0000 7fc68cd95740 4 client.15288 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:02:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.152+0000 7fc68cd95740 1 -- 172.21.15.40:0/3000781050 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e558ce2f90 con 0x55e558d86880 2024-04-27T00:02:15.157 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.152+0000 7fc683400640 1 --2- 172.21.15.40:0/3000781050 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e558d86880 0x55e558d88c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.153+0000 7fc683400640 1 --2- 172.21.15.40:0/3000781050 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e558d86880 0x55e558d88c70 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc0ed28f920badc8 server_cookie=5b2299898c932253 in_seq=0 out_seq=0 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.153+0000 7fc66cc00640 10 client.15288 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.154+0000 7fc66cc00640 1 -- 172.21.15.40:0/3000781050 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e558ce2f90 con 0x55e558d86880 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.154+0000 7fc66cc00640 10 client.15288 handle_command_reply: tid=0 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.154+0000 7fc68cd95740 1 client.15288 shutdown 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.154+0000 7fc68cd95740 1 -- 172.21.15.40:0/3000781050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 msgr2=0x55e558d7b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.154+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3000781050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d7b350 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc6700bcc10 tx=0x7fc670031f40 comp rx=0 tx=0).stop 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.154+0000 7fc68cd95740 1 -- 172.21.15.40:0/3000781050 shutdown_connections 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.154+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3000781050 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e558d86880 0x55e558d88c70 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.154+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3000781050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d76730 0x55e558d82390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.154+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3000781050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d7b350 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.154+0000 7fc66cc00640 0 client.15288 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.154+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3000781050 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d70bd0 0x55e558d7ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.154+0000 7fc68cd95740 1 -- 172.21.15.40:0/3000781050 >> 172.21.15.40:0/3000781050 conn(0x55e558d6c240 msgr2=0x55e558d6b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.155+0000 7fc68cd95740 1 -- 172.21.15.40:0/3000781050 shutdown_connections 2024-04-27T00:02:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.155+0000 7fc68cd95740 1 -- 172.21.15.40:0/3000781050 wait complete. 2024-04-27T00:02:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.155+0000 7fc68cd95740 20 client.15288 trim_cache size 0 max 16384 2024-04-27T00:02:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.171+0000 7fc68cd95740 1 Processor -- start 2024-04-27T00:02:15.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.171+0000 7fc68cd95740 1 -- start start 2024-04-27T00:02:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.171+0000 7fc68cd95740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d70bd0 0x55e558d6bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.171+0000 7fc68cd95740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d82350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.171+0000 7fc68cd95740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d76730 0x55e558d6f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc68cd95740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841b41e0 con 0x55e558d715c0 2024-04-27T00:02:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc68cd95740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841b1ab0 con 0x55e558d76730 2024-04-27T00:02:15.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc68cd95740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841af900 con 0x55e558d70bd0 2024-04-27T00:02:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc682a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d82350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc682a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d82350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45444/0 (socket says 172.21.15.40:45444) 2024-04-27T00:02:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc682a00640 1 -- 172.21.15.40:0/165694555 learned_addr learned my addr 172.21.15.40:0/165694555 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc683e00640 1 --2- 172.21.15.40:0/165694555 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d76730 0x55e558d6f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc683400640 1 --2- 172.21.15.40:0/165694555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d70bd0 0x55e558d6bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc682a00640 1 -- 172.21.15.40:0/165694555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d70bd0 msgr2=0x55e558d6bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc682a00640 1 --2- 172.21.15.40:0/165694555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d70bd0 0x55e558d6bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc682a00640 1 -- 172.21.15.40:0/165694555 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d76730 msgr2=0x55e558d6f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc682a00640 1 --2- 172.21.15.40:0/165694555 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d76730 0x55e558d6f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.172+0000 7fc682a00640 1 -- 172.21.15.40:0/165694555 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e558d67940 con 0x55e558d715c0 2024-04-27T00:02:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.173+0000 7fc682a00640 1 --2- 172.21.15.40:0/165694555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d82350 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc6700bb1c0 tx=0x7fc67003b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95f013653185aa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.173+0000 7fc66f400640 1 -- 172.21.15.40:0/165694555 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670049070 con 0x55e558d715c0 2024-04-27T00:02:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.173+0000 7fc66f400640 1 -- 172.21.15.40:0/165694555 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6700bc360 con 0x55e558d715c0 2024-04-27T00:02:15.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.173+0000 7fc66f400640 1 -- 172.21.15.40:0/165694555 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6700d6200 con 0x55e558d715c0 2024-04-27T00:02:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.173+0000 7fc68cd95740 1 -- 172.21.15.40:0/165694555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 msgr2=0x55e558d82350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.173+0000 7fc68cd95740 1 --2- 172.21.15.40:0/165694555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d82350 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc6700bb1c0 tx=0x7fc67003b8e0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.174+0000 7fc68cd95740 1 -- 172.21.15.40:0/165694555 shutdown_connections 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.174+0000 7fc68cd95740 1 --2- 172.21.15.40:0/165694555 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d76730 0x55e558d6f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.174+0000 7fc68cd95740 1 --2- 172.21.15.40:0/165694555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d82350 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.174+0000 7fc68cd95740 1 --2- 172.21.15.40:0/165694555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d70bd0 0x55e558d6bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.174+0000 7fc68cd95740 1 -- 172.21.15.40:0/165694555 >> 172.21.15.40:0/165694555 conn(0x55e558d6c240 msgr2=0x55e558d6f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.174+0000 7fc68cd95740 1 -- 172.21.15.40:0/165694555 shutdown_connections 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.174+0000 7fc68cd95740 1 -- 172.21.15.40:0/165694555 wait complete. 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.174+0000 7fc68cd95740 1 -- start start 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.174+0000 7fc68cd95740 1 Processor -- start 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc68cd95740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d70bd0 0x55e558d7f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc68cd95740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d7ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc68cd95740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d76730 0x55e558d80d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc68cd95740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841aff70 con 0x55e558d715c0 2024-04-27T00:02:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc68cd95740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841b13e0 con 0x55e558d70bd0 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc68cd95740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841b0640 con 0x55e558d76730 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc68cd95740 10 client.0 _ll_register_callbacks cb 0x7fc6700bc870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc682a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d7ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc682a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d7ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45450/0 (socket says 172.21.15.40:45450) 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc682a00640 1 -- 172.21.15.40:0/3137793850 learned_addr learned my addr 172.21.15.40:0/3137793850 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc683400640 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d70bd0 0x55e558d7f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc683e00640 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d76730 0x55e558d80d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc682a00640 1 -- 172.21.15.40:0/3137793850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d76730 msgr2=0x55e558d80d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.175+0000 7fc682a00640 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d76730 0x55e558d80d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.176+0000 7fc682a00640 1 -- 172.21.15.40:0/3137793850 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d70bd0 msgr2=0x55e558d7f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.176+0000 7fc682a00640 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d70bd0 0x55e558d7f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.176+0000 7fc682a00640 1 -- 172.21.15.40:0/3137793850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e558ab3620 con 0x55e558d715c0 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.176+0000 7fc682a00640 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d7ff10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc6700b98e0 tx=0x7fc6700bcb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c9740e769f51be server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:15.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.176+0000 7fc66cc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.176+0000 7fc66cc00640 1 -- 172.21.15.40:0/3137793850 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670049050 con 0x55e558d715c0 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.176+0000 7fc66cc00640 1 -- 172.21.15.40:0/3137793850 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fc6700b8da0 con 0x55e558d715c0 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.176+0000 7fc68cd95740 1 -- 172.21.15.40:0/3137793850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6700bc360 con 0x55e558d715c0 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.176+0000 7fc66cc00640 1 -- 172.21.15.40:0/3137793850 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc6700b8fe0 con 0x55e558d715c0 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.176+0000 7fc68cd95740 10 client.15294 fetch_fsmap learned FSMap version 24 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.176+0000 7fc68cd95740 1 -- 172.21.15.40:0/3137793850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e558c8fcb0 con 0x55e558d715c0 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.177+0000 7fc66cc00640 1 -- 172.21.15.40:0/3137793850 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6700bc360 con 0x55e558d715c0 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.177+0000 7fc66cc00640 1 -- 172.21.15.40:0/3137793850 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc6700a60d0 con 0x55e558d715c0 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.177+0000 7fc68cd95740 10 client.15294 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.177+0000 7fc68cd95740 10 client.15294 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.177+0000 7fc68cd95740 20 client.15294 populate_metadata read hostname 'smithi040' 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.177+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e558decdc0 0x55e558def1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.177+0000 7fc68cd95740 4 client.15294 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:02:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.177+0000 7fc68cd95740 1 -- 172.21.15.40:0/3137793850 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e558cdce00 con 0x55e558decdc0 2024-04-27T00:02:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.177+0000 7fc683400640 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e558decdc0 0x55e558def1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.178+0000 7fc683400640 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e558decdc0 0x55e558def1b0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=311d8554cdc3bf40 server_cookie=17131af0724f3457 in_seq=0 out_seq=0 2024-04-27T00:02:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.178+0000 7fc66cc00640 10 client.15294 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.181+0000 7fc66cc00640 1 -- 172.21.15.40:0/3137793850 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e558cdce00 con 0x55e558decdc0 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.181+0000 7fc66cc00640 10 client.15294 handle_command_reply: tid=0 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.181+0000 7fc68cd95740 1 client.15294 shutdown 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.182+0000 7fc68cd95740 1 -- 172.21.15.40:0/3137793850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 msgr2=0x55e558d7ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.182+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d7ff10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc6700b98e0 tx=0x7fc6700bcb20 comp rx=0 tx=0).stop 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.182+0000 7fc68cd95740 1 -- 172.21.15.40:0/3137793850 shutdown_connections 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.182+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e558decdc0 0x55e558def1b0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.182+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e558d76730 0x55e558d80d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.182+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e558d715c0 0x55e558d7ff10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.182+0000 7fc68cd95740 1 --2- 172.21.15.40:0/3137793850 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e558d70bd0 0x55e558d7f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.182+0000 7fc66cc00640 0 client.15294 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.182+0000 7fc68cd95740 1 -- 172.21.15.40:0/3137793850 >> 172.21.15.40:0/3137793850 conn(0x55e558d6c240 msgr2=0x55e558d6abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.182+0000 7fc68cd95740 1 -- 172.21.15.40:0/3137793850 shutdown_connections 2024-04-27T00:02:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.182+0000 7fc68cd95740 1 -- 172.21.15.40:0/3137793850 wait complete. 2024-04-27T00:02:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.182+0000 7fc68cd95740 20 client.15294 trim_cache size 0 max 16384 2024-04-27T00:02:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:02:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:02:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:02:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:02:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.183+0000 7fc68a000640 1 -- 172.21.15.40:0/1707655048 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6680a3c00 msgr2=0x7fc6680a60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.183+0000 7fc68a000640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6680a3c00 0x7fc6680a60c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc674009bd0 tx=0x7fc674009890 comp rx=0 tx=0).stop 2024-04-27T00:02:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.183+0000 7fc68a000640 1 -- 172.21.15.40:0/1707655048 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc684076970 msgr2=0x7fc68419f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.183+0000 7fc68a000640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc684076970 0x7fc68419f660 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc670031500 tx=0x7fc670003c10 comp rx=0 tx=0).stop 2024-04-27T00:02:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.183+0000 7fc68a000640 1 -- 172.21.15.40:0/1707655048 shutdown_connections 2024-04-27T00:02:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.183+0000 7fc68a000640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6680a3c00 0x7fc6680a60c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.183+0000 7fc68a000640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc68410ef60 0x7fc6841a66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.183+0000 7fc68a000640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc684076970 0x7fc68419f660 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.183+0000 7fc68a000640 1 --2- 172.21.15.40:0/1707655048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc684075390 0x7fc68419f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.183+0000 7fc68a000640 1 -- 172.21.15.40:0/1707655048 >> 172.21.15.40:0/1707655048 conn(0x7fc6840fdf20 msgr2=0x7fc6841021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:15.188 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.183+0000 7fc68a000640 1 -- 172.21.15.40:0/1707655048 shutdown_connections 2024-04-27T00:02:15.188 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:15.183+0000 7fc68a000640 1 -- 172.21.15.40:0/1707655048 wait complete. 2024-04-27T00:02:16.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:15 smithi040 ceph-mon[28095]: pgmap v429: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-27T00:02:16.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:15 smithi080 ceph-mon[32362]: pgmap v429: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-27T00:02:16.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:15 smithi103 ceph-mon[32952]: pgmap v429: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-27T00:02:18.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:17 smithi040 ceph-mon[28095]: pgmap v430: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-27T00:02:18.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:17 smithi080 ceph-mon[32362]: pgmap v430: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-27T00:02:18.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:17 smithi103 ceph-mon[32952]: pgmap v430: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-27T00:02:20.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:19 smithi040 ceph-mon[28095]: pgmap v431: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:20.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:19 smithi080 ceph-mon[32362]: pgmap v431: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:20.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:19 smithi103 ceph-mon[32952]: pgmap v431: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:22.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:21 smithi040 ceph-mon[28095]: pgmap v432: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:22.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:21 smithi080 ceph-mon[32362]: pgmap v432: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:22.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:21 smithi103 ceph-mon[32952]: pgmap v432: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:24.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:23 smithi040 ceph-mon[28095]: pgmap v433: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:02:24.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:23 smithi080 ceph-mon[32362]: pgmap v433: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:02:24.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:23 smithi103 ceph-mon[32952]: pgmap v433: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:02:26.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:25 smithi040 ceph-mon[28095]: pgmap v434: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:26.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:25 smithi080 ceph-mon[32362]: pgmap v434: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:26.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:25 smithi103 ceph-mon[32952]: pgmap v434: 129 pgs: 129 active+clean; 648 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:28.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:27 smithi040 ceph-mon[28095]: pgmap v435: 129 pgs: 129 active+clean; 658 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-27T00:02:28.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:27 smithi080 ceph-mon[32362]: pgmap v435: 129 pgs: 129 active+clean; 658 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-27T00:02:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.336+0000 7ff997200640 1 -- 172.21.15.40:0/790407382 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff990108510 msgr2=0x7ff99010a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.336+0000 7ff997200640 1 --2- 172.21.15.40:0/790407382 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff990108510 0x7ff99010a900 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009ca0 tx=0x7ff98c02a8c0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.336+0000 7ff997200640 1 -- 172.21.15.40:0/790407382 shutdown_connections 2024-04-27T00:02:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.336+0000 7ff997200640 1 --2- 172.21.15.40:0/790407382 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff990108510 0x7ff99010a900 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.337 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.336+0000 7ff997200640 1 --2- 172.21.15.40:0/790407382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff990105be0 0x7ff990107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.336+0000 7ff997200640 1 --2- 172.21.15.40:0/790407382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9901032b0 0x7ff9901056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.336+0000 7ff997200640 1 -- 172.21.15.40:0/790407382 >> 172.21.15.40:0/790407382 conn(0x7ff9900fcc20 msgr2=0x7ff9900ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.336+0000 7ff997200640 1 -- 172.21.15.40:0/790407382 shutdown_connections 2024-04-27T00:02:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.336+0000 7ff997200640 1 -- 172.21.15.40:0/790407382 wait complete. 2024-04-27T00:02:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.337+0000 7ff997200640 1 Processor -- start 2024-04-27T00:02:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.337+0000 7ff997200640 1 -- start start 2024-04-27T00:02:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.338+0000 7ff997200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9901032b0 0x7ff990102c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.338+0000 7ff997200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff990105be0 0x7ff990101260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.339 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.338+0000 7ff997200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff990108510 0x7ff9901017a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.338+0000 7ff997200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff99010e300 con 0x7ff990105be0 2024-04-27T00:02:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.338+0000 7ff997200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7ff99010e180 con 0x7ff990108510 2024-04-27T00:02:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.338+0000 7ff997200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff99010e480 con 0x7ff9901032b0 2024-04-27T00:02:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.338+0000 7ff995e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9901032b0 0x7ff990102c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.338+0000 7ff995400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff990105be0 0x7ff990101260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.338+0000 7ff995e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9901032b0 0x7ff990102c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43372/0 (socket says 172.21.15.40:43372) 2024-04-27T00:02:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.338+0000 7ff995e00640 1 -- 172.21.15.40:0/1687572938 learned_addr learned my addr 172.21.15.40:0/1687572938 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.338+0000 7ff996800640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff990108510 0x7ff9901017a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.339+0000 7ff995e00640 1 -- 172.21.15.40:0/1687572938 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff990108510 msgr2=0x7ff9901017a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.339+0000 7ff995e00640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff990108510 0x7ff9901017a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.339+0000 7ff995e00640 1 -- 172.21.15.40:0/1687572938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff990105be0 msgr2=0x7ff990101260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.339+0000 7ff995e00640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff990105be0 0x7ff990101260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.339+0000 7ff995e00640 1 -- 172.21.15.40:0/1687572938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff990102030 con 0x7ff9901032b0 2024-04-27T00:02:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.339+0000 7ff995400640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff990105be0 0x7ff990101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:02:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.339+0000 7ff995e00640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9901032b0 0x7ff990102c20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff98400e9f0 tx=0x7ff98400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.339+0000 7ff98aa00640 1 -- 172.21.15.40:0/1687572938 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98400cd80 con 0x7ff9901032b0 2024-04-27T00:02:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.340+0000 7ff98aa00640 1 -- 172.21.15.40:0/1687572938 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9840054d0 con 0x7ff9901032b0 2024-04-27T00:02:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.340+0000 7ff98aa00640 1 -- 172.21.15.40:0/1687572938 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff984010690 con 0x7ff9901032b0 2024-04-27T00:02:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.340+0000 7ff997200640 1 -- 172.21.15.40:0/1687572938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9901a2b20 con 0x7ff9901032b0 2024-04-27T00:02:28.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.340+0000 7ff997200640 1 -- 172.21.15.40:0/1687572938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9901a3020 con 0x7ff9901032b0 2024-04-27T00:02:28.344 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.343+0000 7ff98aa00640 1 -- 172.21.15.40:0/1687572938 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff984004e90 con 0x7ff9901032b0 2024-04-27T00:02:28.344 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.343+0000 7ff99b5ff740 1 Processor -- start 2024-04-27T00:02:28.344 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.343+0000 7ff99b5ff740 1 -- start start 2024-04-27T00:02:28.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.343+0000 7ff99b5ff740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df875de0 0x5623df876200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.343+0000 7ff99b5ff740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df8767d0 0x5623df87b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.343+0000 7ff99b5ff740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df87b940 0x5623df87ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.343+0000 7ff99b5ff740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5623df7da380 con 0x5623df8767d0 2024-04-27T00:02:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.343+0000 7ff99b5ff740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5623df776c30 con 0x5623df875de0 2024-04-27T00:02:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.343+0000 7ff99b5ff740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5623df7d6340 con 0x5623df87b940 2024-04-27T00:02:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.344+0000 7ff995e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df8767d0 0x5623df87b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.344+0000 7ff996800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df875de0 0x5623df876200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.344+0000 7ff995e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df8767d0 0x5623df87b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40700/0 (socket says 172.21.15.40:40700) 2024-04-27T00:02:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.344+0000 7ff995e00640 1 -- 172.21.15.40:0/102121920 learned_addr learned my addr 172.21.15.40:0/102121920 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.344+0000 7ff995400640 1 --2- 172.21.15.40:0/102121920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df87b940 0x5623df87ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.344+0000 7ff995400640 1 -- 172.21.15.40:0/102121920 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df875de0 msgr2=0x5623df876200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.344+0000 7ff995400640 1 --2- 172.21.15.40:0/102121920 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df875de0 0x5623df876200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.344+0000 7ff995400640 1 -- 172.21.15.40:0/102121920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df8767d0 msgr2=0x5623df87b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.344+0000 7ff995400640 1 --2- 172.21.15.40:0/102121920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df8767d0 0x5623df87b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.344+0000 7ff995400640 1 -- 172.21.15.40:0/102121920 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623df798940 con 0x5623df87b940 2024-04-27T00:02:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.345+0000 7ff995400640 1 --2- 172.21.15.40:0/102121920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df87b940 0x5623df87ddd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff99006aac0 tx=0x7ff980010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b73985a3660c7e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.345+0000 7ff98aa00640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff9780a3cb0 0x7ff9780a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.345+0000 7ff97f000640 1 -- 172.21.15.40:0/102121920 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9800117c0 con 0x5623df87b940 2024-04-27T00:02:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.345+0000 7ff97f000640 1 -- 172.21.15.40:0/102121920 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff980011e10 con 0x5623df87b940 2024-04-27T00:02:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.345+0000 7ff97f000640 1 -- 172.21.15.40:0/102121920 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98001ad10 con 0x5623df87b940 2024-04-27T00:02:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.345+0000 7ff98aa00640 1 -- 172.21.15.40:0/1687572938 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7ff9840d0d60 con 0x7ff9901032b0 2024-04-27T00:02:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.345+0000 7ff996800640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff9780a3cb0 0x7ff9780a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.346+0000 7ff99b5ff740 1 -- 172.21.15.40:0/102121920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df87b940 msgr2=0x5623df87ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.346+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/102121920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df87b940 0x5623df87ddd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff99006aac0 tx=0x7ff980010860 comp rx=0 tx=0).stop 2024-04-27T00:02:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.346+0000 7ff99b5ff740 1 -- 172.21.15.40:0/102121920 shutdown_connections 2024-04-27T00:02:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.346+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/102121920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df87b940 0x5623df87ddd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.346+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/102121920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df8767d0 0x5623df87b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.346+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/102121920 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df875de0 0x5623df876200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.346+0000 7ff99b5ff740 1 -- 172.21.15.40:0/102121920 >> 172.21.15.40:0/102121920 conn(0x5623df871450 msgr2=0x5623df871850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.346+0000 7ff99b5ff740 1 -- 172.21.15.40:0/102121920 shutdown_connections 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.346+0000 7ff99b5ff740 1 -- 172.21.15.40:0/102121920 wait complete. 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.346+0000 7ff996800640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff9780a3cb0 0x7ff9780a6170 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff98c031b50 tx=0x7ff98c009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.347+0000 7ff99b5ff740 1 -- start start 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.347+0000 7ff99b5ff740 1 Processor -- start 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.347+0000 7ff99b5ff740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df875de0 0x5623df87ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.347+0000 7ff99b5ff740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df8804b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.347+0000 7ff99b5ff740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df87b940 0x5623df8874f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.347+0000 7ff99b5ff740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5623df87e8f0 con 0x5623df875de0 2024-04-27T00:02:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.347+0000 7ff99b5ff740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5623df761c60 con 0x5623df8767d0 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.347+0000 7ff99b5ff740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5623df87ea70 con 0x5623df87b940 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.347+0000 7ff99b5ff740 10 client.0 _ll_register_callbacks cb 0x5623df78de50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff995e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df875de0 0x5623df87ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff995400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df8804b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff995e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df875de0 0x5623df87ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40702/0 (socket says 172.21.15.40:40702) 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff996800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df87b940 0x5623df8874f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff995e00640 1 -- 172.21.15.40:0/948986394 learned_addr learned my addr 172.21.15.40:0/948986394 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff995400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df8804b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:46252/0 (socket says 172.21.15.40:46252) 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff995400640 1 -- 172.21.15.40:0/948986394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df87b940 msgr2=0x5623df8874f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff995400640 1 --2- 172.21.15.40:0/948986394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df87b940 0x5623df8874f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff995400640 1 -- 172.21.15.40:0/948986394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df875de0 msgr2=0x5623df87ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff995400640 1 --2- 172.21.15.40:0/948986394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df875de0 0x5623df87ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff995400640 1 -- 172.21.15.40:0/948986394 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623df770510 con 0x5623df8767d0 2024-04-27T00:02:28.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff995e00640 1 --2- 172.21.15.40:0/948986394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df875de0 0x5623df87ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:02:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff995400640 1 --2- 172.21.15.40:0/948986394 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df8804b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff980011010 tx=0x7ff98001a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1b2221684f22177 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.348+0000 7ff973400640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:02:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.349+0000 7ff973400640 1 -- 172.21.15.40:0/948986394 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98001aca0 con 0x5623df8767d0 2024-04-27T00:02:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.349+0000 7ff973400640 1 -- 172.21.15.40:0/948986394 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7ff980004020 con 0x5623df8767d0 2024-04-27T00:02:28.353 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.349+0000 7ff99b5ff740 1 -- 172.21.15.40:0/948986394 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff980011e10 con 0x5623df8767d0 2024-04-27T00:02:28.353 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:27 smithi103 ceph-mon[32952]: pgmap v435: 129 pgs: 129 active+clean; 658 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-27T00:02:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.349+0000 7ff973400640 1 -- 172.21.15.40:0/948986394 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff98001ae40 con 0x5623df8767d0 2024-04-27T00:02:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.349+0000 7ff99b5ff740 10 client.25430 fetch_fsmap learned FSMap version 24 2024-04-27T00:02:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.349+0000 7ff99b5ff740 1 -- 172.21.15.40:0/948986394 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5623df4ec210 con 0x5623df8767d0 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.349+0000 7ff973400640 1 -- 172.21.15.40:0/948986394 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff980011e10 con 0x5623df8767d0 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.349+0000 7ff973400640 1 -- 172.21.15.40:0/948986394 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff980005040 con 0x5623df8767d0 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.350+0000 7ff99b5ff740 10 client.25430 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.350+0000 7ff99b5ff740 10 client.25430 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.350+0000 7ff99b5ff740 20 client.25430 populate_metadata read hostname 'smithi040' 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.350+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/948986394 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5623df88b720 0x5623df88dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.350+0000 7ff99b5ff740 4 client.25430 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.350+0000 7ff99b5ff740 1 -- 172.21.15.40:0/948986394 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623df7e7f90 con 0x5623df88b720 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.350+0000 7ff995e00640 1 --2- 172.21.15.40:0/948986394 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5623df88b720 0x5623df88dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.351+0000 7ff995e00640 1 --2- 172.21.15.40:0/948986394 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5623df88b720 0x5623df88dbc0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3cfc3de2b4cf3fc server_cookie=990232f25ab73a17 in_seq=0 out_seq=0 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.351+0000 7ff973400640 10 client.25430 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.351+0000 7ff973400640 1 -- 172.21.15.40:0/948986394 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623df7e7f90 con 0x5623df88b720 2024-04-27T00:02:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.351+0000 7ff973400640 10 client.25430 handle_command_reply: tid=0 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.351+0000 7ff99b5ff740 1 client.25430 shutdown 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.352+0000 7ff99b5ff740 1 -- 172.21.15.40:0/948986394 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 msgr2=0x5623df8804b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.352+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/948986394 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df8804b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff980011010 tx=0x7ff98001a5b0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.352+0000 7ff99b5ff740 1 -- 172.21.15.40:0/948986394 shutdown_connections 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.352+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/948986394 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5623df88b720 0x5623df88dbc0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.352+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/948986394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df87b940 0x5623df8874f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.352+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/948986394 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df8804b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.352+0000 7ff973400640 0 client.25430 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.352+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/948986394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df875de0 0x5623df87ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.352+0000 7ff99b5ff740 1 -- 172.21.15.40:0/948986394 >> 172.21.15.40:0/948986394 conn(0x5623df871450 msgr2=0x5623df870650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.352+0000 7ff99b5ff740 1 -- 172.21.15.40:0/948986394 shutdown_connections 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.352+0000 7ff99b5ff740 1 -- 172.21.15.40:0/948986394 wait complete. 2024-04-27T00:02:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.352+0000 7ff99b5ff740 20 client.25430 trim_cache size 0 max 16384 2024-04-27T00:02:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.369+0000 7ff99b5ff740 1 Processor -- start 2024-04-27T00:02:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.370+0000 7ff99b5ff740 1 -- start start 2024-04-27T00:02:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.370+0000 7ff99b5ff740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 0x5623df893490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.370+0000 7ff99b5ff740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df875de0 0x5623df8939d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.370+0000 7ff99b5ff740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df8874b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.370+0000 7ff99b5ff740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff9901afe50 con 0x5623df871450 2024-04-27T00:02:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.370+0000 7ff99b5ff740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7ff9901ad720 con 0x5623df8767d0 2024-04-27T00:02:28.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.370+0000 7ff99b5ff740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff9901ab570 con 0x5623df875de0 2024-04-27T00:02:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.370+0000 7ff995e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 0x5623df893490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.370+0000 7ff995e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 0x5623df893490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40708/0 (socket says 172.21.15.40:40708) 2024-04-27T00:02:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.370+0000 7ff995e00640 1 -- 172.21.15.40:0/3097962178 learned_addr learned my addr 172.21.15.40:0/3097962178 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:28.372 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.370+0000 7ff996800640 1 --2- 172.21.15.40:0/3097962178 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df8874b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.370+0000 7ff995400640 1 --2- 172.21.15.40:0/3097962178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df875de0 0x5623df8939d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.371+0000 7ff995e00640 1 -- 172.21.15.40:0/3097962178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df875de0 msgr2=0x5623df8939d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.371+0000 7ff995e00640 1 --2- 172.21.15.40:0/3097962178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df875de0 0x5623df8939d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.371+0000 7ff995e00640 1 -- 172.21.15.40:0/3097962178 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 msgr2=0x5623df8874b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.371+0000 7ff995e00640 1 --2- 172.21.15.40:0/3097962178 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df8874b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.371+0000 7ff995e00640 1 -- 172.21.15.40:0/3097962178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623df5b8620 con 0x5623df871450 2024-04-27T00:02:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.371+0000 7ff995e00640 1 --2- 172.21.15.40:0/3097962178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 0x5623df893490 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff984099460 tx=0x7ff9840bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92576ab088df8a2c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.371+0000 7ff97e600640 1 -- 172.21.15.40:0/3097962178 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9840b5020 con 0x5623df871450 2024-04-27T00:02:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.371+0000 7ff97e600640 1 -- 172.21.15.40:0/3097962178 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff984005680 con 0x5623df871450 2024-04-27T00:02:28.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.371+0000 7ff97e600640 1 -- 172.21.15.40:0/3097962178 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9840a5750 con 0x5623df871450 2024-04-27T00:02:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.372+0000 7ff99b5ff740 1 -- 172.21.15.40:0/3097962178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 msgr2=0x5623df893490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.372+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/3097962178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 0x5623df893490 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff984099460 tx=0x7ff9840bd040 comp rx=0 tx=0).stop 2024-04-27T00:02:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.372+0000 7ff99b5ff740 1 -- 172.21.15.40:0/3097962178 shutdown_connections 2024-04-27T00:02:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.372+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/3097962178 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df8874b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.372+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/3097962178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df875de0 0x5623df8939d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.372+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/3097962178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 0x5623df893490 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.372+0000 7ff99b5ff740 1 -- 172.21.15.40:0/3097962178 >> 172.21.15.40:0/3097962178 conn(0x5623df877230 msgr2=0x5623df877950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.372+0000 7ff99b5ff740 1 -- 172.21.15.40:0/3097962178 shutdown_connections 2024-04-27T00:02:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.372+0000 7ff99b5ff740 1 -- 172.21.15.40:0/3097962178 wait complete. 2024-04-27T00:02:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.373+0000 7ff99b5ff740 1 -- start start 2024-04-27T00:02:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.373+0000 7ff99b5ff740 1 Processor -- start 2024-04-27T00:02:28.375 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.373+0000 7ff99b5ff740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 0x5623df884a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.373+0000 7ff99b5ff740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df875de0 0x5623df884fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.373+0000 7ff99b5ff740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df885f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.373+0000 7ff99b5ff740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff9901abbe0 con 0x5623df871450 2024-04-27T00:02:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.373+0000 7ff99b5ff740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7ff9901ad050 con 0x5623df8767d0 2024-04-27T00:02:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.373+0000 7ff99b5ff740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff9901ac2b0 con 0x5623df875de0 2024-04-27T00:02:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.373+0000 7ff99b5ff740 10 client.0 _ll_register_callbacks cb 0x7ff9840d46e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:02:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.374+0000 7ff995e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 0x5623df884a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.374+0000 7ff995e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 0x5623df884a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40710/0 (socket says 172.21.15.40:40710) 2024-04-27T00:02:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.374+0000 7ff995e00640 1 -- 172.21.15.40:0/262175427 learned_addr learned my addr 172.21.15.40:0/262175427 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.374+0000 7ff995400640 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df875de0 0x5623df884fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.374+0000 7ff996800640 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df885f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.374+0000 7ff995e00640 1 -- 172.21.15.40:0/262175427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df875de0 msgr2=0x5623df884fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.374+0000 7ff995e00640 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df875de0 0x5623df884fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.374+0000 7ff995e00640 1 -- 172.21.15.40:0/262175427 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 msgr2=0x5623df885f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.374+0000 7ff995e00640 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df885f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.374+0000 7ff995e00640 1 -- 172.21.15.40:0/262175427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623df794cb0 con 0x5623df871450 2024-04-27T00:02:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.374+0000 7ff995e00640 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 0x5623df884a70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff9840bd010 tx=0x7ff9840d4510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=556b9a2f41023e73 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.374+0000 7ff973400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:02:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.375+0000 7ff973400640 1 -- 172.21.15.40:0/262175427 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9840aa060 con 0x5623df871450 2024-04-27T00:02:28.378 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.375+0000 7ff973400640 1 -- 172.21.15.40:0/262175427 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7ff9840046c0 con 0x5623df871450 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.375+0000 7ff99b5ff740 1 -- 172.21.15.40:0/262175427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff984005680 con 0x5623df871450 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.375+0000 7ff973400640 1 -- 172.21.15.40:0/262175427 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff9840b5050 con 0x5623df871450 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.375+0000 7ff99b5ff740 10 client.15312 fetch_fsmap learned FSMap version 24 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.375+0000 7ff99b5ff740 1 -- 172.21.15.40:0/262175427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5623df8876a0 con 0x5623df871450 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.375+0000 7ff973400640 1 -- 172.21.15.40:0/262175427 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff984005680 con 0x5623df871450 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.375+0000 7ff973400640 1 -- 172.21.15.40:0/262175427 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff9840ce560 con 0x5623df871450 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.375+0000 7ff99b5ff740 10 client.15312 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.375+0000 7ff99b5ff740 10 client.15312 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.375+0000 7ff99b5ff740 20 client.15312 populate_metadata read hostname 'smithi040' 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.376+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5623df87b940 0x5623df8f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.376+0000 7ff99b5ff740 4 client.15312 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.376+0000 7ff99b5ff740 1 -- 172.21.15.40:0/262175427 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5623df7e1e00 con 0x5623df87b940 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.376+0000 7ff995400640 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5623df87b940 0x5623df8f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.376+0000 7ff995400640 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5623df87b940 0x5623df8f42f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af08b3ec0c98870 server_cookie=4e91b080436fdc30 in_seq=0 out_seq=0 2024-04-27T00:02:28.379 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.376+0000 7ff973400640 10 client.15312 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff973400640 1 -- 172.21.15.40:0/262175427 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5623df7e1e00 con 0x5623df87b940 2024-04-27T00:02:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff973400640 10 client.15312 handle_command_reply: tid=0 2024-04-27T00:02:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff99b5ff740 1 client.15312 shutdown 2024-04-27T00:02:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff99b5ff740 1 -- 172.21.15.40:0/262175427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 msgr2=0x5623df884a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 0x5623df884a70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff9840bd010 tx=0x7ff9840d4510 comp rx=0 tx=0).stop 2024-04-27T00:02:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff99b5ff740 1 -- 172.21.15.40:0/262175427 shutdown_connections 2024-04-27T00:02:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5623df87b940 0x5623df8f42f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5623df8767d0 0x5623df885f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623df875de0 0x5623df884fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff99b5ff740 1 --2- 172.21.15.40:0/262175427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5623df871450 0x5623df884a70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff99b5ff740 1 -- 172.21.15.40:0/262175427 >> 172.21.15.40:0/262175427 conn(0x5623df877230 msgr2=0x5623df8741d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff973400640 0 client.15312 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.384+0000 7ff99b5ff740 1 -- 172.21.15.40:0/262175427 shutdown_connections 2024-04-27T00:02:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.385+0000 7ff99b5ff740 1 -- 172.21.15.40:0/262175427 wait complete. 2024-04-27T00:02:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.385+0000 7ff99b5ff740 20 client.15312 trim_cache size 0 max 16384 2024-04-27T00:02:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:02:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:02:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:02:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:02:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.386+0000 7ff997200640 1 -- 172.21.15.40:0/1687572938 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff9780a3cb0 msgr2=0x7ff9780a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.386+0000 7ff997200640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff9780a3cb0 0x7ff9780a6170 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff98c031b50 tx=0x7ff98c009360 comp rx=0 tx=0).stop 2024-04-27T00:02:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.386+0000 7ff997200640 1 -- 172.21.15.40:0/1687572938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9901032b0 msgr2=0x7ff990102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.386+0000 7ff997200640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9901032b0 0x7ff990102c20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff98400e9f0 tx=0x7ff98400edf0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.386+0000 7ff997200640 1 -- 172.21.15.40:0/1687572938 shutdown_connections 2024-04-27T00:02:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.386+0000 7ff997200640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff990108510 0x7ff9901017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.386+0000 7ff997200640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff990105be0 0x7ff990101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.386+0000 7ff997200640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff9780a3cb0 0x7ff9780a6170 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.386+0000 7ff997200640 1 --2- 172.21.15.40:0/1687572938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9901032b0 0x7ff990102c20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.386+0000 7ff997200640 1 -- 172.21.15.40:0/1687572938 >> 172.21.15.40:0/1687572938 conn(0x7ff9900fcc20 msgr2=0x7ff990103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.386+0000 7ff997200640 1 -- 172.21.15.40:0/1687572938 shutdown_connections 2024-04-27T00:02:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:28.386+0000 7ff997200640 1 -- 172.21.15.40:0/1687572938 wait complete. 2024-04-27T00:02:30.217 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:29 smithi040 ceph-mon[28095]: pgmap v436: 129 pgs: 129 active+clean; 658 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-27T00:02:30.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:29 smithi080 ceph-mon[32362]: pgmap v436: 129 pgs: 129 active+clean; 658 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-27T00:02:30.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:29 smithi103 ceph-mon[32952]: pgmap v436: 129 pgs: 129 active+clean; 658 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-27T00:02:32.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:31 smithi040 ceph-mon[28095]: pgmap v437: 129 pgs: 129 active+clean; 658 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-27T00:02:32.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:31 smithi080 ceph-mon[32362]: pgmap v437: 129 pgs: 129 active+clean; 658 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-27T00:02:32.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:31 smithi103 ceph-mon[32952]: pgmap v437: 129 pgs: 129 active+clean; 658 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-27T00:02:34.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:33 smithi040 ceph-mon[28095]: pgmap v438: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-04-27T00:02:34.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:33 smithi080 ceph-mon[32362]: pgmap v438: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-04-27T00:02:34.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:33 smithi103 ceph-mon[32952]: pgmap v438: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-04-27T00:02:36.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:35 smithi040 ceph-mon[28095]: pgmap v439: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-27T00:02:36.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:35 smithi080 ceph-mon[32362]: pgmap v439: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-27T00:02:36.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:35 smithi103 ceph-mon[32952]: pgmap v439: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-27T00:02:38.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:37 smithi040 ceph-mon[28095]: pgmap v440: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-27T00:02:38.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:37 smithi080 ceph-mon[32362]: pgmap v440: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-27T00:02:38.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:37 smithi103 ceph-mon[32952]: pgmap v440: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-27T00:02:39.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:38 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:02:39.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:38 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:02:39.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:38 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:02:40.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:39 smithi040 ceph-mon[28095]: pgmap v441: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 852 B/s wr, 6 op/s 2024-04-27T00:02:40.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:39 smithi080 ceph-mon[32362]: pgmap v441: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 852 B/s wr, 6 op/s 2024-04-27T00:02:40.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:39 smithi103 ceph-mon[32952]: pgmap v441: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 852 B/s wr, 6 op/s 2024-04-27T00:02:41.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:40 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:02:41.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:40 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:02:41.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:40 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:02:41.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:40 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:02:41.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:40 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:02:41.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:40 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:02:41.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:40 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:02:41.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:40 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:02:41.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:40 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:02:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.555+0000 7f0f81c00640 1 -- 172.21.15.40:0/1663042755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f7c101a00 msgr2=0x7f0f7c101e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.555+0000 7f0f81c00640 1 --2- 172.21.15.40:0/1663042755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f7c101a00 0x7f0f7c101e60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0f68009900 tx=0x7f0f6802a480 comp rx=0 tx=0).stop 2024-04-27T00:02:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.555+0000 7f0f81c00640 1 -- 172.21.15.40:0/1663042755 shutdown_connections 2024-04-27T00:02:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.555+0000 7f0f81c00640 1 --2- 172.21.15.40:0/1663042755 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f7c1023a0 0x7f0f7c102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.555+0000 7f0f81c00640 1 --2- 172.21.15.40:0/1663042755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f7c101a00 0x7f0f7c101e60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.555+0000 7f0f81c00640 1 --2- 172.21.15.40:0/1663042755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f7c1010a0 0x7f0f7c1014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.555+0000 7f0f81c00640 1 -- 172.21.15.40:0/1663042755 >> 172.21.15.40:0/1663042755 conn(0x7f0f7c0fcc20 msgr2=0x7f0f7c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.555+0000 7f0f81c00640 1 -- 172.21.15.40:0/1663042755 shutdown_connections 2024-04-27T00:02:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.555+0000 7f0f81c00640 1 -- 172.21.15.40:0/1663042755 wait complete. 2024-04-27T00:02:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.556+0000 7f0f81c00640 1 Processor -- start 2024-04-27T00:02:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.556+0000 7f0f81c00640 1 -- start start 2024-04-27T00:02:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.556+0000 7f0f81c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f7c1010a0 0x7f0f7c19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.556+0000 7f0f81c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f7c101a00 0x7f0f7c19b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.556+0000 7f0f81c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f7c1023a0 0x7f0f7c1a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.556+0000 7f0f81c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1104c0 con 0x7f0f7c101a00 2024-04-27T00:02:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.556+0000 7f0f81c00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c110340 con 0x7f0f7c1010a0 2024-04-27T00:02:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.556+0000 7f0f81c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c110640 con 0x7f0f7c1023a0 2024-04-27T00:02:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.557+0000 7f0f7b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f7c101a00 0x7f0f7c19b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.557+0000 7f0f7be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f7c1010a0 0x7f0f7c19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.557+0000 7f0f7b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f7c101a00 0x7f0f7c19b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45688/0 (socket says 172.21.15.40:45688) 2024-04-27T00:02:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.557+0000 7f0f7b400640 1 -- 172.21.15.40:0/2044405716 learned_addr learned my addr 172.21.15.40:0/2044405716 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.557+0000 7f0f81200640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f7c1023a0 0x7f0f7c1a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.557+0000 7f0f7b400640 1 -- 172.21.15.40:0/2044405716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f7c1023a0 msgr2=0x7f0f7c1a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.557+0000 7f0f7b400640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f7c1023a0 0x7f0f7c1a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.557+0000 7f0f7b400640 1 -- 172.21.15.40:0/2044405716 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f7c1010a0 msgr2=0x7f0f7c19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.557+0000 7f0f7b400640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f7c1010a0 0x7f0f7c19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.557+0000 7f0f7b400640 1 -- 172.21.15.40:0/2044405716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7c1a2810 con 0x7f0f7c101a00 2024-04-27T00:02:41.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.558+0000 7f0f7b400640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f7c101a00 0x7f0f7c19b130 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0f68005940 tx=0x7f0f680042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.558+0000 7f0f78c00640 1 -- 172.21.15.40:0/2044405716 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6803f030 con 0x7f0f7c101a00 2024-04-27T00:02:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.558+0000 7f0f78c00640 1 -- 172.21.15.40:0/2044405716 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f68003cb0 con 0x7f0f7c101a00 2024-04-27T00:02:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.558+0000 7f0f78c00640 1 -- 172.21.15.40:0/2044405716 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f68035070 con 0x7f0f7c101a00 2024-04-27T00:02:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.558+0000 7f0f81c00640 1 -- 172.21.15.40:0/2044405716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f7c1a2aa0 con 0x7f0f7c101a00 2024-04-27T00:02:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.558+0000 7f0f81c00640 1 -- 172.21.15.40:0/2044405716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f7c1a2fe0 con 0x7f0f7c101a00 2024-04-27T00:02:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.561+0000 7f0f861a7740 1 Processor -- start 2024-04-27T00:02:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f861a7740 1 -- start start 2024-04-27T00:02:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f78c00640 1 -- 172.21.15.40:0/2044405716 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0f68042030 con 0x7f0f7c101a00 2024-04-27T00:02:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f861a7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56042051fde0 0x560420520200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f861a7740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604205207d0 0x5604205252d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f861a7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560420525940 0x560420527dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f861a7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560420484380 con 0x56042051fde0 2024-04-27T00:02:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f861a7740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x560420420c30 con 0x5604205207d0 2024-04-27T00:02:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f861a7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560420480340 con 0x560420525940 2024-04-27T00:02:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f81200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56042051fde0 0x560420520200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f81200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56042051fde0 0x560420520200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45694/0 (socket says 172.21.15.40:45694) 2024-04-27T00:02:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f7b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560420525940 0x560420527dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f81200640 1 -- 172.21.15.40:0/3229180830 learned_addr learned my addr 172.21.15.40:0/3229180830 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.562+0000 7f0f7be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604205207d0 0x5604205252d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.563+0000 7f0f81200640 1 -- 172.21.15.40:0/3229180830 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560420525940 msgr2=0x560420527dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.563+0000 7f0f81200640 1 --2- 172.21.15.40:0/3229180830 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560420525940 0x560420527dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.563+0000 7f0f81200640 1 -- 172.21.15.40:0/3229180830 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604205207d0 msgr2=0x5604205252d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.563+0000 7f0f81200640 1 --2- 172.21.15.40:0/3229180830 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604205207d0 0x5604205252d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.563+0000 7f0f81200640 1 -- 172.21.15.40:0/3229180830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560420442940 con 0x56042051fde0 2024-04-27T00:02:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.563+0000 7f0f7be00640 1 --2- 172.21.15.40:0/3229180830 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604205207d0 0x5604205252d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:02:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.563+0000 7f0f81200640 1 --2- 172.21.15.40:0/3229180830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56042051fde0 0x560420520200 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009fd0 tx=0x7f0f740118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28af31b96940c5e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.563+0000 7f0f71600640 1 -- 172.21.15.40:0/3229180830 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f74004d20 con 0x56042051fde0 2024-04-27T00:02:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.563+0000 7f0f71600640 1 -- 172.21.15.40:0/3229180830 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f74004050 con 0x56042051fde0 2024-04-27T00:02:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.563+0000 7f0f71600640 1 -- 172.21.15.40:0/3229180830 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f74005280 con 0x56042051fde0 2024-04-27T00:02:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.564+0000 7f0f78c00640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f600a3cb0 0x7f0f600a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.564+0000 7f0f78c00640 1 -- 172.21.15.40:0/2044405716 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f0f680f0cc0 con 0x7f0f7c101a00 2024-04-27T00:02:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.564+0000 7f0f7be00640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f600a3cb0 0x7f0f600a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.564+0000 7f0f861a7740 1 -- 172.21.15.40:0/3229180830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56042051fde0 msgr2=0x560420520200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.564+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3229180830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56042051fde0 0x560420520200 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009fd0 tx=0x7f0f740118d0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.564+0000 7f0f861a7740 1 -- 172.21.15.40:0/3229180830 shutdown_connections 2024-04-27T00:02:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.564+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3229180830 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560420525940 0x560420527dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.564+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3229180830 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604205207d0 0x5604205252d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.564+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3229180830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56042051fde0 0x560420520200 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.564+0000 7f0f861a7740 1 -- 172.21.15.40:0/3229180830 >> 172.21.15.40:0/3229180830 conn(0x56042051b450 msgr2=0x56042051b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.565+0000 7f0f861a7740 1 -- 172.21.15.40:0/3229180830 shutdown_connections 2024-04-27T00:02:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.565+0000 7f0f861a7740 1 -- 172.21.15.40:0/3229180830 wait complete. 2024-04-27T00:02:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.565+0000 7f0f7be00640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f600a3cb0 0x7f0f600a6170 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x560420520bd0 tx=0x7f0f6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.565+0000 7f0f861a7740 1 -- start start 2024-04-27T00:02:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.565+0000 7f0f861a7740 1 Processor -- start 2024-04-27T00:02:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.566+0000 7f0f861a7740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56042051fde0 0x560420529ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.566+0000 7f0f861a7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 0x56042052a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.566+0000 7f0f861a7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x560420531570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.566+0000 7f0f861a7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5604205288f0 con 0x560420525940 2024-04-27T00:02:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.566+0000 7f0f861a7740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x56042040bc60 con 0x56042051fde0 2024-04-27T00:02:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.566+0000 7f0f861a7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560420528a70 con 0x5604205207d0 2024-04-27T00:02:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.566+0000 7f0f861a7740 10 client.0 _ll_register_callbacks cb 0x560420437e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:02:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.566+0000 7f0f7b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 0x56042052a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.566+0000 7f0f81200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x560420531570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.566+0000 7f0f7b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 0x56042052a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37820/0 (socket says 172.21.15.40:37820) 2024-04-27T00:02:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.566+0000 7f0f7b400640 1 -- 172.21.15.40:0/3924337548 learned_addr learned my addr 172.21.15.40:0/3924337548 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.566+0000 7f0f7be00640 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56042051fde0 0x560420529ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.567+0000 7f0f81200640 1 -- 172.21.15.40:0/3924337548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 msgr2=0x56042052a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.567+0000 7f0f81200640 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 0x56042052a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.567+0000 7f0f81200640 1 -- 172.21.15.40:0/3924337548 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56042051fde0 msgr2=0x560420529ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.567+0000 7f0f81200640 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56042051fde0 0x560420529ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.567+0000 7f0f81200640 1 -- 172.21.15.40:0/3924337548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56042041a510 con 0x560420525940 2024-04-27T00:02:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.567+0000 7f0f7b400640 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 0x56042052a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:02:41.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.567+0000 7f0f81200640 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x560420531570 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0f7401b040 tx=0x7f0f74004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97eadd1c990be71c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.567+0000 7f0f66000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.567+0000 7f0f66000640 1 -- 172.21.15.40:0/3924337548 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f74028020 con 0x560420525940 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.567+0000 7f0f66000640 1 -- 172.21.15.40:0/3924337548 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f0f74004c50 con 0x560420525940 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.567+0000 7f0f861a7740 1 -- 172.21.15.40:0/3924337548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f74004050 con 0x560420525940 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.568+0000 7f0f66000640 1 -- 172.21.15.40:0/3924337548 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0f74033050 con 0x560420525940 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.568+0000 7f0f861a7740 10 client.15324 fetch_fsmap learned FSMap version 24 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.568+0000 7f0f861a7740 1 -- 172.21.15.40:0/3924337548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560420196210 con 0x560420525940 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.568+0000 7f0f66000640 1 -- 172.21.15.40:0/3924337548 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f74004050 con 0x560420525940 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.568+0000 7f0f66000640 1 -- 172.21.15.40:0/3924337548 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f0f74005780 con 0x560420525940 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.568+0000 7f0f861a7740 10 client.15324 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.568+0000 7f0f861a7740 10 client.15324 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.569+0000 7f0f861a7740 20 client.15324 populate_metadata read hostname 'smithi040' 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.569+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5604205358a0 0x560420537d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.569+0000 7f0f861a7740 4 client.15324 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:02:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.569+0000 7f0f861a7740 1 -- 172.21.15.40:0/3924337548 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560420491f90 con 0x5604205358a0 2024-04-27T00:02:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.569+0000 7f0f7be00640 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5604205358a0 0x560420537d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.570+0000 7f0f7be00640 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5604205358a0 0x560420537d00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ba3009fa670ceb9 server_cookie=e461f8f026b881c6 in_seq=0 out_seq=0 2024-04-27T00:02:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.570+0000 7f0f66000640 10 client.15324 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.570+0000 7f0f66000640 1 -- 172.21.15.40:0/3924337548 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560420491f90 con 0x5604205358a0 2024-04-27T00:02:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.570+0000 7f0f66000640 10 client.15324 handle_command_reply: tid=0 2024-04-27T00:02:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.570+0000 7f0f861a7740 1 client.15324 shutdown 2024-04-27T00:02:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.571+0000 7f0f861a7740 1 -- 172.21.15.40:0/3924337548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 msgr2=0x560420531570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.571+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x560420531570 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0f7401b040 tx=0x7f0f74004430 comp rx=0 tx=0).stop 2024-04-27T00:02:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.571+0000 7f0f861a7740 1 -- 172.21.15.40:0/3924337548 shutdown_connections 2024-04-27T00:02:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.571+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5604205358a0 0x560420537d00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.571+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x560420531570 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.571+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 0x56042052a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.571+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3924337548 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56042051fde0 0x560420529ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.571+0000 7f0f861a7740 1 -- 172.21.15.40:0/3924337548 >> 172.21.15.40:0/3924337548 conn(0x56042051b450 msgr2=0x56042051a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.571+0000 7f0f66000640 0 client.15324 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.571+0000 7f0f861a7740 1 -- 172.21.15.40:0/3924337548 shutdown_connections 2024-04-27T00:02:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.571+0000 7f0f861a7740 1 -- 172.21.15.40:0/3924337548 wait complete. 2024-04-27T00:02:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.571+0000 7f0f861a7740 20 client.15324 trim_cache size 0 max 16384 2024-04-27T00:02:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.587+0000 7f0f861a7740 1 Processor -- start 2024-04-27T00:02:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.587+0000 7f0f861a7740 1 -- start start 2024-04-27T00:02:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.587+0000 7f0f861a7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56042051fde0 0x56042051e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.587+0000 7f0f861a7740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604205207d0 0x560420531530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.587+0000 7f0f861a7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x56042051ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.587+0000 7f0f861a7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1afe60 con 0x560420525940 2024-04-27T00:02:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.587+0000 7f0f861a7740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1ad730 con 0x5604205207d0 2024-04-27T00:02:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.587+0000 7f0f861a7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1ab580 con 0x56042051fde0 2024-04-27T00:02:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.588+0000 7f0f7be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56042051fde0 0x56042051e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.588+0000 7f0f81200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x56042051ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.588+0000 7f0f7be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56042051fde0 0x56042051e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37826/0 (socket says 172.21.15.40:37826) 2024-04-27T00:02:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.588+0000 7f0f81200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x56042051ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45720/0 (socket says 172.21.15.40:45720) 2024-04-27T00:02:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.588+0000 7f0f7b400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604205207d0 0x560420531530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.588+0000 7f0f7be00640 1 -- 172.21.15.40:0/2773137569 learned_addr learned my addr 172.21.15.40:0/2773137569 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.588+0000 7f0f7b400640 1 -- 172.21.15.40:0/2773137569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56042051fde0 msgr2=0x56042051e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.588+0000 7f0f7b400640 1 --2- 172.21.15.40:0/2773137569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56042051fde0 0x56042051e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.588+0000 7f0f7b400640 1 -- 172.21.15.40:0/2773137569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 msgr2=0x56042051ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.588+0000 7f0f7b400640 1 --2- 172.21.15.40:0/2773137569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x56042051ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.588+0000 7f0f7b400640 1 -- 172.21.15.40:0/2773137569 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560420262620 con 0x5604205207d0 2024-04-27T00:02:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.588+0000 7f0f81200640 1 --2- 172.21.15.40:0/2773137569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x56042051ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:02:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.589+0000 7f0f7b400640 1 --2- 172.21.15.40:0/2773137569 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604205207d0 0x560420531530 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0f680b94c0 tx=0x7f0f680f4770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e98498e45a94e138 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.589+0000 7f0f70c00640 1 -- 172.21.15.40:0/2773137569 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f68049040 con 0x5604205207d0 2024-04-27T00:02:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.589+0000 7f0f70c00640 1 -- 172.21.15.40:0/2773137569 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f680dd0b0 con 0x5604205207d0 2024-04-27T00:02:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.589+0000 7f0f70c00640 1 -- 172.21.15.40:0/2773137569 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f680bcaf0 con 0x5604205207d0 2024-04-27T00:02:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.590+0000 7f0f861a7740 1 -- 172.21.15.40:0/2773137569 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604205207d0 msgr2=0x560420531530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.590+0000 7f0f861a7740 1 --2- 172.21.15.40:0/2773137569 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604205207d0 0x560420531530 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0f680b94c0 tx=0x7f0f680f4770 comp rx=0 tx=0).stop 2024-04-27T00:02:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.590+0000 7f0f861a7740 1 -- 172.21.15.40:0/2773137569 shutdown_connections 2024-04-27T00:02:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.590+0000 7f0f861a7740 1 --2- 172.21.15.40:0/2773137569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x56042051ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.590+0000 7f0f861a7740 1 --2- 172.21.15.40:0/2773137569 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604205207d0 0x560420531530 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.590+0000 7f0f861a7740 1 --2- 172.21.15.40:0/2773137569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56042051fde0 0x56042051e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.590+0000 7f0f861a7740 1 -- 172.21.15.40:0/2773137569 >> 172.21.15.40:0/2773137569 conn(0x56042051b450 msgr2=0x560420529ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.590+0000 7f0f861a7740 1 -- 172.21.15.40:0/2773137569 shutdown_connections 2024-04-27T00:02:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.590+0000 7f0f861a7740 1 -- 172.21.15.40:0/2773137569 wait complete. 2024-04-27T00:02:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.590+0000 7f0f861a7740 1 -- start start 2024-04-27T00:02:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.590+0000 7f0f861a7740 1 Processor -- start 2024-04-27T00:02:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.591+0000 7f0f861a7740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56042051fde0 0x56042052e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.591+0000 7f0f861a7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 0x56042052e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.591+0000 7f0f861a7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x560420599e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.591+0000 7f0f861a7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1abbf0 con 0x560420525940 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.591+0000 7f0f861a7740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1ad060 con 0x56042051fde0 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.591+0000 7f0f861a7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1ac2c0 con 0x5604205207d0 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.591+0000 7f0f861a7740 10 client.0 _ll_register_callbacks cb 0x7f0f6c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.591+0000 7f0f81200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x560420599e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.591+0000 7f0f7b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 0x56042052e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.591+0000 7f0f81200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x560420599e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45724/0 (socket says 172.21.15.40:45724) 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.591+0000 7f0f81200640 1 -- 172.21.15.40:0/3778160260 learned_addr learned my addr 172.21.15.40:0/3778160260 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.591+0000 7f0f7be00640 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56042051fde0 0x56042052e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.592+0000 7f0f81200640 1 -- 172.21.15.40:0/3778160260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 msgr2=0x56042052e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.592+0000 7f0f81200640 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 0x56042052e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.592+0000 7f0f81200640 1 -- 172.21.15.40:0/3778160260 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56042051fde0 msgr2=0x56042052e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.592+0000 7f0f81200640 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56042051fde0 0x56042052e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.592+0000 7f0f81200640 1 -- 172.21.15.40:0/3778160260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56042043ecb0 con 0x560420525940 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.592+0000 7f0f7be00640 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56042051fde0 0x56042052e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.592+0000 7f0f7b400640 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 0x56042052e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.592+0000 7f0f81200640 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x560420599e30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0f74000f80 tx=0x7f0f7400f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda32be2b0a35cdd server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.592+0000 7f0f66000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.592+0000 7f0f66000640 1 -- 172.21.15.40:0/3778160260 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7403c060 con 0x560420525940 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.592+0000 7f0f861a7740 1 -- 172.21.15.40:0/3778160260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f680dd0b0 con 0x560420525940 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.592+0000 7f0f66000640 1 -- 172.21.15.40:0/3778160260 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f0f7400fe20 con 0x560420525940 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.593+0000 7f0f66000640 1 -- 172.21.15.40:0/3778160260 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0f74043050 con 0x560420525940 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.593+0000 7f0f861a7740 10 client.15330 fetch_fsmap learned FSMap version 24 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.593+0000 7f0f861a7740 1 -- 172.21.15.40:0/3778160260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56042059b070 con 0x560420525940 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.593+0000 7f0f66000640 1 -- 172.21.15.40:0/3778160260 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f680dd0b0 con 0x560420525940 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.593+0000 7f0f66000640 1 -- 172.21.15.40:0/3778160260 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f0f740377b0 con 0x560420525940 2024-04-27T00:02:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.593+0000 7f0f861a7740 10 client.15330 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:02:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.593+0000 7f0f861a7740 10 client.15330 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:02:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.593+0000 7f0f861a7740 20 client.15330 populate_metadata read hostname 'smithi040' 2024-04-27T00:02:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.593+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56042059ba40 0x56042059de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.593+0000 7f0f861a7740 4 client.15330 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:02:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.593+0000 7f0f861a7740 1 -- 172.21.15.40:0/3778160260 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56042048be00 con 0x56042059ba40 2024-04-27T00:02:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.594+0000 7f0f7be00640 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56042059ba40 0x56042059de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.594+0000 7f0f7be00640 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56042059ba40 0x56042059de60 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6dce4eb45022bdf server_cookie=1e60ad4ac09fd97e in_seq=0 out_seq=0 2024-04-27T00:02:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.594+0000 7f0f66000640 10 client.15330 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f66000640 1 -- 172.21.15.40:0/3778160260 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56042048be00 con 0x56042059ba40 2024-04-27T00:02:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f66000640 10 client.15330 handle_command_reply: tid=0 2024-04-27T00:02:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f861a7740 1 client.15330 shutdown 2024-04-27T00:02:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f861a7740 1 -- 172.21.15.40:0/3778160260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 msgr2=0x560420599e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x560420599e30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0f74000f80 tx=0x7f0f7400f9d0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f861a7740 1 -- 172.21.15.40:0/3778160260 shutdown_connections 2024-04-27T00:02:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56042059ba40 0x56042059de60 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560420525940 0x560420599e30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604205207d0 0x56042052e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f861a7740 1 --2- 172.21.15.40:0/3778160260 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56042051fde0 0x56042052e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f66000640 0 client.15330 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f861a7740 1 -- 172.21.15.40:0/3778160260 >> 172.21.15.40:0/3778160260 conn(0x56042051b450 msgr2=0x56042051e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.598+0000 7f0f861a7740 1 -- 172.21.15.40:0/3778160260 shutdown_connections 2024-04-27T00:02:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.599+0000 7f0f861a7740 1 -- 172.21.15.40:0/3778160260 wait complete. 2024-04-27T00:02:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.599+0000 7f0f861a7740 20 client.15330 trim_cache size 0 max 16384 2024-04-27T00:02:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:02:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:02:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:02:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:02:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.599+0000 7f0f81c00640 1 -- 172.21.15.40:0/2044405716 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f600a3cb0 msgr2=0x7f0f600a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.599+0000 7f0f81c00640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f600a3cb0 0x7f0f600a6170 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x560420520bd0 tx=0x7f0f6c005f90 comp rx=0 tx=0).stop 2024-04-27T00:02:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.599+0000 7f0f81c00640 1 -- 172.21.15.40:0/2044405716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f7c101a00 msgr2=0x7f0f7c19b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.599+0000 7f0f81c00640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f7c101a00 0x7f0f7c19b130 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0f68005940 tx=0x7f0f680042f0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.600+0000 7f0f81c00640 1 -- 172.21.15.40:0/2044405716 shutdown_connections 2024-04-27T00:02:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.600+0000 7f0f81c00640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0f600a3cb0 0x7f0f600a6170 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.600+0000 7f0f81c00640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f7c1023a0 0x7f0f7c1a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.600+0000 7f0f81c00640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f7c101a00 0x7f0f7c19b130 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.600+0000 7f0f81c00640 1 --2- 172.21.15.40:0/2044405716 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0f7c1010a0 0x7f0f7c19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.600+0000 7f0f81c00640 1 -- 172.21.15.40:0/2044405716 >> 172.21.15.40:0/2044405716 conn(0x7f0f7c0fcc20 msgr2=0x7f0f7c0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.600+0000 7f0f81c00640 1 -- 172.21.15.40:0/2044405716 shutdown_connections 2024-04-27T00:02:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:41.600+0000 7f0f81c00640 1 -- 172.21.15.40:0/2044405716 wait complete. 2024-04-27T00:02:42.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:42 smithi040 ceph-mon[28095]: pgmap v442: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-27T00:02:42.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:42 smithi080 ceph-mon[32362]: pgmap v442: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-27T00:02:42.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:42 smithi103 ceph-mon[32952]: pgmap v442: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-27T00:02:44.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:44 smithi040 ceph-mon[28095]: pgmap v443: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-27T00:02:44.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:44 smithi080 ceph-mon[32362]: pgmap v443: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-27T00:02:44.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:44 smithi103 ceph-mon[32952]: pgmap v443: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-27T00:02:46.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:46 smithi040 ceph-mon[28095]: pgmap v444: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-27T00:02:46.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:46 smithi080 ceph-mon[32362]: pgmap v444: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-27T00:02:46.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:46 smithi103 ceph-mon[32952]: pgmap v444: 129 pgs: 129 active+clean; 659 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-27T00:02:48.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:48 smithi040 ceph-mon[28095]: pgmap v445: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:02:48.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:48 smithi080 ceph-mon[32362]: pgmap v445: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:02:48.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:48 smithi103 ceph-mon[32952]: pgmap v445: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:02:50.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:50 smithi040 ceph-mon[28095]: pgmap v446: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:50.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:50 smithi080 ceph-mon[32362]: pgmap v446: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:50.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:50 smithi103 ceph-mon[32952]: pgmap v446: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:52.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:52 smithi040 ceph-mon[28095]: pgmap v447: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:52.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:52 smithi080 ceph-mon[32362]: pgmap v447: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:52.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:52 smithi103 ceph-mon[32952]: pgmap v447: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:54.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:54 smithi040 ceph-mon[28095]: pgmap v448: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:02:54.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:54 smithi080 ceph-mon[32362]: pgmap v448: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:02:54.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:54 smithi103 ceph-mon[32952]: pgmap v448: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:02:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.791+0000 7f39c7e00640 1 -- 172.21.15.40:0/2296595769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39c8102380 msgr2=0x7f39c8102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.791+0000 7f39c7e00640 1 --2- 172.21.15.40:0/2296595769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39c8102380 0x7f39c8102800 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f39c0009ca0 tx=0x7f39c002a8c0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.791+0000 7f39c7e00640 1 -- 172.21.15.40:0/2296595769 shutdown_connections 2024-04-27T00:02:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.791+0000 7f39c7e00640 1 --2- 172.21.15.40:0/2296595769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39c8102380 0x7f39c8102800 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.791+0000 7f39c7e00640 1 --2- 172.21.15.40:0/2296595769 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f39c81019e0 0x7f39c8101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.791+0000 7f39c7e00640 1 --2- 172.21.15.40:0/2296595769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39c8101080 0x7f39c81014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.791+0000 7f39c7e00640 1 -- 172.21.15.40:0/2296595769 >> 172.21.15.40:0/2296595769 conn(0x7f39c80fcc20 msgr2=0x7f39c80ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.791+0000 7f39c7e00640 1 -- 172.21.15.40:0/2296595769 shutdown_connections 2024-04-27T00:02:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.791+0000 7f39c7e00640 1 -- 172.21.15.40:0/2296595769 wait complete. 2024-04-27T00:02:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.793+0000 7f39c7e00640 1 Processor -- start 2024-04-27T00:02:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.793+0000 7f39c7e00640 1 -- start start 2024-04-27T00:02:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.793+0000 7f39c7e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f39c8101080 0x7f39c819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.793+0000 7f39c7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39c81019e0 0x7f39c819b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.793+0000 7f39c7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39c8102380 0x7f39c81a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.793+0000 7f39c7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f39c8110520 con 0x7f39c81019e0 2024-04-27T00:02:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.793+0000 7f39c7e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81103a0 con 0x7f39c8101080 2024-04-27T00:02:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.793+0000 7f39c7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81106a0 con 0x7f39c8102380 2024-04-27T00:02:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.794+0000 7f39c6a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f39c8101080 0x7f39c819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.794+0000 7f39c6a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f39c8101080 0x7f39c819abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:36990/0 (socket says 172.21.15.40:36990) 2024-04-27T00:02:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.794+0000 7f39c6a00640 1 -- 172.21.15.40:0/3024774951 learned_addr learned my addr 172.21.15.40:0/3024774951 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.794+0000 7f39c6000640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39c81019e0 0x7f39c819b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.794+0000 7f39c7400640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39c8102380 0x7f39c81a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.794+0000 7f39c6a00640 1 -- 172.21.15.40:0/3024774951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39c8102380 msgr2=0x7f39c81a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.794+0000 7f39c6a00640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39c8102380 0x7f39c81a2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.794+0000 7f39c6a00640 1 -- 172.21.15.40:0/3024774951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39c81019e0 msgr2=0x7f39c819b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.794+0000 7f39c6a00640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39c81019e0 0x7f39c819b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.794+0000 7f39c6a00640 1 -- 172.21.15.40:0/3024774951 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c81a27e0 con 0x7f39c8101080 2024-04-27T00:02:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.795+0000 7f39c6a00640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f39c8101080 0x7f39c819abc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f39b800e9f0 tx=0x7f39b800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.795+0000 7f39bf400640 1 -- 172.21.15.40:0/3024774951 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b800cd80 con 0x7f39c8101080 2024-04-27T00:02:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.795+0000 7f39c7e00640 1 -- 172.21.15.40:0/3024774951 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c81a2ad0 con 0x7f39c8101080 2024-04-27T00:02:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.795+0000 7f39bf400640 1 -- 172.21.15.40:0/3024774951 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39b80054d0 con 0x7f39c8101080 2024-04-27T00:02:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.795+0000 7f39bf400640 1 -- 172.21.15.40:0/3024774951 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b8010690 con 0x7f39c8101080 2024-04-27T00:02:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.795+0000 7f39c7e00640 1 -- 172.21.15.40:0/3024774951 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c81a3020 con 0x7f39c8101080 2024-04-27T00:02:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.800+0000 7f39d067c740 1 Processor -- start 2024-04-27T00:02:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.800+0000 7f39d067c740 1 -- start start 2024-04-27T00:02:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.801+0000 7f39bf400640 1 -- 172.21.15.40:0/3024774951 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f39b8004e90 con 0x7f39c8101080 2024-04-27T00:02:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.801+0000 7f39d067c740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc0de0 0x563a7fdc1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.801+0000 7f39d067c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 0x563a7fdc62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.801+0000 7f39d067c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc6940 0x563a7fdc8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.801+0000 7f39d067c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563a7fd25380 con 0x563a7fdc17d0 2024-04-27T00:02:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.801+0000 7f39d067c740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x563a7fcc1c30 con 0x563a7fdc0de0 2024-04-27T00:02:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.801+0000 7f39d067c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563a7fd21340 con 0x563a7fdc6940 2024-04-27T00:02:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.801+0000 7f39c7400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc0de0 0x563a7fdc1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.801+0000 7f39c7400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc0de0 0x563a7fdc1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:36996/0 (socket says 172.21.15.40:36996) 2024-04-27T00:02:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.802+0000 7f39c6000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc6940 0x563a7fdc8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.802+0000 7f39c7400640 1 -- 172.21.15.40:0/1699838781 learned_addr learned my addr 172.21.15.40:0/1699838781 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.802+0000 7f39c6a00640 1 --2- 172.21.15.40:0/1699838781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 0x563a7fdc62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.802+0000 7f39c7400640 1 -- 172.21.15.40:0/1699838781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc6940 msgr2=0x563a7fdc8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.802+0000 7f39c7400640 1 --2- 172.21.15.40:0/1699838781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc6940 0x563a7fdc8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.802+0000 7f39c7400640 1 -- 172.21.15.40:0/1699838781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 msgr2=0x563a7fdc62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.802+0000 7f39c7400640 1 --2- 172.21.15.40:0/1699838781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 0x563a7fdc62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.802+0000 7f39c7400640 1 -- 172.21.15.40:0/1699838781 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a7fce3940 con 0x563a7fdc0de0 2024-04-27T00:02:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.802+0000 7f39c6a00640 1 --2- 172.21.15.40:0/1699838781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 0x563a7fdc62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:02:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.803+0000 7f39bf400640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f39ac0a3d30 0x7f39ac0a61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.803+0000 7f39c7400640 1 --2- 172.21.15.40:0/1699838781 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc0de0 0x563a7fdc1200 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f39c0031b50 tx=0x7f39c0005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=524e33c54f1a629d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.803+0000 7f39bf400640 1 -- 172.21.15.40:0/3024774951 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f39b80d0dd0 con 0x7f39c8101080 2024-04-27T00:02:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.803+0000 7f39c6a00640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f39ac0a3d30 0x7f39ac0a61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.804+0000 7f39b3e00640 1 -- 172.21.15.40:0/1699838781 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0031650 con 0x563a7fdc0de0 2024-04-27T00:02:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.804+0000 7f39b3e00640 1 -- 172.21.15.40:0/1699838781 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39c0036040 con 0x563a7fdc0de0 2024-04-27T00:02:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.804+0000 7f39b3e00640 1 -- 172.21.15.40:0/1699838781 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0004600 con 0x563a7fdc0de0 2024-04-27T00:02:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.805+0000 7f39d067c740 1 -- 172.21.15.40:0/1699838781 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc0de0 msgr2=0x563a7fdc1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.805+0000 7f39d067c740 1 --2- 172.21.15.40:0/1699838781 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc0de0 0x563a7fdc1200 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f39c0031b50 tx=0x7f39c0005ef0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.805+0000 7f39c6a00640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f39ac0a3d30 0x7f39ac0a61f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x563a7fdc1bd0 tx=0x7f39b80d4800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.805+0000 7f39d067c740 1 -- 172.21.15.40:0/1699838781 shutdown_connections 2024-04-27T00:02:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.805+0000 7f39d067c740 1 --2- 172.21.15.40:0/1699838781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc6940 0x563a7fdc8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.805+0000 7f39d067c740 1 --2- 172.21.15.40:0/1699838781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 0x563a7fdc62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.805+0000 7f39d067c740 1 --2- 172.21.15.40:0/1699838781 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc0de0 0x563a7fdc1200 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.805+0000 7f39d067c740 1 -- 172.21.15.40:0/1699838781 >> 172.21.15.40:0/1699838781 conn(0x563a7fdbc450 msgr2=0x563a7fdbc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.805+0000 7f39d067c740 1 -- 172.21.15.40:0/1699838781 shutdown_connections 2024-04-27T00:02:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.805+0000 7f39d067c740 1 -- 172.21.15.40:0/1699838781 wait complete. 2024-04-27T00:02:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.806+0000 7f39d067c740 1 -- start start 2024-04-27T00:02:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.806+0000 7f39d067c740 1 Processor -- start 2024-04-27T00:02:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.807+0000 7f39d067c740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc0de0 0x563a7fdcaf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.807+0000 7f39d067c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 0x563a7fdcb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.807+0000 7f39d067c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc6940 0x563a7fdd24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.807+0000 7f39d067c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563a7fdc9890 con 0x563a7fdc17d0 2024-04-27T00:02:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.807+0000 7f39d067c740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x563a7fcacc60 con 0x563a7fdc0de0 2024-04-27T00:02:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.807+0000 7f39d067c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563a7fdc9a10 con 0x563a7fdc6940 2024-04-27T00:02:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.807+0000 7f39d067c740 10 client.0 _ll_register_callbacks cb 0x563a7fcd8e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:02:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.807+0000 7f39c7400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc0de0 0x563a7fdcaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.807+0000 7f39c6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 0x563a7fdcb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.807+0000 7f39c6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 0x563a7fdcb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45420/0 (socket says 172.21.15.40:45420) 2024-04-27T00:02:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.807+0000 7f39c6000640 1 -- 172.21.15.40:0/517681656 learned_addr learned my addr 172.21.15.40:0/517681656 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.808+0000 7f39c7400640 1 --2- 172.21.15.40:0/517681656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc6940 0x563a7fdd24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.808+0000 7f39c6000640 1 -- 172.21.15.40:0/517681656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc6940 msgr2=0x563a7fdd24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.808+0000 7f39c6000640 1 --2- 172.21.15.40:0/517681656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc6940 0x563a7fdd24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.808+0000 7f39c6000640 1 -- 172.21.15.40:0/517681656 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc0de0 msgr2=0x563a7fdcaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.808+0000 7f39c6000640 1 --2- 172.21.15.40:0/517681656 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc0de0 0x563a7fdcaf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.808+0000 7f39c6000640 1 -- 172.21.15.40:0/517681656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a7fcbb510 con 0x563a7fdc17d0 2024-04-27T00:02:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.809+0000 7f39c6000640 1 --2- 172.21.15.40:0/517681656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 0x563a7fdcb480 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f39b4012720 tx=0x7f39b4012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b475f5be886a08 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.809+0000 7f39b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:02:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.809+0000 7f39b0c00640 1 -- 172.21.15.40:0/517681656 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b400f8c0 con 0x563a7fdc17d0 2024-04-27T00:02:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.809+0000 7f39d067c740 1 -- 172.21.15.40:0/517681656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39c0036040 con 0x563a7fdc17d0 2024-04-27T00:02:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.809+0000 7f39b0c00640 1 -- 172.21.15.40:0/517681656 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f39b400fa60 con 0x563a7fdc17d0 2024-04-27T00:02:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.810+0000 7f39b0c00640 1 -- 172.21.15.40:0/517681656 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f39b400fca0 con 0x563a7fdc17d0 2024-04-27T00:02:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.810+0000 7f39d067c740 10 client.15342 fetch_fsmap learned FSMap version 24 2024-04-27T00:02:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.810+0000 7f39d067c740 1 -- 172.21.15.40:0/517681656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563a7fa37210 con 0x563a7fdc17d0 2024-04-27T00:02:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.810+0000 7f39b0c00640 1 -- 172.21.15.40:0/517681656 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39c0036040 con 0x563a7fdc17d0 2024-04-27T00:02:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.811+0000 7f39b0c00640 1 -- 172.21.15.40:0/517681656 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f39b4010430 con 0x563a7fdc17d0 2024-04-27T00:02:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.811+0000 7f39d067c740 10 client.15342 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:02:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.811+0000 7f39d067c740 10 client.15342 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:02:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.811+0000 7f39d067c740 20 client.15342 populate_metadata read hostname 'smithi040' 2024-04-27T00:02:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.811+0000 7f39d067c740 1 --2- 172.21.15.40:0/517681656 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563a7fdd6a90 0x563a7fdd8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.811+0000 7f39d067c740 4 client.15342 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:02:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.811+0000 7f39d067c740 1 -- 172.21.15.40:0/517681656 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a7fd32f90 con 0x563a7fdd6a90 2024-04-27T00:02:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.812+0000 7f39c7400640 1 --2- 172.21.15.40:0/517681656 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563a7fdd6a90 0x563a7fdd8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.812+0000 7f39c7400640 1 --2- 172.21.15.40:0/517681656 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563a7fdd6a90 0x563a7fdd8ec0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf62a9cf6f6da9b2 server_cookie=7c4e3831f55f9c34 in_seq=0 out_seq=0 2024-04-27T00:02:54.814 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.812+0000 7f39b0c00640 10 client.15342 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.813+0000 7f39b0c00640 1 -- 172.21.15.40:0/517681656 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a7fd32f90 con 0x563a7fdd6a90 2024-04-27T00:02:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.813+0000 7f39b0c00640 10 client.15342 handle_command_reply: tid=0 2024-04-27T00:02:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.813+0000 7f39d067c740 1 client.15342 shutdown 2024-04-27T00:02:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.814+0000 7f39d067c740 1 -- 172.21.15.40:0/517681656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 msgr2=0x563a7fdcb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.814+0000 7f39d067c740 1 --2- 172.21.15.40:0/517681656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 0x563a7fdcb480 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f39b4012720 tx=0x7f39b4012be0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.814+0000 7f39d067c740 1 -- 172.21.15.40:0/517681656 shutdown_connections 2024-04-27T00:02:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.814+0000 7f39d067c740 1 --2- 172.21.15.40:0/517681656 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563a7fdd6a90 0x563a7fdd8ec0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.814+0000 7f39d067c740 1 --2- 172.21.15.40:0/517681656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc6940 0x563a7fdd24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.814+0000 7f39d067c740 1 --2- 172.21.15.40:0/517681656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc17d0 0x563a7fdcb480 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.814+0000 7f39b0c00640 0 client.15342 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.814+0000 7f39d067c740 1 --2- 172.21.15.40:0/517681656 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc0de0 0x563a7fdcaf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.814+0000 7f39d067c740 1 -- 172.21.15.40:0/517681656 >> 172.21.15.40:0/517681656 conn(0x563a7fdbc450 msgr2=0x563a7fdbb650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.814+0000 7f39d067c740 1 -- 172.21.15.40:0/517681656 shutdown_connections 2024-04-27T00:02:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.814+0000 7f39d067c740 1 -- 172.21.15.40:0/517681656 wait complete. 2024-04-27T00:02:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.814+0000 7f39d067c740 20 client.15342 trim_cache size 0 max 16384 2024-04-27T00:02:54.847 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.846+0000 7f39d067c740 1 Processor -- start 2024-04-27T00:02:54.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.846+0000 7f39d067c740 1 -- start start 2024-04-27T00:02:54.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.847+0000 7f39d067c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc0de0 0x563a7fdbfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.847+0000 7f39d067c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc17d0 0x563a7fdd2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.847+0000 7f39d067c740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc6940 0x563a7fddd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.847+0000 7f39d067c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81afe40 con 0x563a7fdc0de0 2024-04-27T00:02:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.847+0000 7f39d067c740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81ad710 con 0x563a7fdc6940 2024-04-27T00:02:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.847+0000 7f39d067c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81ab560 con 0x563a7fdc17d0 2024-04-27T00:02:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.847+0000 7f39c7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc0de0 0x563a7fdbfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.847+0000 7f39c7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc0de0 0x563a7fdbfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45426/0 (socket says 172.21.15.40:45426) 2024-04-27T00:02:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.847+0000 7f39c7400640 1 -- 172.21.15.40:0/2549280216 learned_addr learned my addr 172.21.15.40:0/2549280216 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.847+0000 7f39c6000640 1 --2- 172.21.15.40:0/2549280216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc17d0 0x563a7fdd2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.847+0000 7f39c7400640 1 --2- 172.21.15.40:0/2549280216 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc6940 0x563a7fddd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.848+0000 7f39c6000640 1 -- 172.21.15.40:0/2549280216 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc6940 msgr2=0x563a7fddd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.848+0000 7f39c6000640 1 --2- 172.21.15.40:0/2549280216 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc6940 0x563a7fddd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.848+0000 7f39c6000640 1 -- 172.21.15.40:0/2549280216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc0de0 msgr2=0x563a7fdbfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.848+0000 7f39c6000640 1 --2- 172.21.15.40:0/2549280216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc0de0 0x563a7fdbfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.848+0000 7f39c6000640 1 -- 172.21.15.40:0/2549280216 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a7fb03620 con 0x563a7fdc17d0 2024-04-27T00:02:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.848+0000 7f39c6000640 1 --2- 172.21.15.40:0/2549280216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc17d0 0x563a7fdd2480 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f39b40126f0 tx=0x7f39b4009f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b8645d72a8edb46 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:54.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.848+0000 7f39b3400640 1 -- 172.21.15.40:0/2549280216 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b400fe40 con 0x563a7fdc17d0 2024-04-27T00:02:54.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.848+0000 7f39b3400640 1 -- 172.21.15.40:0/2549280216 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39b4010430 con 0x563a7fdc17d0 2024-04-27T00:02:54.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.848+0000 7f39b3400640 1 -- 172.21.15.40:0/2549280216 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4005330 con 0x563a7fdc17d0 2024-04-27T00:02:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.849+0000 7f39d067c740 1 -- 172.21.15.40:0/2549280216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc17d0 msgr2=0x563a7fdd2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.849+0000 7f39d067c740 1 --2- 172.21.15.40:0/2549280216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc17d0 0x563a7fdd2480 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f39b40126f0 tx=0x7f39b4009f40 comp rx=0 tx=0).stop 2024-04-27T00:02:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.849+0000 7f39d067c740 1 -- 172.21.15.40:0/2549280216 shutdown_connections 2024-04-27T00:02:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.849+0000 7f39d067c740 1 --2- 172.21.15.40:0/2549280216 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc6940 0x563a7fddd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.849+0000 7f39d067c740 1 --2- 172.21.15.40:0/2549280216 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc17d0 0x563a7fdd2480 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.849+0000 7f39d067c740 1 --2- 172.21.15.40:0/2549280216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc0de0 0x563a7fdbfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.849+0000 7f39d067c740 1 -- 172.21.15.40:0/2549280216 >> 172.21.15.40:0/2549280216 conn(0x563a7fdbc450 msgr2=0x563a7fdbf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.850+0000 7f39d067c740 1 -- 172.21.15.40:0/2549280216 shutdown_connections 2024-04-27T00:02:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.850+0000 7f39d067c740 1 -- 172.21.15.40:0/2549280216 wait complete. 2024-04-27T00:02:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.850+0000 7f39d067c740 1 -- start start 2024-04-27T00:02:54.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.850+0000 7f39d067c740 1 Processor -- start 2024-04-27T00:02:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.851+0000 7f39d067c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc0de0 0x563a7fdd1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.851+0000 7f39d067c740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc17d0 0x563a7fdd21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.851+0000 7f39d067c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc6940 0x563a7fdd02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.851+0000 7f39d067c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81abbd0 con 0x563a7fdc6940 2024-04-27T00:02:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.851+0000 7f39d067c740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81ad040 con 0x563a7fdc17d0 2024-04-27T00:02:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.851+0000 7f39d067c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81ac2a0 con 0x563a7fdc0de0 2024-04-27T00:02:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.851+0000 7f39d067c740 10 client.0 _ll_register_callbacks cb 0x7f39b4009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:02:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.851+0000 7f39c6000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc17d0 0x563a7fdd21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.851+0000 7f39c6000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc17d0 0x563a7fdd21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:37026/0 (socket says 172.21.15.40:37026) 2024-04-27T00:02:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.851+0000 7f39c6000640 1 -- 172.21.15.40:0/1378736193 learned_addr learned my addr 172.21.15.40:0/1378736193 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:02:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.852+0000 7f39c7400640 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc0de0 0x563a7fdd1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.852+0000 7f39c7400640 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc6940 0x563a7fdd02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.852+0000 7f39c6000640 1 -- 172.21.15.40:0/1378736193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc0de0 msgr2=0x563a7fdd1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.852+0000 7f39c6000640 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc0de0 0x563a7fdd1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.852+0000 7f39c6000640 1 -- 172.21.15.40:0/1378736193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc6940 msgr2=0x563a7fdd02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.852+0000 7f39c6000640 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc6940 0x563a7fdd02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.852+0000 7f39c6000640 1 -- 172.21.15.40:0/1378736193 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a7fcdfcb0 con 0x563a7fdc17d0 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.852+0000 7f39c6000640 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc17d0 0x563a7fdd21f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f39b40108f0 tx=0x7f39b4010970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19850eedf71a76a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.852+0000 7f39b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.852+0000 7f39b0c00640 1 -- 172.21.15.40:0/1378736193 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b400fe40 con 0x563a7fdc17d0 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.853+0000 7f39b0c00640 1 -- 172.21.15.40:0/1378736193 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f39b40050d0 con 0x563a7fdc17d0 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.853+0000 7f39d067c740 1 -- 172.21.15.40:0/1378736193 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39b4010430 con 0x563a7fdc17d0 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.853+0000 7f39b0c00640 1 -- 172.21.15.40:0/1378736193 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f39b4038050 con 0x563a7fdc17d0 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.853+0000 7f39d067c740 10 client.25472 fetch_fsmap learned FSMap version 24 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.853+0000 7f39d067c740 1 -- 172.21.15.40:0/1378736193 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563a7fe45fc0 con 0x563a7fdc17d0 2024-04-27T00:02:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.853+0000 7f39b0c00640 1 -- 172.21.15.40:0/1378736193 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39b4010430 con 0x563a7fdc17d0 2024-04-27T00:02:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.854+0000 7f39b0c00640 1 -- 172.21.15.40:0/1378736193 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f39b4030020 con 0x563a7fdc17d0 2024-04-27T00:02:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.854+0000 7f39d067c740 10 client.25472 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:02:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.854+0000 7f39d067c740 10 client.25472 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:02:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.854+0000 7f39d067c740 20 client.25472 populate_metadata read hostname 'smithi040' 2024-04-27T00:02:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.854+0000 7f39d067c740 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563a7fe469e0 0x563a7fe48e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:02:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.854+0000 7f39d067c740 4 client.25472 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:02:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.854+0000 7f39d067c740 1 -- 172.21.15.40:0/1378736193 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563a7fd2ce00 con 0x563a7fe469e0 2024-04-27T00:02:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.854+0000 7f39c7400640 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563a7fe469e0 0x563a7fe48e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:02:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.855+0000 7f39c7400640 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563a7fe469e0 0x563a7fe48e00 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb663603793276a server_cookie=67857455b71d13a5 in_seq=0 out_seq=0 2024-04-27T00:02:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.855+0000 7f39b0c00640 10 client.25472 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.859+0000 7f39b0c00640 1 -- 172.21.15.40:0/1378736193 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563a7fd2ce00 con 0x563a7fe469e0 2024-04-27T00:02:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.859+0000 7f39b0c00640 10 client.25472 handle_command_reply: tid=0 2024-04-27T00:02:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39d067c740 1 client.25472 shutdown 2024-04-27T00:02:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39d067c740 1 -- 172.21.15.40:0/1378736193 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc17d0 msgr2=0x563a7fdd21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39d067c740 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc17d0 0x563a7fdd21f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f39b40108f0 tx=0x7f39b4010970 comp rx=0 tx=0).stop 2024-04-27T00:02:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39d067c740 1 -- 172.21.15.40:0/1378736193 shutdown_connections 2024-04-27T00:02:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39d067c740 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563a7fe469e0 0x563a7fe48e00 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39d067c740 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7fdc6940 0x563a7fdd02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39d067c740 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563a7fdc17d0 0x563a7fdd21f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39d067c740 1 --2- 172.21.15.40:0/1378736193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7fdc0de0 0x563a7fdd1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39d067c740 1 -- 172.21.15.40:0/1378736193 >> 172.21.15.40:0/1378736193 conn(0x563a7fdbc450 msgr2=0x563a7fdbadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39b0c00640 0 client.25472 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:02:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39d067c740 1 -- 172.21.15.40:0/1378736193 shutdown_connections 2024-04-27T00:02:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39d067c740 1 -- 172.21.15.40:0/1378736193 wait complete. 2024-04-27T00:02:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.860+0000 7f39d067c740 20 client.25472 trim_cache size 0 max 16384 2024-04-27T00:02:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:02:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:02:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:02:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:02:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.861+0000 7f39c7e00640 1 -- 172.21.15.40:0/3024774951 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f39ac0a3d30 msgr2=0x7f39ac0a61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.861+0000 7f39c7e00640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f39ac0a3d30 0x7f39ac0a61f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x563a7fdc1bd0 tx=0x7f39b80d4800 comp rx=0 tx=0).stop 2024-04-27T00:02:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.861+0000 7f39c7e00640 1 -- 172.21.15.40:0/3024774951 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f39c8101080 msgr2=0x7f39c819abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:02:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.861+0000 7f39c7e00640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f39c8101080 0x7f39c819abc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f39b800e9f0 tx=0x7f39b800edf0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.862+0000 7f39c7e00640 1 -- 172.21.15.40:0/3024774951 shutdown_connections 2024-04-27T00:02:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.862+0000 7f39c7e00640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f39ac0a3d30 0x7f39ac0a61f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.862+0000 7f39c7e00640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39c8102380 0x7f39c81a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.862+0000 7f39c7e00640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39c81019e0 0x7f39c819b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.862+0000 7f39c7e00640 1 --2- 172.21.15.40:0/3024774951 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f39c8101080 0x7f39c819abc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:02:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.862+0000 7f39c7e00640 1 -- 172.21.15.40:0/3024774951 >> 172.21.15.40:0/3024774951 conn(0x7f39c80fcc20 msgr2=0x7f39c80fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:02:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.862+0000 7f39c7e00640 1 -- 172.21.15.40:0/3024774951 shutdown_connections 2024-04-27T00:02:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:02:54.862+0000 7f39c7e00640 1 -- 172.21.15.40:0/3024774951 wait complete. 2024-04-27T00:02:56.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:56 smithi040 ceph-mon[28095]: pgmap v449: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:56.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:56 smithi080 ceph-mon[32362]: pgmap v449: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:56.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:56 smithi103 ceph-mon[32952]: pgmap v449: 129 pgs: 129 active+clean; 660 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:02:58.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:02:58 smithi040 ceph-mon[28095]: pgmap v450: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-27T00:02:58.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:02:58 smithi080 ceph-mon[32362]: pgmap v450: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-27T00:02:58.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:02:58 smithi103 ceph-mon[32952]: pgmap v450: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-27T00:03:00.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:00 smithi040 ceph-mon[28095]: pgmap v451: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:03:00.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:00 smithi080 ceph-mon[32362]: pgmap v451: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:03:00.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:00 smithi103 ceph-mon[32952]: pgmap v451: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:03:02.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:02 smithi040 ceph-mon[28095]: pgmap v452: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:03:02.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:02 smithi080 ceph-mon[32362]: pgmap v452: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:03:02.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:02 smithi103 ceph-mon[32952]: pgmap v452: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:03:04.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:04 smithi040 ceph-mon[28095]: pgmap v453: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:03:04.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:04 smithi080 ceph-mon[32362]: pgmap v453: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:03:04.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:04 smithi103 ceph-mon[32952]: pgmap v453: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:03:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:06 smithi040 ceph-mon[28095]: pgmap v454: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:03:06.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:06 smithi080 ceph-mon[32362]: pgmap v454: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:03:06.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:06 smithi103 ceph-mon[32952]: pgmap v454: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T00:03:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.993+0000 7f89f3200640 1 -- 172.21.15.40:0/3965703710 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89f41027f0 msgr2=0x7f89f4102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.993+0000 7f89f3200640 1 --2- 172.21.15.40:0/3965703710 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89f41027f0 0x7f89f4102c10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009860 tx=0x7f89e402a340 comp rx=0 tx=0).stop 2024-04-27T00:03:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.993+0000 7f89f3200640 1 -- 172.21.15.40:0/3965703710 shutdown_connections 2024-04-27T00:03:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.993+0000 7f89f3200640 1 --2- 172.21.15.40:0/3965703710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89f41043a0 0x7f89f410ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.993+0000 7f89f3200640 1 --2- 172.21.15.40:0/3965703710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89f41039e0 0x7f89f4103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.993+0000 7f89f3200640 1 --2- 172.21.15.40:0/3965703710 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89f41027f0 0x7f89f4102c10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.993+0000 7f89f3200640 1 -- 172.21.15.40:0/3965703710 >> 172.21.15.40:0/3965703710 conn(0x7f89f40fdd60 msgr2=0x7f89f41001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.994+0000 7f89f3200640 1 -- 172.21.15.40:0/3965703710 shutdown_connections 2024-04-27T00:03:07.996 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.994+0000 7f89f3200640 1 -- 172.21.15.40:0/3965703710 wait complete. 2024-04-27T00:03:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.995+0000 7f89f3200640 1 Processor -- start 2024-04-27T00:03:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.995+0000 7f89f3200640 1 -- start start 2024-04-27T00:03:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.995+0000 7f89f3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89f41027f0 0x7f89f419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.995+0000 7f89f3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89f41039e0 0x7f89f419b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.995+0000 7f89f3200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89f41043a0 0x7f89f41a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.995+0000 7f89f3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f89f410e2f0 con 0x7f89f41027f0 2024-04-27T00:03:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.995+0000 7f89f3200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f89f410e170 con 0x7f89f41043a0 2024-04-27T00:03:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.995+0000 7f89f3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89f410e470 con 0x7f89f41039e0 2024-04-27T00:03:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.995+0000 7f89f1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89f41027f0 0x7f89f419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.995+0000 7f89f1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89f41027f0 0x7f89f419ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54396/0 (socket says 172.21.15.40:54396) 2024-04-27T00:03:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.995+0000 7f89f1e00640 1 -- 172.21.15.40:0/3671665283 learned_addr learned my addr 172.21.15.40:0/3671665283 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.995+0000 7f89f1400640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89f41039e0 0x7f89f419b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.996+0000 7f89f2800640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89f41043a0 0x7f89f41a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.996+0000 7f89f1400640 1 -- 172.21.15.40:0/3671665283 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89f41043a0 msgr2=0x7f89f41a21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.996+0000 7f89f1400640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89f41043a0 0x7f89f41a21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.996+0000 7f89f1400640 1 -- 172.21.15.40:0/3671665283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89f41027f0 msgr2=0x7f89f419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.996+0000 7f89f1400640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89f41027f0 0x7f89f419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.996+0000 7f89f1400640 1 -- 172.21.15.40:0/3671665283 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f41a2860 con 0x7f89f41039e0 2024-04-27T00:03:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.996+0000 7f89f1400640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89f41039e0 0x7f89f419b190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f89e0005fd0 tx=0x7f89e000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.997+0000 7f89eaa00640 1 -- 172.21.15.40:0/3671665283 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e000cd00 con 0x7f89f41039e0 2024-04-27T00:03:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.997+0000 7f89eaa00640 1 -- 172.21.15.40:0/3671665283 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89e00035f0 con 0x7f89f41039e0 2024-04-27T00:03:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.997+0000 7f89eaa00640 1 -- 172.21.15.40:0/3671665283 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e0010690 con 0x7f89f41039e0 2024-04-27T00:03:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.997+0000 7f89f3200640 1 -- 172.21.15.40:0/3671665283 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89f41a2b50 con 0x7f89f41039e0 2024-04-27T00:03:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:07.997+0000 7f89f3200640 1 -- 172.21.15.40:0/3671665283 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89f41a3050 con 0x7f89f41039e0 2024-04-27T00:03:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.001+0000 7f89fc175740 1 Processor -- start 2024-04-27T00:03:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.001+0000 7f89fc175740 1 -- start start 2024-04-27T00:03:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.001+0000 7f89fc175740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 0x562b46295200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.002+0000 7f89fc175740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b462957d0 0x562b4629a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.002+0000 7f89fc175740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b4629a940 0x562b4629cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.002+0000 7f89fc175740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562b461f9380 con 0x562b46294de0 2024-04-27T00:03:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.002+0000 7f89fc175740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x562b46195c30 con 0x562b4629a940 2024-04-27T00:03:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.002+0000 7f89fc175740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562b461f5340 con 0x562b462957d0 2024-04-27T00:03:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.002+0000 7f89f2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 0x562b46295200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.002+0000 7f89f2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 0x562b46295200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54412/0 (socket says 172.21.15.40:54412) 2024-04-27T00:03:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.002+0000 7f89f2800640 1 -- 172.21.15.40:0/2824071636 learned_addr learned my addr 172.21.15.40:0/2824071636 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.002+0000 7f89f1e00640 1 --2- 172.21.15.40:0/2824071636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b462957d0 0x562b4629a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.003+0000 7f89f1e00640 1 -- 172.21.15.40:0/2824071636 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b4629a940 msgr2=0x562b4629cdd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-27T00:03:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.003+0000 7f89f1e00640 1 --2- 172.21.15.40:0/2824071636 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b4629a940 0x562b4629cdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.003+0000 7f89f1e00640 1 -- 172.21.15.40:0/2824071636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 msgr2=0x562b46295200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.003+0000 7f89f1e00640 1 --2- 172.21.15.40:0/2824071636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 0x562b46295200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.003+0000 7f89f1e00640 1 -- 172.21.15.40:0/2824071636 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b461b7940 con 0x562b462957d0 2024-04-27T00:03:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.003+0000 7f89f2800640 1 --2- 172.21.15.40:0/2824071636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 0x562b46295200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:03:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.003+0000 7f89eaa00640 1 -- 172.21.15.40:0/3671665283 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f89e0003a80 con 0x7f89f41039e0 2024-04-27T00:03:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.003+0000 7f89f1e00640 1 --2- 172.21.15.40:0/2824071636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b462957d0 0x562b4629a2d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009830 tx=0x7f89e4005b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fee429ae6f4c2f88 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.004+0000 7f89df400640 1 -- 172.21.15.40:0/2824071636 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e4041020 con 0x562b462957d0 2024-04-27T00:03:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.004+0000 7f89df400640 1 -- 172.21.15.40:0/2824071636 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89e4030dd0 con 0x562b462957d0 2024-04-27T00:03:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.004+0000 7f89df400640 1 -- 172.21.15.40:0/2824071636 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e4037040 con 0x562b462957d0 2024-04-27T00:03:08.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.005+0000 7f89fc175740 1 -- 172.21.15.40:0/2824071636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b462957d0 msgr2=0x562b4629a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:08.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.005+0000 7f89fc175740 1 --2- 172.21.15.40:0/2824071636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b462957d0 0x562b4629a2d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009830 tx=0x7f89e4005b20 comp rx=0 tx=0).stop 2024-04-27T00:03:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.005+0000 7f89fc175740 1 -- 172.21.15.40:0/2824071636 shutdown_connections 2024-04-27T00:03:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.005+0000 7f89fc175740 1 --2- 172.21.15.40:0/2824071636 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b4629a940 0x562b4629cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.005+0000 7f89fc175740 1 --2- 172.21.15.40:0/2824071636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b462957d0 0x562b4629a2d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.005+0000 7f89fc175740 1 --2- 172.21.15.40:0/2824071636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 0x562b46295200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.005+0000 7f89fc175740 1 -- 172.21.15.40:0/2824071636 >> 172.21.15.40:0/2824071636 conn(0x562b46290450 msgr2=0x562b46290850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.005+0000 7f89eaa00640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f89d80a3d30 0x7f89d80a61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.005+0000 7f89fc175740 1 -- 172.21.15.40:0/2824071636 shutdown_connections 2024-04-27T00:03:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.005+0000 7f89eaa00640 1 -- 172.21.15.40:0/3671665283 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f89e00d0cd0 con 0x7f89f41039e0 2024-04-27T00:03:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.006+0000 7f89f1e00640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f89d80a3d30 0x7f89d80a61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.006+0000 7f89fc175740 1 -- 172.21.15.40:0/2824071636 wait complete. 2024-04-27T00:03:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.006+0000 7f89fc175740 1 -- start start 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.006+0000 7f89fc175740 1 Processor -- start 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89f1e00640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f89d80a3d30 0x7f89d80a61f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f89e4031000 tx=0x7f89e402a340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89fc175740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b46294de0 0x562b4629f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89fc175740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b462957d0 0x562b4629f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89fc175740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b4629a940 0x562b462a6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89fc175740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562b4629d890 con 0x562b4629a940 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89fc175740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x562b46180c60 con 0x562b46294de0 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89fc175740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562b4629da10 con 0x562b462957d0 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89fc175740 10 client.0 _ll_register_callbacks cb 0x562b461ace50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89f2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b4629a940 0x562b462a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89f2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b4629a940 0x562b462a6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54414/0 (socket says 172.21.15.40:54414) 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89f1400640 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b462957d0 0x562b4629f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89f2800640 1 -- 172.21.15.40:0/236171784 learned_addr learned my addr 172.21.15.40:0/236171784 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.007+0000 7f89f1e00640 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b46294de0 0x562b4629f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.008+0000 7f89f1400640 1 -- 172.21.15.40:0/236171784 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b46294de0 msgr2=0x562b4629f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.008+0000 7f89f1400640 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b46294de0 0x562b4629f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.008+0000 7f89f1400640 1 -- 172.21.15.40:0/236171784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b4629a940 msgr2=0x562b462a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.008+0000 7f89f1400640 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b4629a940 0x562b462a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.008+0000 7f89f1400640 1 -- 172.21.15.40:0/236171784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b4618f510 con 0x562b462957d0 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.008+0000 7f89f1400640 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b462957d0 0x562b4629f5d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f89e00ce280 tx=0x7f89e00039d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=117703f6f36eddf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.008+0000 7f89d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.008+0000 7f89d3e00640 1 -- 172.21.15.40:0/236171784 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e00bd0d0 con 0x562b462957d0 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.008+0000 7f89fc175740 1 -- 172.21.15.40:0/236171784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89e4030dd0 con 0x562b462957d0 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.008+0000 7f89d3e00640 1 -- 172.21.15.40:0/236171784 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f89e00ce4d0 con 0x562b462957d0 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.009+0000 7f89d3e00640 1 -- 172.21.15.40:0/236171784 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f89e009d150 con 0x562b462957d0 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.009+0000 7f89fc175740 10 client.25504 fetch_fsmap learned FSMap version 24 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.009+0000 7f89fc175740 1 -- 172.21.15.40:0/236171784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562b45f0b210 con 0x562b462957d0 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.009+0000 7f89d3e00640 1 -- 172.21.15.40:0/236171784 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89e4030dd0 con 0x562b462957d0 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.009+0000 7f89d3e00640 1 -- 172.21.15.40:0/236171784 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f89e00870c0 con 0x562b462957d0 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.009+0000 7f89fc175740 10 client.25504 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.010+0000 7f89fc175740 10 client.25504 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.010+0000 7f89fc175740 20 client.25504 populate_metadata read hostname 'smithi040' 2024-04-27T00:03:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.010+0000 7f89fc175740 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b462aabb0 0x562b462acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.010+0000 7f89fc175740 4 client.25504 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:03:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.010+0000 7f89fc175740 1 -- 172.21.15.40:0/236171784 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b46206f90 con 0x562b462aabb0 2024-04-27T00:03:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.010+0000 7f89f1e00640 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b462aabb0 0x562b462acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.011+0000 7f89f1e00640 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b462aabb0 0x562b462acfa0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abb12b3dca59b594 server_cookie=236b6270b06aada6 in_seq=0 out_seq=0 2024-04-27T00:03:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.011+0000 7f89d3e00640 10 client.25504 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.012+0000 7f89d3e00640 1 -- 172.21.15.40:0/236171784 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562b46206f90 con 0x562b462aabb0 2024-04-27T00:03:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.012+0000 7f89d3e00640 10 client.25504 handle_command_reply: tid=0 2024-04-27T00:03:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.012+0000 7f89fc175740 1 client.25504 shutdown 2024-04-27T00:03:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.012+0000 7f89fc175740 1 -- 172.21.15.40:0/236171784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b462957d0 msgr2=0x562b4629f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.012+0000 7f89fc175740 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b462957d0 0x562b4629f5d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f89e00ce280 tx=0x7f89e00039d0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.012+0000 7f89fc175740 1 -- 172.21.15.40:0/236171784 shutdown_connections 2024-04-27T00:03:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.012+0000 7f89fc175740 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b462aabb0 0x562b462acfa0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.012+0000 7f89fc175740 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b4629a940 0x562b462a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.012+0000 7f89fc175740 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b462957d0 0x562b4629f5d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.012+0000 7f89d3e00640 0 client.25504 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.012+0000 7f89fc175740 1 --2- 172.21.15.40:0/236171784 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b46294de0 0x562b4629f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.012+0000 7f89fc175740 1 -- 172.21.15.40:0/236171784 >> 172.21.15.40:0/236171784 conn(0x562b46290450 msgr2=0x562b4628f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.013+0000 7f89fc175740 1 -- 172.21.15.40:0/236171784 shutdown_connections 2024-04-27T00:03:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.013+0000 7f89fc175740 1 -- 172.21.15.40:0/236171784 wait complete. 2024-04-27T00:03:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.013+0000 7f89fc175740 20 client.25504 trim_cache size 0 max 16384 2024-04-27T00:03:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.044+0000 7f89fc175740 1 Processor -- start 2024-04-27T00:03:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.044+0000 7f89fc175740 1 -- start start 2024-04-27T00:03:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.045+0000 7f89fc175740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 0x562b4628fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.045+0000 7f89fc175740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b462957d0 0x562b462a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.045+0000 7f89fc175740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 0x562b4628b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.045+0000 7f89fc175740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41afdc0 con 0x562b46294de0 2024-04-27T00:03:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.045+0000 7f89fc175740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41ad690 con 0x562b462957d0 2024-04-27T00:03:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.045+0000 7f89fc175740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41ab4e0 con 0x562b4629a940 2024-04-27T00:03:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.045+0000 7f89f1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 0x562b4628fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.045+0000 7f89f1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 0x562b4628fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54418/0 (socket says 172.21.15.40:54418) 2024-04-27T00:03:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.045+0000 7f89f1e00640 1 -- 172.21.15.40:0/2408356555 learned_addr learned my addr 172.21.15.40:0/2408356555 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.046+0000 7f89f1400640 1 --2- 172.21.15.40:0/2408356555 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b462957d0 0x562b462a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.046+0000 7f89f2800640 1 --2- 172.21.15.40:0/2408356555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 0x562b4628b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.046+0000 7f89f2800640 1 -- 172.21.15.40:0/2408356555 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b462957d0 msgr2=0x562b462a65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.046+0000 7f89f2800640 1 --2- 172.21.15.40:0/2408356555 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b462957d0 0x562b462a65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.046+0000 7f89f2800640 1 -- 172.21.15.40:0/2408356555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 msgr2=0x562b4628fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.046+0000 7f89f2800640 1 --2- 172.21.15.40:0/2408356555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 0x562b4628fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.046+0000 7f89f2800640 1 -- 172.21.15.40:0/2408356555 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b45fd7620 con 0x562b4629a940 2024-04-27T00:03:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.046+0000 7f89f1e00640 1 --2- 172.21.15.40:0/2408356555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 0x562b4628fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:03:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.047+0000 7f89f2800640 1 --2- 172.21.15.40:0/2408356555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 0x562b4628b2d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009ee0 tx=0x7f89ec012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2fec43efc14b230 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.047+0000 7f89dea00640 1 -- 172.21.15.40:0/2408356555 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec004b20 con 0x562b4629a940 2024-04-27T00:03:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.047+0000 7f89dea00640 1 -- 172.21.15.40:0/2408356555 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89ec004cc0 con 0x562b4629a940 2024-04-27T00:03:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.047+0000 7f89dea00640 1 -- 172.21.15.40:0/2408356555 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec01b740 con 0x562b4629a940 2024-04-27T00:03:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.048+0000 7f89fc175740 1 -- 172.21.15.40:0/2408356555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 msgr2=0x562b4628b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.048+0000 7f89fc175740 1 --2- 172.21.15.40:0/2408356555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 0x562b4628b2d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009ee0 tx=0x7f89ec012820 comp rx=0 tx=0).stop 2024-04-27T00:03:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.048+0000 7f89fc175740 1 -- 172.21.15.40:0/2408356555 shutdown_connections 2024-04-27T00:03:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.048+0000 7f89fc175740 1 --2- 172.21.15.40:0/2408356555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 0x562b4628b2d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.048+0000 7f89fc175740 1 --2- 172.21.15.40:0/2408356555 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b462957d0 0x562b462a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.048+0000 7f89fc175740 1 --2- 172.21.15.40:0/2408356555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b46294de0 0x562b4628fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.048+0000 7f89fc175740 1 -- 172.21.15.40:0/2408356555 >> 172.21.15.40:0/2408356555 conn(0x562b46290450 msgr2=0x562b462934e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.048+0000 7f89fc175740 1 -- 172.21.15.40:0/2408356555 shutdown_connections 2024-04-27T00:03:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.048+0000 7f89fc175740 1 -- 172.21.15.40:0/2408356555 wait complete. 2024-04-27T00:03:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.049+0000 7f89fc175740 1 -- start start 2024-04-27T00:03:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.049+0000 7f89fc175740 1 Processor -- start 2024-04-27T00:03:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.050+0000 7f89fc175740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b46294de0 0x562b462a4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.050+0000 7f89fc175740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b462957d0 0x562b462a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.050+0000 7f89fc175740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 0x562b462a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.050+0000 7f89fc175740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41abb50 con 0x562b462957d0 2024-04-27T00:03:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.050+0000 7f89fc175740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41acfc0 con 0x562b46294de0 2024-04-27T00:03:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.050+0000 7f89fc175740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41ac220 con 0x562b4629a940 2024-04-27T00:03:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.050+0000 7f89fc175740 10 client.0 _ll_register_callbacks cb 0x7f89e0085f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:03:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.050+0000 7f89f2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 0x562b462a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.050+0000 7f89f2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 0x562b462a56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:42398/0 (socket says 172.21.15.40:42398) 2024-04-27T00:03:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.050+0000 7f89f1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b462957d0 0x562b462a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.050+0000 7f89f2800640 1 -- 172.21.15.40:0/360054257 learned_addr learned my addr 172.21.15.40:0/360054257 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.050+0000 7f89f1e00640 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b46294de0 0x562b462a4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.051+0000 7f89f2800640 1 -- 172.21.15.40:0/360054257 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b46294de0 msgr2=0x562b462a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.051+0000 7f89f2800640 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b46294de0 0x562b462a4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.051+0000 7f89f2800640 1 -- 172.21.15.40:0/360054257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b462957d0 msgr2=0x562b462a47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.051+0000 7f89f2800640 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b462957d0 0x562b462a47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.051+0000 7f89f2800640 1 -- 172.21.15.40:0/360054257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b461b3cb0 con 0x562b4629a940 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.051+0000 7f89f1400640 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b462957d0 0x562b462a47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.051+0000 7f89f2800640 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 0x562b462a56d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f89ec01d040 tx=0x7f89ec004f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=937a8d5c57c5950f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.051+0000 7f89d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.052+0000 7f89d3e00640 1 -- 172.21.15.40:0/360054257 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec029020 con 0x562b4629a940 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.052+0000 7f89d3e00640 1 -- 172.21.15.40:0/360054257 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f89ec0053b0 con 0x562b4629a940 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.052+0000 7f89fc175740 1 -- 172.21.15.40:0/360054257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89ec004cc0 con 0x562b4629a940 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.052+0000 7f89d3e00640 1 -- 172.21.15.40:0/360054257 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f89ec034050 con 0x562b4629a940 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.052+0000 7f89fc175740 10 client.25510 fetch_fsmap learned FSMap version 24 2024-04-27T00:03:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.052+0000 7f89fc175740 1 -- 172.21.15.40:0/360054257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562b463113a0 con 0x562b4629a940 2024-04-27T00:03:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.053+0000 7f89d3e00640 1 -- 172.21.15.40:0/360054257 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89ec0055b0 con 0x562b4629a940 2024-04-27T00:03:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.053+0000 7f89d3e00640 1 -- 172.21.15.40:0/360054257 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f89ec01b480 con 0x562b4629a940 2024-04-27T00:03:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.053+0000 7f89fc175740 10 client.25510 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:03:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.053+0000 7f89fc175740 10 client.25510 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:03:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.053+0000 7f89fc175740 20 client.25510 populate_metadata read hostname 'smithi040' 2024-04-27T00:03:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.053+0000 7f89fc175740 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b46311cf0 0x562b46314110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.053+0000 7f89fc175740 4 client.25510 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:03:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.053+0000 7f89fc175740 1 -- 172.21.15.40:0/360054257 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f89ec004cc0 con 0x562b46311cf0 2024-04-27T00:03:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.054+0000 7f89f1e00640 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b46311cf0 0x562b46314110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.054+0000 7f89f1e00640 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b46311cf0 0x562b46314110 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94608561c82a1812 server_cookie=761091629b312f52 in_seq=0 out_seq=0 2024-04-27T00:03:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.054+0000 7f89d3e00640 10 client.25510 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.060+0000 7f89d3e00640 1 -- 172.21.15.40:0/360054257 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f89ec004cc0 con 0x562b46311cf0 2024-04-27T00:03:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.060+0000 7f89d3e00640 10 client.25510 handle_command_reply: tid=0 2024-04-27T00:03:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.060+0000 7f89fc175740 1 client.25510 shutdown 2024-04-27T00:03:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.060+0000 7f89fc175740 1 -- 172.21.15.40:0/360054257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 msgr2=0x562b462a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.060+0000 7f89fc175740 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 0x562b462a56d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f89ec01d040 tx=0x7f89ec004f60 comp rx=0 tx=0).stop 2024-04-27T00:03:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.060+0000 7f89fc175740 1 -- 172.21.15.40:0/360054257 shutdown_connections 2024-04-27T00:03:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.060+0000 7f89fc175740 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b46311cf0 0x562b46314110 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.060+0000 7f89d3e00640 0 client.25510 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.060+0000 7f89fc175740 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b4629a940 0x562b462a56d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.060+0000 7f89fc175740 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b462957d0 0x562b462a47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.060+0000 7f89fc175740 1 --2- 172.21.15.40:0/360054257 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b46294de0 0x562b462a4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.060+0000 7f89fc175740 1 -- 172.21.15.40:0/360054257 >> 172.21.15.40:0/360054257 conn(0x562b46290450 msgr2=0x562b4628edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.061+0000 7f89fc175740 1 -- 172.21.15.40:0/360054257 shutdown_connections 2024-04-27T00:03:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.061+0000 7f89fc175740 1 -- 172.21.15.40:0/360054257 wait complete. 2024-04-27T00:03:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.061+0000 7f89fc175740 20 client.25510 trim_cache size 0 max 16384 2024-04-27T00:03:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:03:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:03:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:03:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:03:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.062+0000 7f89f3200640 1 -- 172.21.15.40:0/3671665283 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f89d80a3d30 msgr2=0x7f89d80a61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.062+0000 7f89f3200640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f89d80a3d30 0x7f89d80a61f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f89e4031000 tx=0x7f89e402a340 comp rx=0 tx=0).stop 2024-04-27T00:03:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.062+0000 7f89f3200640 1 -- 172.21.15.40:0/3671665283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89f41039e0 msgr2=0x7f89f419b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.062+0000 7f89f3200640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89f41039e0 0x7f89f419b190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f89e0005fd0 tx=0x7f89e000ee50 comp rx=0 tx=0).stop 2024-04-27T00:03:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.062+0000 7f89f3200640 1 -- 172.21.15.40:0/3671665283 shutdown_connections 2024-04-27T00:03:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.062+0000 7f89f3200640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f89f41043a0 0x7f89f41a21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.062+0000 7f89f3200640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f89d80a3d30 0x7f89d80a61f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.062+0000 7f89f3200640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89f41039e0 0x7f89f419b190 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.062+0000 7f89f3200640 1 --2- 172.21.15.40:0/3671665283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89f41027f0 0x7f89f419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.062+0000 7f89f3200640 1 -- 172.21.15.40:0/3671665283 >> 172.21.15.40:0/3671665283 conn(0x7f89f40fdd60 msgr2=0x7f89f4109200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.062+0000 7f89f3200640 1 -- 172.21.15.40:0/3671665283 shutdown_connections 2024-04-27T00:03:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:08.062+0000 7f89f3200640 1 -- 172.21.15.40:0/3671665283 wait complete. 2024-04-27T00:03:08.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:08 smithi040 ceph-mon[28095]: pgmap v455: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:03:08.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:08 smithi080 ceph-mon[32362]: pgmap v455: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:03:08.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:08 smithi103 ceph-mon[32952]: pgmap v455: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:03:10.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:10 smithi040 ceph-mon[28095]: pgmap v456: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-27T00:03:10.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:10 smithi080 ceph-mon[32362]: pgmap v456: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-27T00:03:10.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:10 smithi103 ceph-mon[32952]: pgmap v456: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-27T00:03:12.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:12 smithi040 ceph-mon[28095]: pgmap v457: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-27T00:03:12.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:12 smithi080 ceph-mon[32362]: pgmap v457: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-27T00:03:12.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:12 smithi103 ceph-mon[32952]: pgmap v457: 129 pgs: 129 active+clean; 661 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-27T00:03:14.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:14 smithi040 ceph-mon[28095]: pgmap v458: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:03:14.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:14 smithi080 ceph-mon[32362]: pgmap v458: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:03:14.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:14 smithi103 ceph-mon[32952]: pgmap v458: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-27T00:03:16.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:16 smithi040 ceph-mon[28095]: pgmap v459: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:03:16.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:16 smithi080 ceph-mon[32362]: pgmap v459: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:03:16.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:16 smithi103 ceph-mon[32952]: pgmap v459: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:03:17.946 INFO:tasks.workunit.client.0.smithi040.stderr:Updating files: 58% (7169/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 95% (11719/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-27T00:03:18.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:18 smithi040 ceph-mon[28095]: pgmap v460: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-27T00:03:18.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:18 smithi080 ceph-mon[32362]: pgmap v460: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-27T00:03:18.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:18 smithi103 ceph-mon[32952]: pgmap v460: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-27T00:03:18.999 INFO:tasks.workunit.client.0.smithi040.stderr:Note: switching to 'bc612ffb9c1d5bc27536bc917a705262a36d4387'. 2024-04-27T00:03:18.999 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-27T00:03:18.999 INFO:tasks.workunit.client.0.smithi040.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-27T00:03:18.999 INFO:tasks.workunit.client.0.smithi040.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr:state without impacting any branches by switching back to a branch. 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr: git switch -c 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr:Or undo this operation with: 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr: git switch - 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-27T00:03:19.000 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-04-27T00:03:19.001 INFO:tasks.workunit.client.0.smithi040.stderr:HEAD is now at bc612ffb9c1 Merge PR #55144 into wip-rishabh-testing-20240426.111959 2024-04-27T00:03:19.015 DEBUG:teuthology.orchestra.run.smithi040:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-27T00:03:19.126 INFO:tasks.workunit.client.0.smithi040.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-27T00:03:19.134 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-27T00:03:19.134 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-27T00:03:19.926 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-27T00:03:20.038 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-27T00:03:20.130 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-27T00:03:20.135 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-27T00:03:20.136 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-27T00:03:20.225 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-27T00:03:20.235 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-27T00:03:20.236 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-27T00:03:20.282 INFO:tasks.workunit:Running workunits matching fs/test_o_trunc.sh on client.0... 2024-04-27T00:03:20.284 INFO:tasks.workunit:Running workunit fs/test_o_trunc.sh... 2024-04-27T00:03:20.284 DEBUG:teuthology.orchestra.run.smithi040:workunit test fs/test_o_trunc.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bc612ffb9c1d5bc27536bc917a705262a36d4387 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.sh 2024-04-27T00:03:20.401 INFO:tasks.workunit.client.0.smithi040.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.sh 2024-04-27T00:03:20.403 INFO:tasks.workunit.client.0.smithi040.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs 2024-04-27T00:03:20.403 INFO:tasks.workunit.client.0.smithi040.stderr:+ /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc trunc.foo 600 2024-04-27T00:03:20.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:20 smithi040 ceph-mon[28095]: pgmap v461: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:03:20.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:20 smithi080 ceph-mon[32362]: pgmap v461: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:03:20.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:20 smithi103 ceph-mon[32952]: pgmap v461: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:03:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.210+0000 7f0615e00640 1 -- 172.21.15.40:0/808123617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06101065a0 msgr2=0x7f061010ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.210+0000 7f0615e00640 1 --2- 172.21.15.40:0/808123617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06101065a0 0x7f061010ce30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0608009ca0 tx=0x7f060802a8c0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.210+0000 7f0615e00640 1 -- 172.21.15.40:0/808123617 shutdown_connections 2024-04-27T00:03:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.210+0000 7f0615e00640 1 --2- 172.21.15.40:0/808123617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06101065a0 0x7f061010ce30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.210+0000 7f0615e00640 1 --2- 172.21.15.40:0/808123617 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0610105be0 0x7f0610106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.210+0000 7f0615e00640 1 --2- 172.21.15.40:0/808123617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06101049f0 0x7f0610104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.210+0000 7f0615e00640 1 -- 172.21.15.40:0/808123617 >> 172.21.15.40:0/808123617 conn(0x7f06100fff60 msgr2=0x7f06101023a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.210+0000 7f0615e00640 1 -- 172.21.15.40:0/808123617 shutdown_connections 2024-04-27T00:03:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.210+0000 7f0615e00640 1 -- 172.21.15.40:0/808123617 wait complete. 2024-04-27T00:03:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.211+0000 7f0615e00640 1 Processor -- start 2024-04-27T00:03:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.212+0000 7f0615e00640 1 -- start start 2024-04-27T00:03:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.212+0000 7f0615e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f06101049f0 0x7f061019ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.212+0000 7f0615e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0610105be0 0x7f061019b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.212+0000 7f0615e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06101065a0 0x7f06101a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.212+0000 7f0615e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0610110550 con 0x7f0610105be0 2024-04-27T00:03:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.212+0000 7f0615e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f06101103d0 con 0x7f06101049f0 2024-04-27T00:03:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.212+0000 7f0615e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f06101106d0 con 0x7f06101065a0 2024-04-27T00:03:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.212+0000 7f060fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0610105be0 0x7f061019b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.212+0000 7f060fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0610105be0 0x7f061019b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46246/0 (socket says 172.21.15.40:46246) 2024-04-27T00:03:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.212+0000 7f060fe00640 1 -- 172.21.15.40:0/1827692066 learned_addr learned my addr 172.21.15.40:0/1827692066 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.212+0000 7f0615400640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06101065a0 0x7f06101a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.212+0000 7f0614a00640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f06101049f0 0x7f061019ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.213+0000 7f0615400640 1 -- 172.21.15.40:0/1827692066 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f06101049f0 msgr2=0x7f061019ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:21.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.213+0000 7f0615400640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f06101049f0 0x7f061019ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.213+0000 7f0615400640 1 -- 172.21.15.40:0/1827692066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0610105be0 msgr2=0x7f061019b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:21.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.213+0000 7f0615400640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0610105be0 0x7f061019b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.213+0000 7f0615400640 1 -- 172.21.15.40:0/1827692066 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06101a2830 con 0x7f06101065a0 2024-04-27T00:03:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.213+0000 7f060fe00640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0610105be0 0x7f061019b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:03:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.213+0000 7f0615400640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06101065a0 0x7f06101a2190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f060802aeb0 tx=0x7f06080316f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.213+0000 7f060d600640 1 -- 172.21.15.40:0/1827692066 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0608003b70 con 0x7f06101065a0 2024-04-27T00:03:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.213+0000 7f060d600640 1 -- 172.21.15.40:0/1827692066 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0608003d10 con 0x7f06101065a0 2024-04-27T00:03:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.213+0000 7f0615e00640 1 -- 172.21.15.40:0/1827692066 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06101a2ac0 con 0x7f06101065a0 2024-04-27T00:03:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.213+0000 7f060d600640 1 -- 172.21.15.40:0/1827692066 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060802fae0 con 0x7f06101065a0 2024-04-27T00:03:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.214+0000 7f0615e00640 1 -- 172.21.15.40:0/1827692066 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06101a3060 con 0x7f06101065a0 2024-04-27T00:03:21.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.217+0000 7f060d600640 1 -- 172.21.15.40:0/1827692066 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0608004610 con 0x7f06101065a0 2024-04-27T00:03:21.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.217+0000 7f061a3a0740 1 Processor -- start 2024-04-27T00:03:21.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.217+0000 7f061a3a0740 1 -- start start 2024-04-27T00:03:21.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.217+0000 7f061a3a0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e1d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.217+0000 7f061a3a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e1d7d0 0x558de6e222d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.217+0000 7f061a3a0740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e22940 0x558de6e24dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.217+0000 7f061a3a0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558de6d81380 con 0x558de6e1cde0 2024-04-27T00:03:21.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.217+0000 7f061a3a0740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x558de6d1dc30 con 0x558de6e22940 2024-04-27T00:03:21.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.217+0000 7f061a3a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558de6d7d340 con 0x558de6e1d7d0 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.218+0000 7f060fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e1d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.218+0000 7f060fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e1d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46258/0 (socket says 172.21.15.40:46258) 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.218+0000 7f060fe00640 1 -- 172.21.15.40:0/4080427490 learned_addr learned my addr 172.21.15.40:0/4080427490 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.218+0000 7f0614a00640 1 --2- 172.21.15.40:0/4080427490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e1d7d0 0x558de6e222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.218+0000 7f060fe00640 1 --2- 172.21.15.40:0/4080427490 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e22940 0x558de6e24dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.218+0000 7f0614a00640 1 -- 172.21.15.40:0/4080427490 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e22940 msgr2=0x558de6e24dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.218+0000 7f0614a00640 1 --2- 172.21.15.40:0/4080427490 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e22940 0x558de6e24dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.218+0000 7f0614a00640 1 -- 172.21.15.40:0/4080427490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 msgr2=0x558de6e1d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.218+0000 7f0614a00640 1 --2- 172.21.15.40:0/4080427490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e1d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.218+0000 7f0614a00640 1 -- 172.21.15.40:0/4080427490 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558de6d3f940 con 0x558de6e1d7d0 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.219+0000 7f0614a00640 1 --2- 172.21.15.40:0/4080427490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e1d7d0 0x558de6e222d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0610105a40 tx=0x7f0604011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50f1abab10b57a0e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.219+0000 7f0601c00640 1 -- 172.21.15.40:0/4080427490 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604009d70 con 0x558de6e1d7d0 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.219+0000 7f0601c00640 1 -- 172.21.15.40:0/4080427490 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06040044d0 con 0x558de6e1d7d0 2024-04-27T00:03:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.219+0000 7f0601c00640 1 -- 172.21.15.40:0/4080427490 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604019410 con 0x558de6e1d7d0 2024-04-27T00:03:21.222 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.219+0000 7f060d600640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f05f80a3cb0 0x7f05f80a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.222 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.219+0000 7f060d600640 1 -- 172.21.15.40:0/1827692066 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f06080f1bb0 con 0x7f06101065a0 2024-04-27T00:03:21.222 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.219+0000 7f060fe00640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f05f80a3cb0 0x7f05f80a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.225 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.220+0000 7f061a3a0740 1 -- 172.21.15.40:0/4080427490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e1d7d0 msgr2=0x558de6e222d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:21.225 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.220+0000 7f061a3a0740 1 --2- 172.21.15.40:0/4080427490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e1d7d0 0x558de6e222d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0610105a40 tx=0x7f0604011810 comp rx=0 tx=0).stop 2024-04-27T00:03:21.225 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.220+0000 7f061a3a0740 1 -- 172.21.15.40:0/4080427490 shutdown_connections 2024-04-27T00:03:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.220+0000 7f061a3a0740 1 --2- 172.21.15.40:0/4080427490 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e22940 0x558de6e24dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.220+0000 7f061a3a0740 1 --2- 172.21.15.40:0/4080427490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e1d7d0 0x558de6e222d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.220+0000 7f061a3a0740 1 --2- 172.21.15.40:0/4080427490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e1d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.220+0000 7f061a3a0740 1 -- 172.21.15.40:0/4080427490 >> 172.21.15.40:0/4080427490 conn(0x558de6e18450 msgr2=0x558de6e18850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.220+0000 7f060fe00640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f05f80a3cb0 0x7f05f80a6170 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x558de6e1e030 tx=0x7f05fc009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.220+0000 7f061a3a0740 1 -- 172.21.15.40:0/4080427490 shutdown_connections 2024-04-27T00:03:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.220+0000 7f061a3a0740 1 -- 172.21.15.40:0/4080427490 wait complete. 2024-04-27T00:03:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.220+0000 7f061a3a0740 1 -- start start 2024-04-27T00:03:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.221+0000 7f061a3a0740 1 Processor -- start 2024-04-27T00:03:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.221+0000 7f061a3a0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e26fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.221+0000 7f061a3a0740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1d7d0 0x558de6e27510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.221+0000 7f061a3a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e22940 0x558de6e2e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.221+0000 7f061a3a0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558de6e258f0 con 0x558de6e1cde0 2024-04-27T00:03:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.221+0000 7f061a3a0740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x558de6d08c60 con 0x558de6e1d7d0 2024-04-27T00:03:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.221+0000 7f061a3a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558de6e25a70 con 0x558de6e22940 2024-04-27T00:03:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.221+0000 7f061a3a0740 10 client.0 _ll_register_callbacks cb 0x558de6d34e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:03:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.221+0000 7f0614a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e26fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.221+0000 7f0614a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e26fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46262/0 (socket says 172.21.15.40:46262) 2024-04-27T00:03:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.221+0000 7f0614a00640 1 -- 172.21.15.40:0/4167924085 learned_addr learned my addr 172.21.15.40:0/4167924085 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.222+0000 7f0615400640 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e22940 0x558de6e2e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.222+0000 7f060fe00640 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1d7d0 0x558de6e27510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.222+0000 7f0614a00640 1 -- 172.21.15.40:0/4167924085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e22940 msgr2=0x558de6e2e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.222+0000 7f0614a00640 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e22940 0x558de6e2e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.222+0000 7f0614a00640 1 -- 172.21.15.40:0/4167924085 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1d7d0 msgr2=0x558de6e27510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.222+0000 7f0614a00640 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1d7d0 0x558de6e27510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.222+0000 7f0614a00640 1 -- 172.21.15.40:0/4167924085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558de6d17510 con 0x558de6e1cde0 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.222+0000 7f0615400640 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e22940 0x558de6e2e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.222+0000 7f0614a00640 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e26fd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f060401b040 tx=0x7f0604009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc6a461578abe09 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.222+0000 7f05f6000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.223+0000 7f05f6000640 1 -- 172.21.15.40:0/4167924085 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604028020 con 0x558de6e1cde0 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.223+0000 7f05f6000640 1 -- 172.21.15.40:0/4167924085 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f0604016040 con 0x558de6e1cde0 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.223+0000 7f061a3a0740 1 -- 172.21.15.40:0/4167924085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06040044d0 con 0x558de6e1cde0 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.223+0000 7f05f6000640 1 -- 172.21.15.40:0/4167924085 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0604033050 con 0x558de6e1cde0 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.223+0000 7f061a3a0740 10 client.15378 fetch_fsmap learned FSMap version 24 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.223+0000 7f061a3a0740 1 -- 172.21.15.40:0/4167924085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558de6a93210 con 0x558de6e1cde0 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.223+0000 7f05f6000640 1 -- 172.21.15.40:0/4167924085 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06040044d0 con 0x558de6e1cde0 2024-04-27T00:03:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.224+0000 7f05f6000640 1 -- 172.21.15.40:0/4167924085 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f0604004730 con 0x558de6e1cde0 2024-04-27T00:03:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.224+0000 7f061a3a0740 10 client.15378 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:03:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.224+0000 7f061a3a0740 10 client.15378 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:03:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.224+0000 7f061a3a0740 20 client.15378 populate_metadata read hostname 'smithi040' 2024-04-27T00:03:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.224+0000 7f061a3a0740 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558de6e328d0 0x558de6e34d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.224+0000 7f061a3a0740 4 client.15378 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:03:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.224+0000 7f061a3a0740 1 -- 172.21.15.40:0/4167924085 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558de6d8ef90 con 0x558de6e328d0 2024-04-27T00:03:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.224+0000 7f060fe00640 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558de6e328d0 0x558de6e34d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.225+0000 7f060fe00640 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558de6e328d0 0x558de6e34d00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50464d1b2dc873be server_cookie=5e23f2c49ba0631c in_seq=0 out_seq=0 2024-04-27T00:03:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.225+0000 7f05f6000640 10 client.15378 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f05f6000640 1 -- 172.21.15.40:0/4167924085 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558de6d8ef90 con 0x558de6e328d0 2024-04-27T00:03:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f05f6000640 10 client.15378 handle_command_reply: tid=0 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f061a3a0740 1 client.15378 shutdown 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f061a3a0740 1 -- 172.21.15.40:0/4167924085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 msgr2=0x558de6e26fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f061a3a0740 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e26fd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f060401b040 tx=0x7f0604009fa0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f061a3a0740 1 -- 172.21.15.40:0/4167924085 shutdown_connections 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f061a3a0740 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558de6e328d0 0x558de6e34d00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f061a3a0740 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e22940 0x558de6e2e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f061a3a0740 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1d7d0 0x558de6e27510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f061a3a0740 1 --2- 172.21.15.40:0/4167924085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e26fd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f05f6000640 0 client.15378 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f061a3a0740 1 -- 172.21.15.40:0/4167924085 >> 172.21.15.40:0/4167924085 conn(0x558de6e18450 msgr2=0x558de6e17650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f061a3a0740 1 -- 172.21.15.40:0/4167924085 shutdown_connections 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f061a3a0740 1 -- 172.21.15.40:0/4167924085 wait complete. 2024-04-27T00:03:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.226+0000 7f061a3a0740 20 client.15378 trim_cache size 0 max 16384 2024-04-27T00:03:21.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f061a3a0740 1 Processor -- start 2024-04-27T00:03:21.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f061a3a0740 1 -- start start 2024-04-27T00:03:21.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f061a3a0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e1b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f061a3a0740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1d7d0 0x558de6e2e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f061a3a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e22940 0x558de6e265f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f061a3a0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f06101afd80 con 0x558de6e1cde0 2024-04-27T00:03:21.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f061a3a0740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f06101ad650 con 0x558de6e1d7d0 2024-04-27T00:03:21.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f061a3a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f06101ab4a0 con 0x558de6e22940 2024-04-27T00:03:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f0614a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e1b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f0614a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e1b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46276/0 (socket says 172.21.15.40:46276) 2024-04-27T00:03:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f0614a00640 1 -- 172.21.15.40:0/3858564750 learned_addr learned my addr 172.21.15.40:0/3858564750 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f0615400640 1 --2- 172.21.15.40:0/3858564750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e22940 0x558de6e265f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.243+0000 7f060fe00640 1 --2- 172.21.15.40:0/3858564750 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1d7d0 0x558de6e2e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.244+0000 7f0614a00640 1 -- 172.21.15.40:0/3858564750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e22940 msgr2=0x558de6e265f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.244+0000 7f0614a00640 1 --2- 172.21.15.40:0/3858564750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e22940 0x558de6e265f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.244+0000 7f0614a00640 1 -- 172.21.15.40:0/3858564750 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1d7d0 msgr2=0x558de6e2e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.244+0000 7f0614a00640 1 --2- 172.21.15.40:0/3858564750 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1d7d0 0x558de6e2e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.244+0000 7f0614a00640 1 -- 172.21.15.40:0/3858564750 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558de6b5f620 con 0x558de6e1cde0 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.244+0000 7f0614a00640 1 --2- 172.21.15.40:0/3858564750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e1b8e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0604004b50 tx=0x7f0604003ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16fef20c1b8e74b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.244+0000 7f0601200640 1 -- 172.21.15.40:0/3858564750 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604028040 con 0x558de6e1cde0 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.244+0000 7f0601200640 1 -- 172.21.15.40:0/3858564750 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0604005700 con 0x558de6e1cde0 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.244+0000 7f0601200640 1 -- 172.21.15.40:0/3858564750 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604023db0 con 0x558de6e1cde0 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.245+0000 7f061a3a0740 1 -- 172.21.15.40:0/3858564750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 msgr2=0x558de6e1b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.245+0000 7f061a3a0740 1 --2- 172.21.15.40:0/3858564750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e1b8e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0604004b50 tx=0x7f0604003ac0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.245+0000 7f061a3a0740 1 -- 172.21.15.40:0/3858564750 shutdown_connections 2024-04-27T00:03:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.245+0000 7f061a3a0740 1 --2- 172.21.15.40:0/3858564750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e22940 0x558de6e265f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.245+0000 7f061a3a0740 1 --2- 172.21.15.40:0/3858564750 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1d7d0 0x558de6e2e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.245+0000 7f061a3a0740 1 --2- 172.21.15.40:0/3858564750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e1cde0 0x558de6e1b8e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.245+0000 7f061a3a0740 1 -- 172.21.15.40:0/3858564750 >> 172.21.15.40:0/3858564750 conn(0x558de6e18450 msgr2=0x558de6e1b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.245+0000 7f061a3a0740 1 -- 172.21.15.40:0/3858564750 shutdown_connections 2024-04-27T00:03:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.245+0000 7f061a3a0740 1 -- 172.21.15.40:0/3858564750 wait complete. 2024-04-27T00:03:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.245+0000 7f061a3a0740 1 -- start start 2024-04-27T00:03:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.246+0000 7f061a3a0740 1 Processor -- start 2024-04-27T00:03:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.246+0000 7f061a3a0740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1cde0 0x558de6e2b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.246+0000 7f061a3a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e1d7d0 0x558de6e2bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.246+0000 7f061a3a0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e22940 0x558de6e2cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.246+0000 7f061a3a0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f06101abb10 con 0x558de6e22940 2024-04-27T00:03:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.246+0000 7f061a3a0740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f06101acf80 con 0x558de6e1cde0 2024-04-27T00:03:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.246+0000 7f061a3a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f06101ac1e0 con 0x558de6e1d7d0 2024-04-27T00:03:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.246+0000 7f061a3a0740 10 client.0 _ll_register_callbacks cb 0x7f05fc013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.246+0000 7f0615400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e22940 0x558de6e2cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.246+0000 7f0615400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e22940 0x558de6e2cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46286/0 (socket says 172.21.15.40:46286) 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.246+0000 7f0615400640 1 -- 172.21.15.40:0/2242966537 learned_addr learned my addr 172.21.15.40:0/2242966537 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.246+0000 7f0614a00640 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1cde0 0x558de6e2b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.247+0000 7f060fe00640 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e1d7d0 0x558de6e2bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.247+0000 7f0615400640 1 -- 172.21.15.40:0/2242966537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e1d7d0 msgr2=0x558de6e2bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.247+0000 7f0615400640 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e1d7d0 0x558de6e2bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.247+0000 7f0615400640 1 -- 172.21.15.40:0/2242966537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1cde0 msgr2=0x558de6e2b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.247+0000 7f0615400640 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1cde0 0x558de6e2b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.247+0000 7f0615400640 1 -- 172.21.15.40:0/2242966537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558de6d3bcb0 con 0x558de6e22940 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.247+0000 7f0614a00640 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1cde0 0x558de6e2b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.247+0000 7f0615400640 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e22940 0x558de6e2cc20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0608005e30 tx=0x7f06080bd940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db0e3e2f9c16580 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.247+0000 7f05f6000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.247+0000 7f05f6000640 1 -- 172.21.15.40:0/2242966537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060802fe80 con 0x558de6e22940 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.247+0000 7f05f6000640 1 -- 172.21.15.40:0/2242966537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f06080bdca0 con 0x558de6e22940 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.247+0000 7f061a3a0740 1 -- 172.21.15.40:0/2242966537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0604005700 con 0x558de6e22940 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.248+0000 7f05f6000640 1 -- 172.21.15.40:0/2242966537 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f06080a4050 con 0x558de6e22940 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.248+0000 7f061a3a0740 10 client.15384 fetch_fsmap learned FSMap version 24 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.248+0000 7f061a3a0740 1 -- 172.21.15.40:0/2242966537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558de6e2e420 con 0x558de6e22940 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.248+0000 7f05f6000640 1 -- 172.21.15.40:0/2242966537 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0604005700 con 0x558de6e22940 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.248+0000 7f05f6000640 1 -- 172.21.15.40:0/2242966537 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f06080c6220 con 0x558de6e22940 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.248+0000 7f061a3a0740 10 client.15384 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.248+0000 7f061a3a0740 10 client.15384 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.248+0000 7f061a3a0740 20 client.15384 populate_metadata read hostname 'smithi040' 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.248+0000 7f061a3a0740 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558de6e991f0 0x558de6e9b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.248+0000 7f061a3a0740 4 client.15384 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:03:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.248+0000 7f061a3a0740 1 -- 172.21.15.40:0/2242966537 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558de6d88e00 con 0x558de6e991f0 2024-04-27T00:03:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.249+0000 7f0614a00640 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558de6e991f0 0x558de6e9b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.249+0000 7f0614a00640 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558de6e991f0 0x558de6e9b610 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a09fbef32ce63d31 server_cookie=70c33c1c8034350 in_seq=0 out_seq=0 2024-04-27T00:03:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.249+0000 7f05f6000640 10 client.15384 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.261+0000 7f05f6000640 1 -- 172.21.15.40:0/2242966537 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558de6d88e00 con 0x558de6e991f0 2024-04-27T00:03:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.261+0000 7f05f6000640 10 client.15384 handle_command_reply: tid=0 2024-04-27T00:03:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.261+0000 7f061a3a0740 1 client.15384 shutdown 2024-04-27T00:03:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.262+0000 7f061a3a0740 1 -- 172.21.15.40:0/2242966537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e22940 msgr2=0x558de6e2cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.262+0000 7f061a3a0740 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e22940 0x558de6e2cc20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0608005e30 tx=0x7f06080bd940 comp rx=0 tx=0).stop 2024-04-27T00:03:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.262+0000 7f061a3a0740 1 -- 172.21.15.40:0/2242966537 shutdown_connections 2024-04-27T00:03:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.262+0000 7f061a3a0740 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558de6e991f0 0x558de6e9b610 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.262+0000 7f061a3a0740 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558de6e22940 0x558de6e2cc20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.262+0000 7f061a3a0740 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558de6e1d7d0 0x558de6e2bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.262+0000 7f061a3a0740 1 --2- 172.21.15.40:0/2242966537 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558de6e1cde0 0x558de6e2b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.262+0000 7f061a3a0740 1 -- 172.21.15.40:0/2242966537 >> 172.21.15.40:0/2242966537 conn(0x558de6e18450 msgr2=0x558de6e1ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.262+0000 7f05f6000640 0 client.15384 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.262+0000 7f061a3a0740 1 -- 172.21.15.40:0/2242966537 shutdown_connections 2024-04-27T00:03:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.262+0000 7f061a3a0740 1 -- 172.21.15.40:0/2242966537 wait complete. 2024-04-27T00:03:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.262+0000 7f061a3a0740 20 client.15384 trim_cache size 0 max 16384 2024-04-27T00:03:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:03:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:03:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:03:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:03:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.263+0000 7f0615e00640 1 -- 172.21.15.40:0/1827692066 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f05f80a3cb0 msgr2=0x7f05f80a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.263+0000 7f0615e00640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f05f80a3cb0 0x7f05f80a6170 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x558de6e1e030 tx=0x7f05fc009550 comp rx=0 tx=0).stop 2024-04-27T00:03:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.263+0000 7f0615e00640 1 -- 172.21.15.40:0/1827692066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06101065a0 msgr2=0x7f06101a2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.263+0000 7f0615e00640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06101065a0 0x7f06101a2190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f060802aeb0 tx=0x7f06080316f0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.263+0000 7f0615e00640 1 -- 172.21.15.40:0/1827692066 shutdown_connections 2024-04-27T00:03:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.263+0000 7f0615e00640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f05f80a3cb0 0x7f05f80a6170 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.263+0000 7f0615e00640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06101065a0 0x7f06101a2190 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.263+0000 7f0615e00640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0610105be0 0x7f061019b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.263+0000 7f0615e00640 1 --2- 172.21.15.40:0/1827692066 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f06101049f0 0x7f061019ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.263+0000 7f0615e00640 1 -- 172.21.15.40:0/1827692066 >> 172.21.15.40:0/1827692066 conn(0x7f06100fff60 msgr2=0x7f0610108e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.263+0000 7f0615e00640 1 -- 172.21.15.40:0/1827692066 shutdown_connections 2024-04-27T00:03:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:21.263+0000 7f0615e00640 1 -- 172.21.15.40:0/1827692066 wait complete. 2024-04-27T00:03:22.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:22 smithi040 ceph-mon[28095]: pgmap v462: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:03:22.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:22 smithi080 ceph-mon[32362]: pgmap v462: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:03:22.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:22 smithi103 ceph-mon[32952]: pgmap v462: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:03:24.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:24 smithi040 ceph-mon[28095]: pgmap v463: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-27T00:03:24.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:24 smithi080 ceph-mon[32362]: pgmap v463: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-27T00:03:24.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:24 smithi103 ceph-mon[32952]: pgmap v463: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-04-27T00:03:26.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:26 smithi040 ceph-mon[28095]: pgmap v464: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-27T00:03:26.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:26 smithi080 ceph-mon[32362]: pgmap v464: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-27T00:03:26.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:26 smithi103 ceph-mon[32952]: pgmap v464: 129 pgs: 129 active+clean; 662 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-27T00:03:28.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:28 smithi080 ceph-mon[32362]: pgmap v465: 129 pgs: 129 active+clean; 729 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.7 KiB/s wr, 12 op/s 2024-04-27T00:03:28.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:28 smithi103 ceph-mon[32952]: pgmap v465: 129 pgs: 129 active+clean; 729 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.7 KiB/s wr, 12 op/s 2024-04-27T00:03:28.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:28 smithi040 ceph-mon[28095]: pgmap v465: 129 pgs: 129 active+clean; 729 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.7 KiB/s wr, 12 op/s 2024-04-27T00:03:30.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:30 smithi080 ceph-mon[32362]: pgmap v466: 129 pgs: 129 active+clean; 729 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.7 KiB/s wr, 10 op/s 2024-04-27T00:03:30.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:30 smithi103 ceph-mon[32952]: pgmap v466: 129 pgs: 129 active+clean; 729 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.7 KiB/s wr, 10 op/s 2024-04-27T00:03:30.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:30 smithi040 ceph-mon[28095]: pgmap v466: 129 pgs: 129 active+clean; 729 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.7 KiB/s wr, 10 op/s 2024-04-27T00:03:32.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:32 smithi080 ceph-mon[32362]: pgmap v467: 129 pgs: 129 active+clean; 729 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.7 KiB/s wr, 10 op/s 2024-04-27T00:03:32.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:32 smithi103 ceph-mon[32952]: pgmap v467: 129 pgs: 129 active+clean; 729 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.7 KiB/s wr, 10 op/s 2024-04-27T00:03:32.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:32 smithi040 ceph-mon[28095]: pgmap v467: 129 pgs: 129 active+clean; 729 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.7 KiB/s wr, 10 op/s 2024-04-27T00:03:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.411+0000 7f4a19200640 1 -- 172.21.15.40:0/30186883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14101b60 msgr2=0x7f4a14101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.411+0000 7f4a19200640 1 --2- 172.21.15.40:0/30186883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14101b60 0x7f4a14101fc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009900 tx=0x7f4a0002a3f0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.411+0000 7f4a19200640 1 -- 172.21.15.40:0/30186883 shutdown_connections 2024-04-27T00:03:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.411+0000 7f4a19200640 1 --2- 172.21.15.40:0/30186883 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4a14102500 0x7f4a14102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.411+0000 7f4a19200640 1 --2- 172.21.15.40:0/30186883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14101b60 0x7f4a14101fc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.411+0000 7f4a19200640 1 --2- 172.21.15.40:0/30186883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a14101200 0x7f4a14101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.412+0000 7f4a19200640 1 -- 172.21.15.40:0/30186883 >> 172.21.15.40:0/30186883 conn(0x7f4a140fce00 msgr2=0x7f4a140ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.412+0000 7f4a19200640 1 -- 172.21.15.40:0/30186883 shutdown_connections 2024-04-27T00:03:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.412+0000 7f4a19200640 1 -- 172.21.15.40:0/30186883 wait complete. 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.413+0000 7f4a19200640 1 Processor -- start 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.413+0000 7f4a19200640 1 -- start start 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.413+0000 7f4a19200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14101200 0x7f4a1419acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.413+0000 7f4a19200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4a14101b60 0x7f4a1419b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a19200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a14102500 0x7f4a141a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a19200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141106d0 con 0x7f4a14101200 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a19200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f4a14110550 con 0x7f4a14101b60 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a19200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4a14110850 con 0x7f4a14102500 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a11e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14101200 0x7f4a1419acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a11400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4a14101b60 0x7f4a1419b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a11e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14101200 0x7f4a1419acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55848/0 (socket says 172.21.15.40:55848) 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a11e00640 1 -- 172.21.15.40:0/4030406465 learned_addr learned my addr 172.21.15.40:0/4030406465 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a12800640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a14102500 0x7f4a141a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a11e00640 1 -- 172.21.15.40:0/4030406465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a14102500 msgr2=0x7f4a141a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:34.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a11e00640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a14102500 0x7f4a141a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a11e00640 1 -- 172.21.15.40:0/4030406465 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4a14101b60 msgr2=0x7f4a1419b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:34.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a11e00640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4a14101b60 0x7f4a1419b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.414+0000 7f4a11e00640 1 -- 172.21.15.40:0/4030406465 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a141a2900 con 0x7f4a14101200 2024-04-27T00:03:34.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.415+0000 7f4a11e00640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14101200 0x7f4a1419acc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400ea60 tx=0x7f4a0400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.415+0000 7f4a0aa00640 1 -- 172.21.15.40:0/4030406465 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0400cde0 con 0x7f4a14101200 2024-04-27T00:03:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.415+0000 7f4a0aa00640 1 -- 172.21.15.40:0/4030406465 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a040035f0 con 0x7f4a14101200 2024-04-27T00:03:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.415+0000 7f4a19200640 1 -- 172.21.15.40:0/4030406465 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a141a2bf0 con 0x7f4a14101200 2024-04-27T00:03:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.415+0000 7f4a0aa00640 1 -- 172.21.15.40:0/4030406465 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a04010690 con 0x7f4a14101200 2024-04-27T00:03:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.415+0000 7f4a19200640 1 -- 172.21.15.40:0/4030406465 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a141a3160 con 0x7f4a14101200 2024-04-27T00:03:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.420+0000 7f4a1bfbc740 1 Processor -- start 2024-04-27T00:03:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.420+0000 7f4a1bfbc740 1 -- start start 2024-04-27T00:03:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.420+0000 7f4a1bfbc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b1969bd0 0x5604b1969ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.420+0000 7f4a0aa00640 1 -- 172.21.15.40:0/4030406465 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4a040037a0 con 0x7f4a14101200 2024-04-27T00:03:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.420+0000 7f4a1bfbc740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b196a5c0 0x5604b196f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.420+0000 7f4a1bfbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 0x5604b1971bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.420+0000 7f4a1bfbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5604b18ce380 con 0x5604b196f730 2024-04-27T00:03:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.420+0000 7f4a1bfbc740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5604b186ac30 con 0x5604b196a5c0 2024-04-27T00:03:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.420+0000 7f4a1bfbc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5604b18ca340 con 0x5604b1969bd0 2024-04-27T00:03:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.421+0000 7f4a11400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 0x5604b1971bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.421+0000 7f4a11400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 0x5604b1971bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55856/0 (socket says 172.21.15.40:55856) 2024-04-27T00:03:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.421+0000 7f4a12800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b1969bd0 0x5604b1969ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.421+0000 7f4a11400640 1 -- 172.21.15.40:0/4041977379 learned_addr learned my addr 172.21.15.40:0/4041977379 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:34.423 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.421+0000 7f4a11e00640 1 --2- 172.21.15.40:0/4041977379 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b196a5c0 0x5604b196f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.421+0000 7f4a12800640 1 -- 172.21.15.40:0/4041977379 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b196a5c0 msgr2=0x5604b196f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.421+0000 7f4a12800640 1 --2- 172.21.15.40:0/4041977379 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b196a5c0 0x5604b196f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.421+0000 7f4a12800640 1 -- 172.21.15.40:0/4041977379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 msgr2=0x5604b1971bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.421+0000 7f4a12800640 1 --2- 172.21.15.40:0/4041977379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 0x5604b1971bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.421+0000 7f4a12800640 1 -- 172.21.15.40:0/4041977379 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604b188c940 con 0x5604b1969bd0 2024-04-27T00:03:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.421+0000 7f4a11400640 1 --2- 172.21.15.40:0/4041977379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 0x5604b1971bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:03:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.422+0000 7f4a12800640 1 --2- 172.21.15.40:0/4041977379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b1969bd0 0x5604b1969ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009f00 tx=0x7f4a0c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a0b190ed508581d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.422+0000 7f49ff400640 1 -- 172.21.15.40:0/4041977379 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c004d20 con 0x5604b1969bd0 2024-04-27T00:03:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.422+0000 7f49ff400640 1 -- 172.21.15.40:0/4041977379 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a0c004030 con 0x5604b1969bd0 2024-04-27T00:03:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.422+0000 7f49ff400640 1 -- 172.21.15.40:0/4041977379 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c005280 con 0x5604b1969bd0 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.422+0000 7f4a0aa00640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f49f80a3d70 0x7f49f80a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.422+0000 7f4a11e00640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f49f80a3d70 0x7f49f80a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.422+0000 7f4a0aa00640 1 -- 172.21.15.40:0/4030406465 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f4a040d0bd0 con 0x7f4a14101200 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.423+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/4041977379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b1969bd0 msgr2=0x5604b1969ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.423+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/4041977379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b1969bd0 0x5604b1969ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009f00 tx=0x7f4a0c011810 comp rx=0 tx=0).stop 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.423+0000 7f4a11e00640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f49f80a3d70 0x7f49f80a6230 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x5604b196a9c0 tx=0x7f4a040d4680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.423+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/4041977379 shutdown_connections 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.423+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/4041977379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 0x5604b1971bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.423+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/4041977379 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b196a5c0 0x5604b196f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.423+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/4041977379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b1969bd0 0x5604b1969ff0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.423+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/4041977379 >> 172.21.15.40:0/4041977379 conn(0x5604b1965240 msgr2=0x5604b1965640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.423+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/4041977379 shutdown_connections 2024-04-27T00:03:34.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.423+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/4041977379 wait complete. 2024-04-27T00:03:34.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.424+0000 7f4a1bfbc740 1 -- start start 2024-04-27T00:03:34.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.424+0000 7f4a1bfbc740 1 Processor -- start 2024-04-27T00:03:34.430 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.424+0000 7f4a1bfbc740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1973ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.424+0000 7f4a1bfbc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196a5c0 0x5604b1974410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.424+0000 7f4a1bfbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 0x5604b197b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.424+0000 7f4a1bfbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5604b19726e0 con 0x5604b196f730 2024-04-27T00:03:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.424+0000 7f4a1bfbc740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5604b1855c60 con 0x5604b1969bd0 2024-04-27T00:03:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.424+0000 7f4a1bfbc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5604b1972860 con 0x5604b196a5c0 2024-04-27T00:03:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.424+0000 7f4a1bfbc740 10 client.0 _ll_register_callbacks cb 0x5604b1881e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:03:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.425+0000 7f4a12800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1973ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.425+0000 7f4a12800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 0x5604b197b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.425+0000 7f4a12800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 0x5604b197b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55870/0 (socket says 172.21.15.40:55870) 2024-04-27T00:03:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.425+0000 7f4a12800640 1 -- 172.21.15.40:0/972106558 learned_addr learned my addr 172.21.15.40:0/972106558 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:34.431 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.425+0000 7f4a11400640 1 --2- 172.21.15.40:0/972106558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196a5c0 0x5604b1974410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.425+0000 7f4a12800640 1 -- 172.21.15.40:0/972106558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196a5c0 msgr2=0x5604b1974410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.425+0000 7f4a12800640 1 --2- 172.21.15.40:0/972106558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196a5c0 0x5604b1974410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.425+0000 7f4a12800640 1 -- 172.21.15.40:0/972106558 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 msgr2=0x5604b1973ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.425+0000 7f4a12800640 1 --2- 172.21.15.40:0/972106558 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1973ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.425+0000 7f4a12800640 1 -- 172.21.15.40:0/972106558 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604b1864510 con 0x5604b196f730 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.425+0000 7f4a12800640 1 --2- 172.21.15.40:0/972106558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 0x5604b197b3e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c011d10 tx=0x7f4a0c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c73cf9dfccfd43 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.425+0000 7f49f3e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.426+0000 7f49f3e00640 1 -- 172.21.15.40:0/972106558 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c01e040 con 0x5604b196f730 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.426+0000 7f49f3e00640 1 -- 172.21.15.40:0/972106558 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f4a0c004a40 con 0x5604b196f730 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.426+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/972106558 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a0c004030 con 0x5604b196f730 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.426+0000 7f49f3e00640 1 -- 172.21.15.40:0/972106558 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4a0c004c40 con 0x5604b196f730 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.426+0000 7f4a1bfbc740 10 client.15396 fetch_fsmap learned FSMap version 24 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.426+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/972106558 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5604b15e0210 con 0x5604b196f730 2024-04-27T00:03:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.426+0000 7f49f3e00640 1 -- 172.21.15.40:0/972106558 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a0c004030 con 0x5604b196f730 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.427+0000 7f49f3e00640 1 -- 172.21.15.40:0/972106558 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4a0c024070 con 0x5604b196f730 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.427+0000 7f4a1bfbc740 10 client.15396 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.427+0000 7f4a1bfbc740 10 client.15396 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.427+0000 7f4a1bfbc740 20 client.15396 populate_metadata read hostname 'smithi040' 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.427+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/972106558 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5604b197f6b0 0x5604b1981aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.427+0000 7f4a1bfbc740 4 client.15396 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.427+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/972106558 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604b18dbf90 con 0x5604b197f6b0 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.427+0000 7f4a11400640 1 --2- 172.21.15.40:0/972106558 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5604b197f6b0 0x5604b1981aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.428+0000 7f4a11400640 1 --2- 172.21.15.40:0/972106558 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5604b197f6b0 0x5604b1981aa0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76562aaa4cbf566a server_cookie=311f7b384f283d85 in_seq=0 out_seq=0 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.428+0000 7f49f3e00640 10 client.15396 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f49f3e00640 1 -- 172.21.15.40:0/972106558 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5604b18dbf90 con 0x5604b197f6b0 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f49f3e00640 10 client.15396 handle_command_reply: tid=0 2024-04-27T00:03:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f4a1bfbc740 1 client.15396 shutdown 2024-04-27T00:03:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/972106558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 msgr2=0x5604b197b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/972106558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 0x5604b197b3e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c011d10 tx=0x7f4a0c000f40 comp rx=0 tx=0).stop 2024-04-27T00:03:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/972106558 shutdown_connections 2024-04-27T00:03:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/972106558 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5604b197f6b0 0x5604b1981aa0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/972106558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196f730 0x5604b197b3e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/972106558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196a5c0 0x5604b1974410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/972106558 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1973ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f49f3e00640 0 client.15396 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/972106558 >> 172.21.15.40:0/972106558 conn(0x5604b1965240 msgr2=0x5604b1964440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/972106558 shutdown_connections 2024-04-27T00:03:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/972106558 wait complete. 2024-04-27T00:03:34.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.429+0000 7f4a1bfbc740 20 client.15396 trim_cache size 0 max 16384 2024-04-27T00:03:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.446+0000 7f4a1bfbc740 1 Processor -- start 2024-04-27T00:03:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.446+0000 7f4a1bfbc740 1 -- start start 2024-04-27T00:03:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.447+0000 7f4a1bfbc740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1964bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.447+0000 7f4a1bfbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196a5c0 0x5604b197b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.447+0000 7f4a1bfbc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196f730 0x5604b1968900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.447+0000 7f4a1bfbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141aff50 con 0x5604b196a5c0 2024-04-27T00:03:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.447+0000 7f4a1bfbc740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141ad820 con 0x5604b1969bd0 2024-04-27T00:03:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.447+0000 7f4a1bfbc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141ab670 con 0x5604b196f730 2024-04-27T00:03:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.447+0000 7f4a11400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196a5c0 0x5604b197b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.447+0000 7f4a12800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1964bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.447+0000 7f4a12800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1964bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:42142/0 (socket says 172.21.15.40:42142) 2024-04-27T00:03:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.447+0000 7f4a11400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196a5c0 0x5604b197b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55882/0 (socket says 172.21.15.40:55882) 2024-04-27T00:03:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.447+0000 7f4a12800640 1 -- 172.21.15.40:0/3551586230 learned_addr learned my addr 172.21.15.40:0/3551586230 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.447+0000 7f4a12800640 1 --2- 172.21.15.40:0/3551586230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196f730 0x5604b1968900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.448+0000 7f4a11400640 1 -- 172.21.15.40:0/3551586230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196f730 msgr2=0x5604b1968900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.448+0000 7f4a11400640 1 --2- 172.21.15.40:0/3551586230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196f730 0x5604b1968900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.448+0000 7f4a11400640 1 -- 172.21.15.40:0/3551586230 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 msgr2=0x5604b1964bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.448+0000 7f4a11400640 1 --2- 172.21.15.40:0/3551586230 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1964bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.448+0000 7f4a11400640 1 -- 172.21.15.40:0/3551586230 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604b16ac620 con 0x5604b196a5c0 2024-04-27T00:03:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.448+0000 7f4a12800640 1 --2- 172.21.15.40:0/3551586230 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1964bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:03:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.448+0000 7f4a11400640 1 --2- 172.21.15.40:0/3551586230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196a5c0 0x5604b197b3a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4a00000c00 tx=0x7f4a00004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1510b5c5a5099a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.448+0000 7f49fea00640 1 -- 172.21.15.40:0/3551586230 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a00035040 con 0x5604b196a5c0 2024-04-27T00:03:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.448+0000 7f49fea00640 1 -- 172.21.15.40:0/3551586230 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a000315a0 con 0x5604b196a5c0 2024-04-27T00:03:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.448+0000 7f49fea00640 1 -- 172.21.15.40:0/3551586230 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a00003850 con 0x5604b196a5c0 2024-04-27T00:03:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.449+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/3551586230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196a5c0 msgr2=0x5604b197b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.449+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/3551586230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196a5c0 0x5604b197b3a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4a00000c00 tx=0x7f4a00004530 comp rx=0 tx=0).stop 2024-04-27T00:03:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.449+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/3551586230 shutdown_connections 2024-04-27T00:03:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.449+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/3551586230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196f730 0x5604b1968900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.449+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/3551586230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196a5c0 0x5604b197b3a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.449+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/3551586230 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1964bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.449+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/3551586230 >> 172.21.15.40:0/3551586230 conn(0x5604b1965240 msgr2=0x5604b19681f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.449+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/3551586230 shutdown_connections 2024-04-27T00:03:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.449+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/3551586230 wait complete. 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.449+0000 7f4a1bfbc740 1 -- start start 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.450+0000 7f4a1bfbc740 1 Processor -- start 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.450+0000 7f4a1bfbc740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1978570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.450+0000 7f4a1bfbc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196a5c0 0x5604b1978ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.450+0000 7f4a1bfbc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196f730 0x5604b1979ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.450+0000 7f4a1bfbc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141abce0 con 0x5604b196a5c0 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.450+0000 7f4a1bfbc740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141ad150 con 0x5604b1969bd0 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.450+0000 7f4a1bfbc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4a141ac3b0 con 0x5604b196f730 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.450+0000 7f4a1bfbc740 10 client.0 _ll_register_callbacks cb 0x7f4a04005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.450+0000 7f4a12800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1978570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.450+0000 7f4a12800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1978570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:42150/0 (socket says 172.21.15.40:42150) 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.450+0000 7f4a12800640 1 -- 172.21.15.40:0/1192561065 learned_addr learned my addr 172.21.15.40:0/1192561065 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.450+0000 7f4a11400640 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196a5c0 0x5604b1978ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.451+0000 7f4a12800640 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196f730 0x5604b1979ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.451+0000 7f4a12800640 1 -- 172.21.15.40:0/1192561065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196f730 msgr2=0x5604b1979ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.451+0000 7f4a12800640 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196f730 0x5604b1979ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.451+0000 7f4a12800640 1 -- 172.21.15.40:0/1192561065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196a5c0 msgr2=0x5604b1978ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.451+0000 7f4a12800640 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196a5c0 0x5604b1978ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.451+0000 7f4a12800640 1 -- 172.21.15.40:0/1192561065 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604b1888cb0 con 0x5604b1969bd0 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.451+0000 7f4a12800640 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1978570 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00f8d0 tx=0x7f4a0c005cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b321289fbf75e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.451+0000 7f49f3e00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.451+0000 7f49f3e00640 1 -- 172.21.15.40:0/1192561065 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c00fb40 con 0x5604b1969bd0 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.451+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/1192561065 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a000315a0 con 0x5604b1969bd0 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.451+0000 7f49f3e00640 1 -- 172.21.15.40:0/1192561065 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f4a0c00fce0 con 0x5604b1969bd0 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.452+0000 7f49f3e00640 1 -- 172.21.15.40:0/1192561065 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4a0c047050 con 0x5604b1969bd0 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.452+0000 7f4a1bfbc740 10 client.25520 fetch_fsmap learned FSMap version 24 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.452+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/1192561065 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5604b197b1f0 con 0x5604b1969bd0 2024-04-27T00:03:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.452+0000 7f49f3e00640 1 -- 172.21.15.40:0/1192561065 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a000315a0 con 0x5604b1969bd0 2024-04-27T00:03:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.452+0000 7f49f3e00640 1 -- 172.21.15.40:0/1192561065 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f4a0c024070 con 0x5604b1969bd0 2024-04-27T00:03:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.452+0000 7f4a1bfbc740 10 client.25520 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:03:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.452+0000 7f4a1bfbc740 10 client.25520 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:03:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.453+0000 7f4a1bfbc740 20 client.25520 populate_metadata read hostname 'smithi040' 2024-04-27T00:03:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.453+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5604b19ee590 0x5604b19f0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.453+0000 7f4a1bfbc740 4 client.25520 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:03:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.453+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/1192561065 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5604b18d5e00 con 0x5604b19ee590 2024-04-27T00:03:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.453+0000 7f4a11400640 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5604b19ee590 0x5604b19f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.453+0000 7f4a11400640 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5604b19ee590 0x5604b19f0980 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27a7010a6bcf5882 server_cookie=15b2af7fb57592e9 in_seq=0 out_seq=0 2024-04-27T00:03:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.453+0000 7f49f3e00640 10 client.25520 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.466+0000 7f49f3e00640 1 -- 172.21.15.40:0/1192561065 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5604b18d5e00 con 0x5604b19ee590 2024-04-27T00:03:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.466+0000 7f49f3e00640 10 client.25520 handle_command_reply: tid=0 2024-04-27T00:03:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.466+0000 7f4a1bfbc740 1 client.25520 shutdown 2024-04-27T00:03:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.467+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/1192561065 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 msgr2=0x5604b1978570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.467+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1978570 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00f8d0 tx=0x7f4a0c005cf0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.467+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/1192561065 shutdown_connections 2024-04-27T00:03:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.467+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5604b19ee590 0x5604b19f0980 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.467+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604b196f730 0x5604b1979ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.467+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604b196a5c0 0x5604b1978ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.467+0000 7f4a1bfbc740 1 --2- 172.21.15.40:0/1192561065 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5604b1969bd0 0x5604b1978570 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.467+0000 7f49f3e00640 0 client.25520 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.467+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/1192561065 >> 172.21.15.40:0/1192561065 conn(0x5604b1965240 msgr2=0x5604b1963bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.467+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/1192561065 shutdown_connections 2024-04-27T00:03:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.467+0000 7f4a1bfbc740 1 -- 172.21.15.40:0/1192561065 wait complete. 2024-04-27T00:03:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.467+0000 7f4a1bfbc740 20 client.25520 trim_cache size 0 max 16384 2024-04-27T00:03:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:03:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:03:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:03:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:03:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.468+0000 7f4a19200640 1 -- 172.21.15.40:0/4030406465 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f49f80a3d70 msgr2=0x7f49f80a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.468+0000 7f4a19200640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f49f80a3d70 0x7f49f80a6230 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x5604b196a9c0 tx=0x7f4a040d4680 comp rx=0 tx=0).stop 2024-04-27T00:03:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.468+0000 7f4a19200640 1 -- 172.21.15.40:0/4030406465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14101200 msgr2=0x7f4a1419acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.468+0000 7f4a19200640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14101200 0x7f4a1419acc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400ea60 tx=0x7f4a0400ef20 comp rx=0 tx=0).stop 2024-04-27T00:03:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.468+0000 7f4a19200640 1 -- 172.21.15.40:0/4030406465 shutdown_connections 2024-04-27T00:03:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.468+0000 7f4a19200640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f49f80a3d70 0x7f49f80a6230 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.468+0000 7f4a19200640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a14102500 0x7f4a141a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.468+0000 7f4a19200640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4a14101b60 0x7f4a1419b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.468+0000 7f4a19200640 1 --2- 172.21.15.40:0/4030406465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a14101200 0x7f4a1419acc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.468+0000 7f4a19200640 1 -- 172.21.15.40:0/4030406465 >> 172.21.15.40:0/4030406465 conn(0x7f4a140fce00 msgr2=0x7f4a140fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.468+0000 7f4a19200640 1 -- 172.21.15.40:0/4030406465 shutdown_connections 2024-04-27T00:03:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:34.468+0000 7f4a19200640 1 -- 172.21.15.40:0/4030406465 wait complete. 2024-04-27T00:03:34.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:34 smithi080 ceph-mon[32362]: pgmap v468: 129 pgs: 129 active+clean; 792 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:03:34.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:34 smithi103 ceph-mon[32952]: pgmap v468: 129 pgs: 129 active+clean; 792 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:03:34.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:34 smithi040 ceph-mon[28095]: pgmap v468: 129 pgs: 129 active+clean; 792 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:03:35.717 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80882]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:35.718 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80882]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:35.718 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80882]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:35.718 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80882]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:35.719 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80885]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:35.719 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80885]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:35.719 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80885]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:35.720 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80885]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:35.846 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65207]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:35.847 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65207]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:35.847 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65207]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:35.847 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65207]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:35.847 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65210]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:35.847 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65210]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:35.847 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65210]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:35.847 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65210]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:35.893 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65882]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:35.893 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65882]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:35.893 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65882]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:35.893 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65882]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:35.894 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65885]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:35.894 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65885]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:35.894 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65885]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:35.894 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65885]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.001 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80894]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:36.001 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80894]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.001 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80894]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.001 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80888]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:36.002 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80888]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.002 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80888]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.002 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80888]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.002 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80891]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:36.002 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80891]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.002 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80891]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.002 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:03:35 smithi040 sudo[80891]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.098 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65213]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:36.098 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65213]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.098 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65213]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.098 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65213]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.099 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65216]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:36.099 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65216]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.099 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65216]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.099 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:03:35 smithi080 sudo[65216]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.099 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65219]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:36.100 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65219]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.100 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65219]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.100 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65219]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.100 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65222]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:36.100 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65222]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.100 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65222]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.101 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65222]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.143 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65889]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:36.143 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65889]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.143 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65889]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.143 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65889]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.143 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65892]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:36.143 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65892]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.143 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65892]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.143 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:03:35 smithi103 sudo[65892]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.144 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65895]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:36.144 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65895]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.144 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65895]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.144 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65895]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.144 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65898]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:36.144 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65898]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.144 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65898]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.267 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80900]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:36.267 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80900]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.267 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80900]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.267 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80900]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.268 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80894]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.268 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80897]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:36.268 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80897]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.268 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80897]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.268 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80897]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.511 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65898]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.512 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65901]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:36.512 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65901]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.512 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65901]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.512 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65901]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.513 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65904]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:36.513 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65904]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.513 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65904]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.513 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65904]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.544 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65225]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:36.545 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65225]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.545 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65225]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.545 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65225]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.545 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65228]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:36.545 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65228]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.546 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65228]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.546 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65228]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80906]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:36.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80906]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80906]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80906]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80909]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:36.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80909]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80909]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80909]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.548 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80903]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:36.549 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80903]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.549 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80903]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.549 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:03:36 smithi040 sudo[80903]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 ceph-mon[32362]: pgmap v469: 129 pgs: 129 active+clean; 792 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:03:36.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 ceph-mon[32362]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 ceph-mon[32362]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65231]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65231]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65231]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65231]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65234]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:36.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65234]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65234]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.801 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:36 smithi080 sudo[65234]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65907]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65907]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65907]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65907]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65910]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65910]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65910]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 sudo[65910]: pam_unix(sudo:session): session closed for user root 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 ceph-mon[32952]: pgmap v469: 129 pgs: 129 active+clean; 792 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 ceph-mon[32952]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T00:03:36.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 ceph-mon[32952]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T00:03:36.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T00:03:36.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T00:03:36.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T00:03:36.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T00:03:36.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T00:03:36.849 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:36 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T00:03:36.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 ceph-mon[28095]: pgmap v469: 129 pgs: 129 active+clean; 792 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:03:36.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:03:36.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:03:36.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:03:36.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 ceph-mon[28095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T00:03:36.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 ceph-mon[28095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T00:03:36.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T00:03:36.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T00:03:36.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T00:03:36.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T00:03:36.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T00:03:36.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:36 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T00:03:37.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:37 smithi103 ceph-mon[32952]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T00:03:37.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:37 smithi103 ceph-mon[32952]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T00:03:37.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:37 smithi103 ceph-mon[32952]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T00:03:37.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:37 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T00:03:37.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:37 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T00:03:37.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:37 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T00:03:37.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:37 smithi103 ceph-mon[32952]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T00:03:37.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:37 smithi040 ceph-mon[28095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T00:03:37.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:37 smithi040 ceph-mon[28095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T00:03:37.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:37 smithi040 ceph-mon[28095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T00:03:37.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:37 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T00:03:37.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:37 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T00:03:37.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:37 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T00:03:37.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:37 smithi040 ceph-mon[28095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T00:03:38.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:37 smithi080 ceph-mon[32362]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T00:03:38.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:37 smithi080 ceph-mon[32362]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T00:03:38.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:37 smithi080 ceph-mon[32362]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T00:03:38.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:37 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T00:03:38.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:37 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T00:03:38.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:37 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T00:03:38.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:37 smithi080 ceph-mon[32362]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T00:03:38.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:38 smithi103 ceph-mon[32952]: pgmap v470: 129 pgs: 129 active+clean; 856 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:03:38.950 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:38 smithi080 ceph-mon[32362]: pgmap v470: 129 pgs: 129 active+clean; 856 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:03:38.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:38 smithi040 ceph-mon[28095]: pgmap v470: 129 pgs: 129 active+clean; 856 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:03:40.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:40 smithi103 ceph-mon[32952]: pgmap v471: 129 pgs: 129 active+clean; 856 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 133 KiB/s wr, 21 op/s 2024-04-27T00:03:40.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:40 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:03:40.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:40 smithi040 ceph-mon[28095]: pgmap v471: 129 pgs: 129 active+clean; 856 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 133 KiB/s wr, 21 op/s 2024-04-27T00:03:40.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:40 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:03:41.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:40 smithi080 ceph-mon[32362]: pgmap v471: 129 pgs: 129 active+clean; 856 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 133 KiB/s wr, 21 op/s 2024-04-27T00:03:41.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:40 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:03:42.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:42 smithi103 ceph-mon[32952]: pgmap v472: 129 pgs: 129 active+clean; 856 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 133 KiB/s wr, 21 op/s 2024-04-27T00:03:42.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:42 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:03:42.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:42 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:03:42.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:42 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:03:42.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:42 smithi040 ceph-mon[28095]: pgmap v472: 129 pgs: 129 active+clean; 856 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 133 KiB/s wr, 21 op/s 2024-04-27T00:03:42.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:42 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:03:42.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:42 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:03:42.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:42 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:03:43.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:42 smithi080 ceph-mon[32362]: pgmap v472: 129 pgs: 129 active+clean; 856 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 133 KiB/s wr, 21 op/s 2024-04-27T00:03:43.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:42 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:03:43.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:42 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:03:43.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:42 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:03:44.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:44 smithi103 ceph-mon[32952]: pgmap v473: 129 pgs: 129 active+clean; 917 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 140 KiB/s wr, 25 op/s 2024-04-27T00:03:44.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:44 smithi040 ceph-mon[28095]: pgmap v473: 129 pgs: 129 active+clean; 917 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 140 KiB/s wr, 25 op/s 2024-04-27T00:03:45.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:44 smithi080 ceph-mon[32362]: pgmap v473: 129 pgs: 129 active+clean; 917 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 140 KiB/s wr, 25 op/s 2024-04-27T00:03:46.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:46 smithi103 ceph-mon[32952]: pgmap v474: 129 pgs: 129 active+clean; 917 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 133 KiB/s wr, 21 op/s 2024-04-27T00:03:46.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:46 smithi040 ceph-mon[28095]: pgmap v474: 129 pgs: 129 active+clean; 917 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 133 KiB/s wr, 21 op/s 2024-04-27T00:03:47.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:46 smithi080 ceph-mon[32362]: pgmap v474: 129 pgs: 129 active+clean; 917 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 133 KiB/s wr, 21 op/s 2024-04-27T00:03:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.617+0000 7fb2e9c00640 1 -- 172.21.15.40:0/2590479525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e41027d0 msgr2=0x7fb2e4102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.617+0000 7fb2e9c00640 1 --2- 172.21.15.40:0/2590479525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e41027d0 0x7fb2e4102bf0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009900 tx=0x7fb2d402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.618+0000 7fb2e9c00640 1 -- 172.21.15.40:0/2590479525 shutdown_connections 2024-04-27T00:03:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.618+0000 7fb2e9c00640 1 --2- 172.21.15.40:0/2590479525 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2e4104380 0x7fb2e410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.618+0000 7fb2e9c00640 1 --2- 172.21.15.40:0/2590479525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2e41039c0 0x7fb2e4103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.618+0000 7fb2e9c00640 1 --2- 172.21.15.40:0/2590479525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e41027d0 0x7fb2e4102bf0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.618+0000 7fb2e9c00640 1 -- 172.21.15.40:0/2590479525 >> 172.21.15.40:0/2590479525 conn(0x7fb2e40fdd40 msgr2=0x7fb2e4100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.618+0000 7fb2e9c00640 1 -- 172.21.15.40:0/2590479525 shutdown_connections 2024-04-27T00:03:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.618+0000 7fb2e9c00640 1 -- 172.21.15.40:0/2590479525 wait complete. 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.619+0000 7fb2e9c00640 1 Processor -- start 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.619+0000 7fb2e9c00640 1 -- start start 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.619+0000 7fb2e9c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2e41027d0 0x7fb2e419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.619+0000 7fb2e9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e41039c0 0x7fb2e419b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.619+0000 7fb2e9c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2e4104380 0x7fb2e41a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.619+0000 7fb2e9c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e410e320 con 0x7fb2e41039c0 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.619+0000 7fb2e9c00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e410e1a0 con 0x7fb2e41027d0 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.619+0000 7fb2e9c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e410e4a0 con 0x7fb2e4104380 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.620+0000 7fb2db400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e41039c0 0x7fb2e419b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.620+0000 7fb2db400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e41039c0 0x7fb2e419b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58468/0 (socket says 172.21.15.40:58468) 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.620+0000 7fb2e3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2e41027d0 0x7fb2e419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.620+0000 7fb2db400640 1 -- 172.21.15.40:0/3995276653 learned_addr learned my addr 172.21.15.40:0/3995276653 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.620+0000 7fb2e9200640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2e4104380 0x7fb2e41a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.620+0000 7fb2db400640 1 -- 172.21.15.40:0/3995276653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2e4104380 msgr2=0x7fb2e41a21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.620+0000 7fb2db400640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2e4104380 0x7fb2e41a21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.620+0000 7fb2db400640 1 -- 172.21.15.40:0/3995276653 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2e41027d0 msgr2=0x7fb2e419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.620+0000 7fb2db400640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2e41027d0 0x7fb2e419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.620+0000 7fb2db400640 1 -- 172.21.15.40:0/3995276653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e41a2860 con 0x7fb2e41039c0 2024-04-27T00:03:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.621+0000 7fb2db400640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e41039c0 0x7fb2e419b190 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00ea60 tx=0x7fb2cc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.621+0000 7fb2e1600640 1 -- 172.21.15.40:0/3995276653 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc00cde0 con 0x7fb2e41039c0 2024-04-27T00:03:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.621+0000 7fb2e1600640 1 -- 172.21.15.40:0/3995276653 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2cc0035f0 con 0x7fb2e41039c0 2024-04-27T00:03:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.621+0000 7fb2e1600640 1 -- 172.21.15.40:0/3995276653 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc010690 con 0x7fb2e41039c0 2024-04-27T00:03:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.621+0000 7fb2e9c00640 1 -- 172.21.15.40:0/3995276653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e41a2b50 con 0x7fb2e41039c0 2024-04-27T00:03:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.621+0000 7fb2e9c00640 1 -- 172.21.15.40:0/3995276653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e41a3020 con 0x7fb2e41039c0 2024-04-27T00:03:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.624+0000 7fb2ee079740 1 Processor -- start 2024-04-27T00:03:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.624+0000 7fb2ee079740 1 -- start start 2024-04-27T00:03:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.625+0000 7fb2ee079740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 0x55fbb539f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.625+0000 7fb2e1600640 1 -- 172.21.15.40:0/3995276653 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb2cc0037a0 con 0x7fb2e41039c0 2024-04-27T00:03:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.625+0000 7fb2ee079740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 0x55fbb53a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.625+0000 7fb2ee079740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a4940 0x55fbb53a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.625+0000 7fb2ee079740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fbb5303380 con 0x55fbb539f7d0 2024-04-27T00:03:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.625+0000 7fb2ee079740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55fbb529fc30 con 0x55fbb539ede0 2024-04-27T00:03:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.625+0000 7fb2ee079740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fbb52ff340 con 0x55fbb53a4940 2024-04-27T00:03:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.625+0000 7fb2e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 0x55fbb53a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.625+0000 7fb2db400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a4940 0x55fbb53a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.625+0000 7fb2e9200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 0x55fbb539f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.625+0000 7fb2e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 0x55fbb53a42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58480/0 (socket says 172.21.15.40:58480) 2024-04-27T00:03:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.625+0000 7fb2e3e00640 1 -- 172.21.15.40:0/1465558110 learned_addr learned my addr 172.21.15.40:0/1465558110 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.626+0000 7fb2e9200640 1 -- 172.21.15.40:0/1465558110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a4940 msgr2=0x55fbb53a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.626+0000 7fb2e9200640 1 --2- 172.21.15.40:0/1465558110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a4940 0x55fbb53a6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.626+0000 7fb2e9200640 1 -- 172.21.15.40:0/1465558110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 msgr2=0x55fbb53a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.626+0000 7fb2e9200640 1 --2- 172.21.15.40:0/1465558110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 0x55fbb53a42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.626+0000 7fb2e9200640 1 -- 172.21.15.40:0/1465558110 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbb52c1940 con 0x55fbb539ede0 2024-04-27T00:03:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.626+0000 7fb2e9200640 1 --2- 172.21.15.40:0/1465558110 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 0x55fbb539f200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009f00 tx=0x7fb2dc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3823713b5d30ea95 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.626+0000 7fb2d9600640 1 -- 172.21.15.40:0/1465558110 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc004d20 con 0x55fbb539ede0 2024-04-27T00:03:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.626+0000 7fb2d9600640 1 -- 172.21.15.40:0/1465558110 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2dc004030 con 0x55fbb539ede0 2024-04-27T00:03:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.626+0000 7fb2d9600640 1 -- 172.21.15.40:0/1465558110 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc005280 con 0x55fbb539ede0 2024-04-27T00:03:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.626+0000 7fb2e1600640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb2c80a3cb0 0x7fb2c80a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.627+0000 7fb2e1600640 1 -- 172.21.15.40:0/3995276653 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fb2cc0d0c30 con 0x7fb2e41039c0 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.627+0000 7fb2e3e00640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb2c80a3cb0 0x7fb2c80a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.627+0000 7fb2ee079740 1 -- 172.21.15.40:0/1465558110 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 msgr2=0x55fbb539f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.627+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1465558110 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 0x55fbb539f200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009f00 tx=0x7fb2dc011810 comp rx=0 tx=0).stop 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.627+0000 7fb2ee079740 1 -- 172.21.15.40:0/1465558110 shutdown_connections 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.627+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1465558110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a4940 0x55fbb53a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.627+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1465558110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 0x55fbb53a42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.627+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1465558110 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 0x55fbb539f200 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.627+0000 7fb2ee079740 1 -- 172.21.15.40:0/1465558110 >> 172.21.15.40:0/1465558110 conn(0x55fbb539a450 msgr2=0x55fbb539a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.627+0000 7fb2ee079740 1 -- 172.21.15.40:0/1465558110 shutdown_connections 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.627+0000 7fb2ee079740 1 -- 172.21.15.40:0/1465558110 wait complete. 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.628+0000 7fb2e3e00640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb2c80a3cb0 0x7fb2c80a6170 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x55fbb53a0240 tx=0x7fb2d4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.628+0000 7fb2ee079740 1 -- start start 2024-04-27T00:03:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.628+0000 7fb2ee079740 1 Processor -- start 2024-04-27T00:03:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2ee079740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 0x55fbb53a8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2ee079740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb539f7d0 0x55fbb53a9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2ee079740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb53a4940 0x55fbb53b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2ee079740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fbb53a78f0 con 0x55fbb53a4940 2024-04-27T00:03:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2ee079740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55fbb528ac60 con 0x55fbb539ede0 2024-04-27T00:03:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2ee079740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fbb53a7a70 con 0x55fbb539f7d0 2024-04-27T00:03:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2ee079740 10 client.0 _ll_register_callbacks cb 0x55fbb52b6e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:03:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2e3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 0x55fbb53a8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2e3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 0x55fbb53a8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:42590/0 (socket says 172.21.15.40:42590) 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2e3e00640 1 -- 172.21.15.40:0/1316099727 learned_addr learned my addr 172.21.15.40:0/1316099727 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2e9200640 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb53a4940 0x55fbb53b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2db400640 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb539f7d0 0x55fbb53a9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2e3e00640 1 -- 172.21.15.40:0/1316099727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb539f7d0 msgr2=0x55fbb53a9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2e3e00640 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb539f7d0 0x55fbb53a9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2e3e00640 1 -- 172.21.15.40:0/1316099727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb53a4940 msgr2=0x55fbb53b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2e3e00640 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb53a4940 0x55fbb53b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.629+0000 7fb2e3e00640 1 -- 172.21.15.40:0/1316099727 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbb5299510 con 0x55fbb539ede0 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.630+0000 7fb2e3e00640 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 0x55fbb53a8ff0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009230 tx=0x7fb2d4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f83af93e6e9ad853 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.630+0000 7fb2d2000640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.630+0000 7fb2d2000640 1 -- 172.21.15.40:0/1316099727 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4035040 con 0x55fbb539ede0 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.630+0000 7fb2d2000640 1 -- 172.21.15.40:0/1316099727 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fb2d40319b0 con 0x55fbb539ede0 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.630+0000 7fb2ee079740 1 -- 172.21.15.40:0/1316099727 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2dc004030 con 0x55fbb539ede0 2024-04-27T00:03:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.630+0000 7fb2d2000640 1 -- 172.21.15.40:0/1316099727 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb2d4003e80 con 0x55fbb539ede0 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.630+0000 7fb2ee079740 10 client.25535 fetch_fsmap learned FSMap version 24 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.630+0000 7fb2ee079740 1 -- 172.21.15.40:0/1316099727 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fbb5015210 con 0x55fbb539ede0 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.631+0000 7fb2d2000640 1 -- 172.21.15.40:0/1316099727 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2dc004030 con 0x55fbb539ede0 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.631+0000 7fb2d2000640 1 -- 172.21.15.40:0/1316099727 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb2d4044030 con 0x55fbb539ede0 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.631+0000 7fb2ee079740 10 client.25535 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.631+0000 7fb2ee079740 10 client.25535 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.631+0000 7fb2ee079740 20 client.25535 populate_metadata read hostname 'smithi040' 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.631+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55fbb53b4a60 0x55fbb53b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.631+0000 7fb2ee079740 4 client.25535 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.631+0000 7fb2ee079740 1 -- 172.21.15.40:0/1316099727 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbb5310f90 con 0x55fbb53b4a60 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.631+0000 7fb2db400640 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55fbb53b4a60 0x55fbb53b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.632+0000 7fb2db400640 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55fbb53b4a60 0x55fbb53b6eb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=479b733cbee89509 server_cookie=e7a1c049ba2af17a in_seq=0 out_seq=0 2024-04-27T00:03:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.632+0000 7fb2d2000640 10 client.25535 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2d2000640 1 -- 172.21.15.40:0/1316099727 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fbb5310f90 con 0x55fbb53b4a60 2024-04-27T00:03:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2d2000640 10 client.25535 handle_command_reply: tid=0 2024-04-27T00:03:47.636 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2ee079740 1 client.25535 shutdown 2024-04-27T00:03:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2ee079740 1 -- 172.21.15.40:0/1316099727 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 msgr2=0x55fbb53a8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 0x55fbb53a8ff0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009230 tx=0x7fb2d4009260 comp rx=0 tx=0).stop 2024-04-27T00:03:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2ee079740 1 -- 172.21.15.40:0/1316099727 shutdown_connections 2024-04-27T00:03:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55fbb53b4a60 0x55fbb53b6eb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb53a4940 0x55fbb53b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb539f7d0 0x55fbb53a9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1316099727 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539ede0 0x55fbb53a8ff0 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009230 tx=0x7fb2d4009260 comp rx=0 tx=0).stop 2024-04-27T00:03:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2d2000640 0 client.25535 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2ee079740 1 -- 172.21.15.40:0/1316099727 >> 172.21.15.40:0/1316099727 conn(0x55fbb539a450 msgr2=0x55fbb5399650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.633+0000 7fb2ee079740 1 -- 172.21.15.40:0/1316099727 shutdown_connections 2024-04-27T00:03:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.634+0000 7fb2ee079740 1 -- 172.21.15.40:0/1316099727 wait complete. 2024-04-27T00:03:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.634+0000 7fb2ee079740 20 client.25535 trim_cache size 0 max 16384 2024-04-27T00:03:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2ee079740 1 Processor -- start 2024-04-27T00:03:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2ee079740 1 -- start start 2024-04-27T00:03:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2ee079740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539f7d0 0x55fbb53a8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2ee079740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a4940 0x55fbb53b0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2ee079740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb53a9140 0x55fbb5398dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2ee079740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e41afe50 con 0x55fbb53a9140 2024-04-27T00:03:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2ee079740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e41ad720 con 0x55fbb539f7d0 2024-04-27T00:03:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2ee079740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e41ab570 con 0x55fbb53a4940 2024-04-27T00:03:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2e9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb53a9140 0x55fbb5398dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2e9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb53a9140 0x55fbb5398dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58490/0 (socket says 172.21.15.40:58490) 2024-04-27T00:03:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2e9200640 1 -- 172.21.15.40:0/3756850055 learned_addr learned my addr 172.21.15.40:0/3756850055 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2db400640 1 --2- 172.21.15.40:0/3756850055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a4940 0x55fbb53b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.652+0000 7fb2e3e00640 1 --2- 172.21.15.40:0/3756850055 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539f7d0 0x55fbb53a8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.653+0000 7fb2e9200640 1 -- 172.21.15.40:0/3756850055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a4940 msgr2=0x55fbb53b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.653+0000 7fb2e9200640 1 --2- 172.21.15.40:0/3756850055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a4940 0x55fbb53b0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.653+0000 7fb2e9200640 1 -- 172.21.15.40:0/3756850055 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539f7d0 msgr2=0x55fbb53a8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.653+0000 7fb2e9200640 1 --2- 172.21.15.40:0/3756850055 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539f7d0 0x55fbb53a8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.653+0000 7fb2e9200640 1 -- 172.21.15.40:0/3756850055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbb50e1620 con 0x55fbb53a9140 2024-04-27T00:03:47.655 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.653+0000 7fb2e3e00640 1 --2- 172.21.15.40:0/3756850055 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539f7d0 0x55fbb53a8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:03:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.653+0000 7fb2e9200640 1 --2- 172.21.15.40:0/3756850055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb53a9140 0x55fbb5398dc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009f00 tx=0x7fb2dc009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4581a246ed848b6b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.654+0000 7fb2d8c00640 1 -- 172.21.15.40:0/3756850055 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc028070 con 0x55fbb53a9140 2024-04-27T00:03:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.654+0000 7fb2d8c00640 1 -- 172.21.15.40:0/3756850055 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2dc003b70 con 0x55fbb53a9140 2024-04-27T00:03:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.654+0000 7fb2d8c00640 1 -- 172.21.15.40:0/3756850055 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc0246a0 con 0x55fbb53a9140 2024-04-27T00:03:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.654+0000 7fb2ee079740 1 -- 172.21.15.40:0/3756850055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb53a9140 msgr2=0x55fbb5398dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.654+0000 7fb2ee079740 1 --2- 172.21.15.40:0/3756850055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb53a9140 0x55fbb5398dc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009f00 tx=0x7fb2dc009b30 comp rx=0 tx=0).stop 2024-04-27T00:03:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.655+0000 7fb2ee079740 1 -- 172.21.15.40:0/3756850055 shutdown_connections 2024-04-27T00:03:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.655+0000 7fb2ee079740 1 --2- 172.21.15.40:0/3756850055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb53a9140 0x55fbb5398dc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.655+0000 7fb2ee079740 1 --2- 172.21.15.40:0/3756850055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a4940 0x55fbb53b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.655+0000 7fb2ee079740 1 --2- 172.21.15.40:0/3756850055 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb539f7d0 0x55fbb53a8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.655+0000 7fb2ee079740 1 -- 172.21.15.40:0/3756850055 >> 172.21.15.40:0/3756850055 conn(0x55fbb539a450 msgr2=0x55fbb539a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.655+0000 7fb2ee079740 1 -- 172.21.15.40:0/3756850055 shutdown_connections 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.655+0000 7fb2ee079740 1 -- 172.21.15.40:0/3756850055 wait complete. 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.655+0000 7fb2ee079740 1 -- start start 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.655+0000 7fb2ee079740 1 Processor -- start 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.656+0000 7fb2ee079740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 0x55fbb53ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.656+0000 7fb2ee079740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb53a4940 0x55fbb53adc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.656+0000 7fb2ee079740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a9140 0x55fbb53aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.656+0000 7fb2ee079740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e41abbe0 con 0x55fbb539f7d0 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.656+0000 7fb2ee079740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e41ad050 con 0x55fbb53a4940 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.656+0000 7fb2ee079740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e41ac2b0 con 0x55fbb53a9140 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.656+0000 7fb2ee079740 10 client.0 _ll_register_callbacks cb 0x7fb2d4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:03:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.656+0000 7fb2e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 0x55fbb53ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.656+0000 7fb2e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 0x55fbb53ad6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58498/0 (socket says 172.21.15.40:58498) 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.656+0000 7fb2e3e00640 1 -- 172.21.15.40:0/1959843983 learned_addr learned my addr 172.21.15.40:0/1959843983 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.656+0000 7fb2db400640 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb53a4940 0x55fbb53adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.656+0000 7fb2e9200640 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a9140 0x55fbb53aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2e3e00640 1 -- 172.21.15.40:0/1959843983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a9140 msgr2=0x55fbb53aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2e3e00640 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a9140 0x55fbb53aebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2e3e00640 1 -- 172.21.15.40:0/1959843983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb53a4940 msgr2=0x55fbb53adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2e3e00640 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb53a4940 0x55fbb53adc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2e3e00640 1 -- 172.21.15.40:0/1959843983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbb52bdcb0 con 0x55fbb539f7d0 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2db400640 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb53a4940 0x55fbb53adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2e3e00640 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 0x55fbb53ad6c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4051e10 tx=0x7fb2d403d700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675df3e074f612ab server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2d2000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2d2000640 1 -- 172.21.15.40:0/1959843983 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4063060 con 0x55fbb539f7d0 2024-04-27T00:03:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2ee079740 1 -- 172.21.15.40:0/1959843983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2dc003b70 con 0x55fbb539f7d0 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2d2000640 1 -- 172.21.15.40:0/1959843983 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fb2d403dc30 con 0x55fbb539f7d0 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2d2000640 1 -- 172.21.15.40:0/1959843983 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb2d4035040 con 0x55fbb539f7d0 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.657+0000 7fb2ee079740 10 client.15423 fetch_fsmap learned FSMap version 24 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.658+0000 7fb2ee079740 1 -- 172.21.15.40:0/1959843983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fbb53b02f0 con 0x55fbb539f7d0 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.658+0000 7fb2d2000640 1 -- 172.21.15.40:0/1959843983 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2dc003b70 con 0x55fbb539f7d0 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.658+0000 7fb2d2000640 1 -- 172.21.15.40:0/1959843983 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb2d4051590 con 0x55fbb539f7d0 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.658+0000 7fb2ee079740 10 client.15423 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.658+0000 7fb2ee079740 10 client.15423 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.658+0000 7fb2ee079740 20 client.15423 populate_metadata read hostname 'smithi040' 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.658+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55fbb5423a50 0x55fbb5425e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.658+0000 7fb2ee079740 4 client.15423 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.658+0000 7fb2ee079740 1 -- 172.21.15.40:0/1959843983 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fbb530ae00 con 0x55fbb5423a50 2024-04-27T00:03:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.658+0000 7fb2db400640 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55fbb5423a50 0x55fbb5425e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:03:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.659+0000 7fb2db400640 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55fbb5423a50 0x55fbb5425e40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bcf9fd648cf55af server_cookie=ccfc93e080b34b7 in_seq=0 out_seq=0 2024-04-27T00:03:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.659+0000 7fb2d2000640 10 client.15423 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.671+0000 7fb2d2000640 1 -- 172.21.15.40:0/1959843983 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fbb530ae00 con 0x55fbb5423a50 2024-04-27T00:03:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.671+0000 7fb2d2000640 10 client.15423 handle_command_reply: tid=0 2024-04-27T00:03:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.671+0000 7fb2ee079740 1 client.15423 shutdown 2024-04-27T00:03:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.671+0000 7fb2ee079740 1 -- 172.21.15.40:0/1959843983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 msgr2=0x55fbb53ad6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.671+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 0x55fbb53ad6c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4051e10 tx=0x7fb2d403d700 comp rx=0 tx=0).stop 2024-04-27T00:03:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.671+0000 7fb2ee079740 1 -- 172.21.15.40:0/1959843983 shutdown_connections 2024-04-27T00:03:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.671+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55fbb5423a50 0x55fbb5425e40 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.671+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fbb53a9140 0x55fbb53aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.671+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55fbb53a4940 0x55fbb53adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.671+0000 7fb2ee079740 1 --2- 172.21.15.40:0/1959843983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fbb539f7d0 0x55fbb53ad6c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.671+0000 7fb2ee079740 1 -- 172.21.15.40:0/1959843983 >> 172.21.15.40:0/1959843983 conn(0x55fbb539a450 msgr2=0x55fbb5398dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.671+0000 7fb2d2000640 0 client.15423 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:03:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.672+0000 7fb2ee079740 1 -- 172.21.15.40:0/1959843983 shutdown_connections 2024-04-27T00:03:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.672+0000 7fb2ee079740 1 -- 172.21.15.40:0/1959843983 wait complete. 2024-04-27T00:03:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.672+0000 7fb2ee079740 20 client.15423 trim_cache size 0 max 16384 2024-04-27T00:03:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:03:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:03:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:03:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:03:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.673+0000 7fb2e9c00640 1 -- 172.21.15.40:0/3995276653 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb2c80a3cb0 msgr2=0x7fb2c80a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.673+0000 7fb2e9c00640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb2c80a3cb0 0x7fb2c80a6170 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x55fbb53a0240 tx=0x7fb2d4005a20 comp rx=0 tx=0).stop 2024-04-27T00:03:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.673+0000 7fb2e9c00640 1 -- 172.21.15.40:0/3995276653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e41039c0 msgr2=0x7fb2e419b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:03:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.673+0000 7fb2e9c00640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e41039c0 0x7fb2e419b190 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00ea60 tx=0x7fb2cc00ef20 comp rx=0 tx=0).stop 2024-04-27T00:03:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.673+0000 7fb2e9c00640 1 -- 172.21.15.40:0/3995276653 shutdown_connections 2024-04-27T00:03:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.673+0000 7fb2e9c00640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb2c80a3cb0 0x7fb2c80a6170 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.673+0000 7fb2e9c00640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2e4104380 0x7fb2e41a21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.673+0000 7fb2e9c00640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2e41039c0 0x7fb2e419b190 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.673+0000 7fb2e9c00640 1 --2- 172.21.15.40:0/3995276653 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb2e41027d0 0x7fb2e419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:03:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.673+0000 7fb2e9c00640 1 -- 172.21.15.40:0/3995276653 >> 172.21.15.40:0/3995276653 conn(0x7fb2e40fdd40 msgr2=0x7fb2e4074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:03:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.673+0000 7fb2e9c00640 1 -- 172.21.15.40:0/3995276653 shutdown_connections 2024-04-27T00:03:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:03:47.673+0000 7fb2e9c00640 1 -- 172.21.15.40:0/3995276653 wait complete. 2024-04-27T00:03:48.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:48 smithi103 ceph-mon[32952]: pgmap v475: 129 pgs: 129 active+clean; 975 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 139 KiB/s wr, 25 op/s 2024-04-27T00:03:48.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:48 smithi040 ceph-mon[28095]: pgmap v475: 129 pgs: 129 active+clean; 975 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 139 KiB/s wr, 25 op/s 2024-04-27T00:03:49.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:48 smithi080 ceph-mon[32362]: pgmap v475: 129 pgs: 129 active+clean; 975 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 139 KiB/s wr, 25 op/s 2024-04-27T00:03:50.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:50 smithi103 ceph-mon[32952]: pgmap v476: 129 pgs: 129 active+clean; 975 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 132 KiB/s wr, 19 op/s 2024-04-27T00:03:50.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:50 smithi040 ceph-mon[28095]: pgmap v476: 129 pgs: 129 active+clean; 975 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 132 KiB/s wr, 19 op/s 2024-04-27T00:03:51.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:50 smithi080 ceph-mon[32362]: pgmap v476: 129 pgs: 129 active+clean; 975 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 132 KiB/s wr, 19 op/s 2024-04-27T00:03:52.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:52 smithi103 ceph-mon[32952]: pgmap v477: 129 pgs: 129 active+clean; 975 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:03:52.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:52 smithi040 ceph-mon[28095]: pgmap v477: 129 pgs: 129 active+clean; 975 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:03:53.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:52 smithi080 ceph-mon[32362]: pgmap v477: 129 pgs: 129 active+clean; 975 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:03:54.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:54 smithi040 ceph-mon[28095]: pgmap v478: 129 pgs: 129 active+clean; 1.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:03:55.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:54 smithi080 ceph-mon[32362]: pgmap v478: 129 pgs: 129 active+clean; 1.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:03:55.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:54 smithi103 ceph-mon[32952]: pgmap v478: 129 pgs: 129 active+clean; 1.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:03:56.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:56 smithi040 ceph-mon[28095]: pgmap v479: 129 pgs: 129 active+clean; 1.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:03:57.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:56 smithi080 ceph-mon[32362]: pgmap v479: 129 pgs: 129 active+clean; 1.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:03:57.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:56 smithi103 ceph-mon[32952]: pgmap v479: 129 pgs: 129 active+clean; 1.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:03:58.899 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:03:58 smithi103 ceph-mon[32952]: pgmap v480: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:03:58.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:03:58 smithi040 ceph-mon[28095]: pgmap v480: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:03:59.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:03:58 smithi080 ceph-mon[32362]: pgmap v480: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.825+0000 7fb003e00640 1 -- 172.21.15.40:0/4188789784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb004076970 msgr2=0x7fb004074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.825+0000 7fb003e00640 1 --2- 172.21.15.40:0/4188789784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb004076970 0x7fb004074dc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7faff4009930 tx=0x7faff402a440 comp rx=0 tx=0).stop 2024-04-27T00:04:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.825+0000 7fb003e00640 1 -- 172.21.15.40:0/4188789784 shutdown_connections 2024-04-27T00:04:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.825+0000 7fb003e00640 1 --2- 172.21.15.40:0/4188789784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb00410eda0 0x7fb004111190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.825+0000 7fb003e00640 1 --2- 172.21.15.40:0/4188789784 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb004075390 0x7fb004075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.825+0000 7fb003e00640 1 --2- 172.21.15.40:0/4188789784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb004076970 0x7fb004074dc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.825+0000 7fb003e00640 1 -- 172.21.15.40:0/4188789784 >> 172.21.15.40:0/4188789784 conn(0x7fb0040fdd40 msgr2=0x7fb004100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.825+0000 7fb003e00640 1 -- 172.21.15.40:0/4188789784 shutdown_connections 2024-04-27T00:04:00.827 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.825+0000 7fb003e00640 1 -- 172.21.15.40:0/4188789784 wait complete. 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.826+0000 7fb003e00640 1 Processor -- start 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.826+0000 7fb003e00640 1 -- start start 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.827+0000 7fb003e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb004075390 0x7fb00419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.827+0000 7fb003e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb004076970 0x7fb00419f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.827+0000 7fb003e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb00410eda0 0x7fb0041a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.827+0000 7fb003e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb0041148f0 con 0x7fb004075390 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.827+0000 7fb003e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb004114770 con 0x7fb004076970 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.827+0000 7fb003e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb004114a70 con 0x7fb00410eda0 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.827+0000 7fb002a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb004075390 0x7fb00419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.827+0000 7fb003400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb00410eda0 0x7fb0041a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.827+0000 7fb003400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb00410eda0 0x7fb0041a65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:58256/0 (socket says 172.21.15.40:58256) 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.827+0000 7fb003400640 1 -- 172.21.15.40:0/599834655 learned_addr learned my addr 172.21.15.40:0/599834655 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:00.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.827+0000 7fb002000640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb004076970 0x7fb00419f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.828+0000 7fb003400640 1 -- 172.21.15.40:0/599834655 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb004076970 msgr2=0x7fb00419f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:00.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.828+0000 7fb003400640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb004076970 0x7fb00419f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.828+0000 7fb003400640 1 -- 172.21.15.40:0/599834655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb004075390 msgr2=0x7fb00419f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.828+0000 7fb003400640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb004075390 0x7fb00419f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.828+0000 7fb003400640 1 -- 172.21.15.40:0/599834655 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0041a6c60 con 0x7fb00410eda0 2024-04-27T00:04:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.828+0000 7fb002a00640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb004075390 0x7fb00419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:04:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.828+0000 7fb003400640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb00410eda0 0x7fb0041a65f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7faffc00eb10 tx=0x7faffc00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.828+0000 7fb008a00640 1 -- 172.21.15.40:0/599834655 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc004b20 con 0x7fb00410eda0 2024-04-27T00:04:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.828+0000 7fb003e00640 1 -- 172.21.15.40:0/599834655 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0041a6f50 con 0x7fb00410eda0 2024-04-27T00:04:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.828+0000 7fb008a00640 1 -- 172.21.15.40:0/599834655 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faffc004cc0 con 0x7fb00410eda0 2024-04-27T00:04:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.828+0000 7fb008a00640 1 -- 172.21.15.40:0/599834655 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc010690 con 0x7fb00410eda0 2024-04-27T00:04:00.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.828+0000 7fb003e00640 1 -- 172.21.15.40:0/599834655 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0041a7410 con 0x7fb00410eda0 2024-04-27T00:04:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.832+0000 7fb00cc2a740 1 Processor -- start 2024-04-27T00:04:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.832+0000 7fb008a00640 1 -- 172.21.15.40:0/599834655 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faffc005020 con 0x7fb00410eda0 2024-04-27T00:04:00.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.832+0000 7fb00cc2a740 1 -- start start 2024-04-27T00:04:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.832+0000 7fb00cc2a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441cde0 0x55b3b441d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.832+0000 7fb00cc2a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b441d7d0 0x55b3b44222d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.832+0000 7fb00cc2a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b4422940 0x55b3b4424dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.832+0000 7fb00cc2a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b3b4381380 con 0x55b3b4422940 2024-04-27T00:04:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.832+0000 7fb00cc2a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55b3b431dc30 con 0x55b3b441d7d0 2024-04-27T00:04:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.832+0000 7fb00cc2a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b3b437d340 con 0x55b3b441cde0 2024-04-27T00:04:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.833+0000 7fb002000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441cde0 0x55b3b441d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.833+0000 7fb002a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b441d7d0 0x55b3b44222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.834 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.833+0000 7fb002000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441cde0 0x55b3b441d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:58272/0 (socket says 172.21.15.40:58272) 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.833+0000 7fb002000640 1 -- 172.21.15.40:0/826112315 learned_addr learned my addr 172.21.15.40:0/826112315 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.833+0000 7fb002000640 1 --2- 172.21.15.40:0/826112315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b4422940 0x55b3b4424dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.833+0000 7fb002a00640 1 -- 172.21.15.40:0/826112315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441cde0 msgr2=0x55b3b441d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.833+0000 7fb002a00640 1 --2- 172.21.15.40:0/826112315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441cde0 0x55b3b441d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.833+0000 7fb002a00640 1 -- 172.21.15.40:0/826112315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b4422940 msgr2=0x55b3b4424dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.833+0000 7fb002a00640 1 --2- 172.21.15.40:0/826112315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b4422940 0x55b3b4424dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.833+0000 7fb002a00640 1 -- 172.21.15.40:0/826112315 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3b433f940 con 0x55b3b441d7d0 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.833+0000 7fb002a00640 1 --2- 172.21.15.40:0/826112315 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b441d7d0 0x55b3b44222d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7faff4005ab0 tx=0x7faff4005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcefe863e14727e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.834+0000 7fafefe00640 1 -- 172.21.15.40:0/826112315 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff4035070 con 0x55b3b441d7d0 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.834+0000 7fb008a00640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fafe80a3c30 0x7fafe80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.834+0000 7fafefe00640 1 -- 172.21.15.40:0/826112315 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faff40318a0 con 0x55b3b441d7d0 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.834+0000 7fafefe00640 1 -- 172.21.15.40:0/826112315 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff4003d50 con 0x55b3b441d7d0 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.834+0000 7fb008a00640 1 -- 172.21.15.40:0/599834655 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7faffc0d0e60 con 0x7fb00410eda0 2024-04-27T00:04:00.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.834+0000 7fb002000640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fafe80a3c30 0x7fafe80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.834+0000 7fb00cc2a740 1 -- 172.21.15.40:0/826112315 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b441d7d0 msgr2=0x55b3b44222d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.834+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/826112315 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b441d7d0 0x55b3b44222d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7faff4005ab0 tx=0x7faff4005e80 comp rx=0 tx=0).stop 2024-04-27T00:04:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.835+0000 7fb00cc2a740 1 -- 172.21.15.40:0/826112315 shutdown_connections 2024-04-27T00:04:00.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.835+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/826112315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b4422940 0x55b3b4424dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.835+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/826112315 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b441d7d0 0x55b3b44222d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.835+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/826112315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441cde0 0x55b3b441d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.835+0000 7fb00cc2a740 1 -- 172.21.15.40:0/826112315 >> 172.21.15.40:0/826112315 conn(0x55b3b4418450 msgr2=0x55b3b4418850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.835+0000 7fb00cc2a740 1 -- 172.21.15.40:0/826112315 shutdown_connections 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.835+0000 7fb00cc2a740 1 -- 172.21.15.40:0/826112315 wait complete. 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.835+0000 7fb002000640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fafe80a3c30 0x7fafe80a60f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55b3b441e030 tx=0x7faff000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.835+0000 7fb00cc2a740 1 -- start start 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.835+0000 7fb00cc2a740 1 Processor -- start 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.836+0000 7fb00cc2a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b441cde0 0x55b3b4426f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.836+0000 7fb00cc2a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 0x55b3b44274d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.836+0000 7fb00cc2a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b4422940 0x55b3b442e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.836+0000 7fb00cc2a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b3b4425890 con 0x55b3b441d7d0 2024-04-27T00:04:00.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.836+0000 7fb00cc2a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55b3b4308c60 con 0x55b3b441cde0 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.836+0000 7fb00cc2a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b3b4425a10 con 0x55b3b4422940 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.836+0000 7fb00cc2a740 10 client.0 _ll_register_callbacks cb 0x55b3b4334e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.836+0000 7fb002000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 0x55b3b44274d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.836+0000 7fb002000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 0x55b3b44274d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57192/0 (socket says 172.21.15.40:57192) 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.836+0000 7fb002000640 1 -- 172.21.15.40:0/2553904516 learned_addr learned my addr 172.21.15.40:0/2553904516 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.836+0000 7fb002a00640 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b441cde0 0x55b3b4426f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.836+0000 7fb003400640 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b4422940 0x55b3b442e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.837+0000 7fb002000640 1 -- 172.21.15.40:0/2553904516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b4422940 msgr2=0x55b3b442e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.837+0000 7fb002000640 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b4422940 0x55b3b442e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.837+0000 7fb002000640 1 -- 172.21.15.40:0/2553904516 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b441cde0 msgr2=0x55b3b4426f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.837+0000 7fb002000640 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b441cde0 0x55b3b4426f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.837+0000 7fb002000640 1 -- 172.21.15.40:0/2553904516 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3b4317510 con 0x55b3b441d7d0 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.837+0000 7fb002000640 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 0x55b3b44274d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7faff0005d00 tx=0x7faff0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797001f7eb643e7f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:00.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.837+0000 7fafecc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.838+0000 7fafecc00640 1 -- 172.21.15.40:0/2553904516 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff0004480 con 0x55b3b441d7d0 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.838+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2553904516 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faff40318a0 con 0x55b3b441d7d0 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.838+0000 7fafecc00640 1 -- 172.21.15.40:0/2553904516 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7faff0012790 con 0x55b3b441d7d0 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.838+0000 7fafecc00640 1 -- 172.21.15.40:0/2553904516 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faff0012990 con 0x55b3b441d7d0 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.838+0000 7fb00cc2a740 10 client.15435 fetch_fsmap learned FSMap version 24 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.838+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2553904516 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b3b4093210 con 0x55b3b441d7d0 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.838+0000 7fafecc00640 1 -- 172.21.15.40:0/2553904516 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faff40318a0 con 0x55b3b441d7d0 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.838+0000 7fafecc00640 1 -- 172.21.15.40:0/2553904516 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7faff0028070 con 0x55b3b441d7d0 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.839+0000 7fb00cc2a740 10 client.15435 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.839+0000 7fb00cc2a740 10 client.15435 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.839+0000 7fb00cc2a740 20 client.15435 populate_metadata read hostname 'smithi040' 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.839+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b3b4432a80 0x55b3b4434f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.839+0000 7fb00cc2a740 4 client.15435 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:04:00.843 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.839+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2553904516 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3b438ef90 con 0x55b3b4432a80 2024-04-27T00:04:00.843 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.839+0000 7fb002a00640 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b3b4432a80 0x55b3b4434f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.840+0000 7fb002a00640 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b3b4432a80 0x55b3b4434f40 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e324ba3e73ef8cb2 server_cookie=cb34d476c603bd9 in_seq=0 out_seq=0 2024-04-27T00:04:00.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.840+0000 7fafecc00640 10 client.15435 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:00.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.840+0000 7fafecc00640 1 -- 172.21.15.40:0/2553904516 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3b438ef90 con 0x55b3b4432a80 2024-04-27T00:04:00.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.840+0000 7fafecc00640 10 client.15435 handle_command_reply: tid=0 2024-04-27T00:04:00.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fb00cc2a740 1 client.15435 shutdown 2024-04-27T00:04:00.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2553904516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 msgr2=0x55b3b44274d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:00.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 0x55b3b44274d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7faff0005d00 tx=0x7faff0004730 comp rx=0 tx=0).stop 2024-04-27T00:04:00.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2553904516 shutdown_connections 2024-04-27T00:04:00.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b3b4432a80 0x55b3b4434f40 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b4422940 0x55b3b442e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 0x55b3b44274d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fafecc00640 0 client.15435 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:00.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/2553904516 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b441cde0 0x55b3b4426f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2553904516 >> 172.21.15.40:0/2553904516 conn(0x55b3b4418450 msgr2=0x55b3b4417650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:00.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2553904516 shutdown_connections 2024-04-27T00:04:00.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2553904516 wait complete. 2024-04-27T00:04:00.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.841+0000 7fb00cc2a740 20 client.15435 trim_cache size 0 max 16384 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb00cc2a740 1 Processor -- start 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb00cc2a740 1 -- start start 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb00cc2a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441cde0 0x55b3b443a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb00cc2a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 0x55b3b443ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb00cc2a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b4422940 0x55b3b442e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb00cc2a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb0041b4250 con 0x55b3b441d7d0 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb00cc2a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb0041b1b20 con 0x55b3b4422940 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb00cc2a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb0041af970 con 0x55b3b441cde0 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb002000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 0x55b3b443ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb002000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 0x55b3b443ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57204/0 (socket says 172.21.15.40:57204) 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb002000640 1 -- 172.21.15.40:0/2338729702 learned_addr learned my addr 172.21.15.40:0/2338729702 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb003400640 1 --2- 172.21.15.40:0/2338729702 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b4422940 0x55b3b442e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.858+0000 7fb002a00640 1 --2- 172.21.15.40:0/2338729702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441cde0 0x55b3b443a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.859+0000 7fb002000640 1 -- 172.21.15.40:0/2338729702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441cde0 msgr2=0x55b3b443a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.859+0000 7fb002000640 1 --2- 172.21.15.40:0/2338729702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441cde0 0x55b3b443a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.859+0000 7fb002000640 1 -- 172.21.15.40:0/2338729702 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b4422940 msgr2=0x55b3b442e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.859+0000 7fb002000640 1 --2- 172.21.15.40:0/2338729702 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b4422940 0x55b3b442e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.859+0000 7fb002000640 1 -- 172.21.15.40:0/2338729702 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3b415f620 con 0x55b3b441d7d0 2024-04-27T00:04:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.859+0000 7fb002000640 1 --2- 172.21.15.40:0/2338729702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 0x55b3b443ad90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7faff000e010 tx=0x7faff0004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda6ddecd5cfd0f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.859+0000 7fafef400640 1 -- 172.21.15.40:0/2338729702 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff0010430 con 0x55b3b441d7d0 2024-04-27T00:04:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.859+0000 7fafef400640 1 -- 172.21.15.40:0/2338729702 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faff00105d0 con 0x55b3b441d7d0 2024-04-27T00:04:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.859+0000 7fafef400640 1 -- 172.21.15.40:0/2338729702 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff002a980 con 0x55b3b441d7d0 2024-04-27T00:04:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.860+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2338729702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 msgr2=0x55b3b443ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.860+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/2338729702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 0x55b3b443ad90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7faff000e010 tx=0x7faff0004500 comp rx=0 tx=0).stop 2024-04-27T00:04:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.860+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2338729702 shutdown_connections 2024-04-27T00:04:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.860+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/2338729702 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b4422940 0x55b3b442e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.860+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/2338729702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441d7d0 0x55b3b443ad90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.860+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/2338729702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441cde0 0x55b3b443a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.860+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2338729702 >> 172.21.15.40:0/2338729702 conn(0x55b3b4418450 msgr2=0x55b3b4418830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.860+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2338729702 shutdown_connections 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.860+0000 7fb00cc2a740 1 -- 172.21.15.40:0/2338729702 wait complete. 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.861+0000 7fb00cc2a740 1 -- start start 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.861+0000 7fb00cc2a740 1 Processor -- start 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.861+0000 7fb00cc2a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441cde0 0x55b3b442b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.861+0000 7fb00cc2a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441d7d0 0x55b3b442bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.861+0000 7fb00cc2a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b4422940 0x55b3b44970b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.861+0000 7fb00cc2a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb0041affe0 con 0x55b3b441cde0 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.861+0000 7fb00cc2a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb0041b1450 con 0x55b3b4422940 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.861+0000 7fb00cc2a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb0041b06b0 con 0x55b3b441d7d0 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.861+0000 7fb00cc2a740 10 client.0 _ll_register_callbacks cb 0x7faff0005c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fb002a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441cde0 0x55b3b442b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fb002a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441cde0 0x55b3b442b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57212/0 (socket says 172.21.15.40:57212) 2024-04-27T00:04:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fb002000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441d7d0 0x55b3b442bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fb002a00640 1 -- 172.21.15.40:0/3688446602 learned_addr learned my addr 172.21.15.40:0/3688446602 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fb003400640 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b4422940 0x55b3b44970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fb002000640 1 -- 172.21.15.40:0/3688446602 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b4422940 msgr2=0x55b3b44970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fb002000640 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b4422940 0x55b3b44970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fb002000640 1 -- 172.21.15.40:0/3688446602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441cde0 msgr2=0x55b3b442b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fb002000640 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441cde0 0x55b3b442b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fb002000640 1 -- 172.21.15.40:0/3688446602 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3b433bcb0 con 0x55b3b441d7d0 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fb002a00640 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441cde0 0x55b3b442b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fb002000640 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441d7d0 0x55b3b442bae0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faff003a040 tx=0x7faff0004190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc10fcea4e588d82 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.862+0000 7fafecc00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.863+0000 7fafecc00640 1 -- 172.21.15.40:0/3688446602 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff0046060 con 0x55b3b441d7d0 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.863+0000 7fb00cc2a740 1 -- 172.21.15.40:0/3688446602 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faff00105d0 con 0x55b3b441d7d0 2024-04-27T00:04:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.863+0000 7fafecc00640 1 -- 172.21.15.40:0/3688446602 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7faff002aa90 con 0x55b3b441d7d0 2024-04-27T00:04:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.863+0000 7fafecc00640 1 -- 172.21.15.40:0/3688446602 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faff004d050 con 0x55b3b441d7d0 2024-04-27T00:04:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.863+0000 7fb00cc2a740 10 client.25585 fetch_fsmap learned FSMap version 24 2024-04-27T00:04:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.863+0000 7fb00cc2a740 1 -- 172.21.15.40:0/3688446602 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b3b44987e0 con 0x55b3b441d7d0 2024-04-27T00:04:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.863+0000 7fafecc00640 1 -- 172.21.15.40:0/3688446602 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faff00105d0 con 0x55b3b441d7d0 2024-04-27T00:04:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.864+0000 7fafecc00640 1 -- 172.21.15.40:0/3688446602 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7faff0028070 con 0x55b3b441d7d0 2024-04-27T00:04:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.864+0000 7fb00cc2a740 10 client.25585 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:04:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.864+0000 7fb00cc2a740 10 client.25585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:04:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.864+0000 7fb00cc2a740 20 client.25585 populate_metadata read hostname 'smithi040' 2024-04-27T00:04:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.864+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b3b4499100 0x55b3b449b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.864+0000 7fb00cc2a740 4 client.25585 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:04:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.864+0000 7fb00cc2a740 1 -- 172.21.15.40:0/3688446602 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b3b4388e00 con 0x55b3b4499100 2024-04-27T00:04:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.864+0000 7fb002a00640 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b3b4499100 0x55b3b449b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.865+0000 7fb002a00640 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b3b4499100 0x55b3b449b520 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fd85bac4b751ddd server_cookie=c3f99576dda43143 in_seq=0 out_seq=0 2024-04-27T00:04:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.865+0000 7fafecc00640 10 client.25585 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.878+0000 7fafecc00640 1 -- 172.21.15.40:0/3688446602 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b3b4388e00 con 0x55b3b4499100 2024-04-27T00:04:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.878+0000 7fafecc00640 10 client.25585 handle_command_reply: tid=0 2024-04-27T00:04:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fb00cc2a740 1 client.25585 shutdown 2024-04-27T00:04:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fb00cc2a740 1 -- 172.21.15.40:0/3688446602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441d7d0 msgr2=0x55b3b442bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441d7d0 0x55b3b442bae0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faff003a040 tx=0x7faff0004190 comp rx=0 tx=0).stop 2024-04-27T00:04:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fb00cc2a740 1 -- 172.21.15.40:0/3688446602 shutdown_connections 2024-04-27T00:04:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b3b4499100 0x55b3b449b520 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b3b4422940 0x55b3b44970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3b441d7d0 0x55b3b442bae0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fb00cc2a740 1 --2- 172.21.15.40:0/3688446602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b3b441cde0 0x55b3b442b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fafecc00640 0 client.25585 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fb00cc2a740 1 -- 172.21.15.40:0/3688446602 >> 172.21.15.40:0/3688446602 conn(0x55b3b4418450 msgr2=0x55b3b4426f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fb00cc2a740 1 -- 172.21.15.40:0/3688446602 shutdown_connections 2024-04-27T00:04:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fb00cc2a740 1 -- 172.21.15.40:0/3688446602 wait complete. 2024-04-27T00:04:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.879+0000 7fb00cc2a740 20 client.25585 trim_cache size 0 max 16384 2024-04-27T00:04:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:04:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:04:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:04:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:04:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.880+0000 7fb003e00640 1 -- 172.21.15.40:0/599834655 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fafe80a3c30 msgr2=0x7fafe80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.880+0000 7fb003e00640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fafe80a3c30 0x7fafe80a60f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55b3b441e030 tx=0x7faff000b390 comp rx=0 tx=0).stop 2024-04-27T00:04:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.880+0000 7fb003e00640 1 -- 172.21.15.40:0/599834655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb00410eda0 msgr2=0x7fb0041a65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.880+0000 7fb003e00640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb00410eda0 0x7fb0041a65f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7faffc00eb10 tx=0x7faffc00c2d0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.880+0000 7fb003e00640 1 -- 172.21.15.40:0/599834655 shutdown_connections 2024-04-27T00:04:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.880+0000 7fb003e00640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fafe80a3c30 0x7fafe80a60f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.880+0000 7fb003e00640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb00410eda0 0x7fb0041a65f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.880+0000 7fb003e00640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb004076970 0x7fb00419f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.880+0000 7fb003e00640 1 --2- 172.21.15.40:0/599834655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb004075390 0x7fb00419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.880+0000 7fb003e00640 1 -- 172.21.15.40:0/599834655 >> 172.21.15.40:0/599834655 conn(0x7fb0040fdd40 msgr2=0x7fb004102140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.881+0000 7fb003e00640 1 -- 172.21.15.40:0/599834655 shutdown_connections 2024-04-27T00:04:00.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:00.881+0000 7fb003e00640 1 -- 172.21.15.40:0/599834655 wait complete. 2024-04-27T00:04:00.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:00 smithi040 ceph-mon[28095]: pgmap v481: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:01.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:00 smithi080 ceph-mon[32362]: pgmap v481: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:01.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:00 smithi103 ceph-mon[32952]: pgmap v481: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:01.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:01 smithi040 ceph-mon[28095]: pgmap v482: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:02.050 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:01 smithi080 ceph-mon[32362]: pgmap v482: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:02.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:01 smithi103 ceph-mon[32952]: pgmap v482: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:04.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:03 smithi040 ceph-mon[28095]: pgmap v483: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-04-27T00:04:04.302 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:03 smithi080 ceph-mon[32362]: pgmap v483: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-04-27T00:04:04.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:03 smithi103 ceph-mon[32952]: pgmap v483: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-04-27T00:04:06.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:05 smithi040 ceph-mon[28095]: pgmap v484: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:06.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:05 smithi080 ceph-mon[32362]: pgmap v484: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:06.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:05 smithi103 ceph-mon[32952]: pgmap v484: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:08.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:07 smithi040 ceph-mon[28095]: pgmap v485: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:08.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:07 smithi080 ceph-mon[32362]: pgmap v485: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:08.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:07 smithi103 ceph-mon[32952]: pgmap v485: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:10.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:09 smithi040 ceph-mon[28095]: pgmap v486: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:04:10.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:09 smithi080 ceph-mon[32362]: pgmap v486: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:04:10.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:09 smithi103 ceph-mon[32952]: pgmap v486: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:04:12.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:11 smithi040 ceph-mon[28095]: pgmap v487: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:04:12.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:11 smithi080 ceph-mon[32362]: pgmap v487: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:04:12.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:11 smithi103 ceph-mon[32952]: pgmap v487: 129 pgs: 129 active+clean; 1.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:04:14.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.029+0000 7efcbd400640 1 -- 172.21.15.40:0/321557751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8106580 msgr2=0x7efcb810ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:14.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.029+0000 7efcbd400640 1 --2- 172.21.15.40:0/321557751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8106580 0x7efcb810ce10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009ca0 tx=0x7efcb002a8b0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.029+0000 7efcbd400640 1 -- 172.21.15.40:0/321557751 shutdown_connections 2024-04-27T00:04:14.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.029+0000 7efcbd400640 1 --2- 172.21.15.40:0/321557751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8106580 0x7efcb810ce10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.029+0000 7efcbd400640 1 --2- 172.21.15.40:0/321557751 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcb8105bc0 0x7efcb8106040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.029+0000 7efcbd400640 1 --2- 172.21.15.40:0/321557751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcb81049d0 0x7efcb8104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.029+0000 7efcbd400640 1 -- 172.21.15.40:0/321557751 >> 172.21.15.40:0/321557751 conn(0x7efcb80fff40 msgr2=0x7efcb8102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:14.032 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.029+0000 7efcbd400640 1 -- 172.21.15.40:0/321557751 shutdown_connections 2024-04-27T00:04:14.032 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.029+0000 7efcbd400640 1 -- 172.21.15.40:0/321557751 wait complete. 2024-04-27T00:04:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.030+0000 7efcbd400640 1 Processor -- start 2024-04-27T00:04:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.030+0000 7efcbd400640 1 -- start start 2024-04-27T00:04:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.031+0000 7efcbd400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcb81049d0 0x7efcb819ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.031+0000 7efcbd400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcb8105bc0 0x7efcb819b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.031+0000 7efcbd400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8106580 0x7efcb81a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.031+0000 7efcbd400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efcb8110530 con 0x7efcb8106580 2024-04-27T00:04:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.031+0000 7efcbd400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81103b0 con 0x7efcb81049d0 2024-04-27T00:04:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.031+0000 7efcbd400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81106b0 con 0x7efcb8105bc0 2024-04-27T00:04:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.031+0000 7efcbca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8106580 0x7efcb81a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.031+0000 7efcbca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8106580 0x7efcb81a2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36752/0 (socket says 172.21.15.40:36752) 2024-04-27T00:04:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.031+0000 7efcb7e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcb81049d0 0x7efcb819ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.031+0000 7efcbca00640 1 -- 172.21.15.40:0/3161389642 learned_addr learned my addr 172.21.15.40:0/3161389642 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.031+0000 7efcb7400640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcb8105bc0 0x7efcb819b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.032+0000 7efcb7e00640 1 -- 172.21.15.40:0/3161389642 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcb8105bc0 msgr2=0x7efcb819b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:14.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.032+0000 7efcb7e00640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcb8105bc0 0x7efcb819b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.032+0000 7efcb7e00640 1 -- 172.21.15.40:0/3161389642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8106580 msgr2=0x7efcb81a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:14.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.032+0000 7efcb7e00640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8106580 0x7efcb81a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.032+0000 7efcb7e00640 1 -- 172.21.15.40:0/3161389642 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb81a27a0 con 0x7efcb81049d0 2024-04-27T00:04:14.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.032+0000 7efcb7e00640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcb81049d0 0x7efcb819ab80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efca800e9c0 tx=0x7efca800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:14.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.032+0000 7efcb4c00640 1 -- 172.21.15.40:0/3161389642 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca800ccc0 con 0x7efcb81049d0 2024-04-27T00:04:14.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.032+0000 7efcbd400640 1 -- 172.21.15.40:0/3161389642 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb81a2a90 con 0x7efcb81049d0 2024-04-27T00:04:14.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.032+0000 7efcb4c00640 1 -- 172.21.15.40:0/3161389642 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efca80035f0 con 0x7efcb81049d0 2024-04-27T00:04:14.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.032+0000 7efcb4c00640 1 -- 172.21.15.40:0/3161389642 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca800ce60 con 0x7efcb81049d0 2024-04-27T00:04:14.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.033+0000 7efcbd400640 1 -- 172.21.15.40:0/3161389642 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb81a3030 con 0x7efcb81049d0 2024-04-27T00:04:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.036+0000 7efcc19bc740 1 Processor -- start 2024-04-27T00:04:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.036+0000 7efcc19bc740 1 -- start start 2024-04-27T00:04:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.036+0000 7efcb4c00640 1 -- 172.21.15.40:0/3161389642 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7efca8004000 con 0x7efcb81049d0 2024-04-27T00:04:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.036+0000 7efcc19bc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a99de0 0x561866a9a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.036+0000 7efcc19bc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9a7d0 0x561866a9f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.036+0000 7efcc19bc740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a9f940 0x561866aa1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.036+0000 7efcc19bc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5618669fe380 con 0x561866a99de0 2024-04-27T00:04:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.036+0000 7efcc19bc740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x56186699ac30 con 0x561866a9f940 2024-04-27T00:04:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.036+0000 7efcc19bc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5618669fa340 con 0x561866a9a7d0 2024-04-27T00:04:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.037+0000 7efcbca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a99de0 0x561866a9a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.037+0000 7efcb7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9a7d0 0x561866a9f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.037+0000 7efcbca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a99de0 0x561866a9a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36758/0 (socket says 172.21.15.40:36758) 2024-04-27T00:04:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.037+0000 7efcbca00640 1 -- 172.21.15.40:0/3462924202 learned_addr learned my addr 172.21.15.40:0/3462924202 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.037+0000 7efcb7400640 1 --2- 172.21.15.40:0/3462924202 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a9f940 0x561866aa1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.037+0000 7efcbca00640 1 -- 172.21.15.40:0/3462924202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9a7d0 msgr2=0x561866a9f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.037+0000 7efcbca00640 1 --2- 172.21.15.40:0/3462924202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9a7d0 0x561866a9f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.037+0000 7efcbca00640 1 -- 172.21.15.40:0/3462924202 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a9f940 msgr2=0x561866aa1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.037+0000 7efcbca00640 1 --2- 172.21.15.40:0/3462924202 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a9f940 0x561866aa1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.037+0000 7efcbca00640 1 -- 172.21.15.40:0/3462924202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618669bc940 con 0x561866a99de0 2024-04-27T00:04:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.038+0000 7efcbca00640 1 --2- 172.21.15.40:0/3462924202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a99de0 0x561866a9a200 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009a50 tx=0x7efcb0005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2305a85084c47ffe server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.038+0000 7efcad200640 1 -- 172.21.15.40:0/3462924202 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0030900 con 0x561866a99de0 2024-04-27T00:04:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.038+0000 7efcad200640 1 -- 172.21.15.40:0/3462924202 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcb0036030 con 0x561866a99de0 2024-04-27T00:04:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.038+0000 7efcad200640 1 -- 172.21.15.40:0/3462924202 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb00044c0 con 0x561866a99de0 2024-04-27T00:04:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.038+0000 7efcb4c00640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7efc9c0a3c30 0x7efc9c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.038+0000 7efcb4c00640 1 -- 172.21.15.40:0/3161389642 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7efca80d0dc0 con 0x7efcb81049d0 2024-04-27T00:04:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.038+0000 7efcb7e00640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7efc9c0a3c30 0x7efc9c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.039+0000 7efcc19bc740 1 -- 172.21.15.40:0/3462924202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a99de0 msgr2=0x561866a9a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.039+0000 7efcc19bc740 1 --2- 172.21.15.40:0/3462924202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a99de0 0x561866a9a200 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009a50 tx=0x7efcb0005ef0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.039+0000 7efcc19bc740 1 -- 172.21.15.40:0/3462924202 shutdown_connections 2024-04-27T00:04:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.039+0000 7efcc19bc740 1 --2- 172.21.15.40:0/3462924202 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a9f940 0x561866aa1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.039+0000 7efcc19bc740 1 --2- 172.21.15.40:0/3462924202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9a7d0 0x561866a9f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.039+0000 7efcc19bc740 1 --2- 172.21.15.40:0/3462924202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a99de0 0x561866a9a200 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.039+0000 7efcc19bc740 1 -- 172.21.15.40:0/3462924202 >> 172.21.15.40:0/3462924202 conn(0x561866a95450 msgr2=0x561866a95850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.039+0000 7efcc19bc740 1 -- 172.21.15.40:0/3462924202 shutdown_connections 2024-04-27T00:04:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.039+0000 7efcc19bc740 1 -- 172.21.15.40:0/3462924202 wait complete. 2024-04-27T00:04:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.039+0000 7efcb7e00640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7efc9c0a3c30 0x7efc9c0a60f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x561866a9abd0 tx=0x7efca80d4870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.039+0000 7efcc19bc740 1 -- start start 2024-04-27T00:04:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.040+0000 7efcc19bc740 1 Processor -- start 2024-04-27T00:04:14.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.040+0000 7efcc19bc740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a99de0 0x561866aa3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.040+0000 7efcc19bc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 0x561866aa44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.040+0000 7efcc19bc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9f940 0x561866aab4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.040+0000 7efcc19bc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561866aa2890 con 0x561866a9a7d0 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.040+0000 7efcc19bc740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x561866985c60 con 0x561866a99de0 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.040+0000 7efcc19bc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561866aa2a10 con 0x561866a9f940 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.040+0000 7efcc19bc740 10 client.0 _ll_register_callbacks cb 0x5618669b1e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efcb7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 0x561866aa44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efcb7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 0x561866aa44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36774/0 (socket says 172.21.15.40:36774) 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efcb7400640 1 -- 172.21.15.40:0/2790661812 learned_addr learned my addr 172.21.15.40:0/2790661812 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efcbca00640 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a99de0 0x561866aa3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efcbca00640 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9f940 0x561866aab4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efcb7400640 1 -- 172.21.15.40:0/2790661812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9f940 msgr2=0x561866aab4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efcb7400640 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9f940 0x561866aab4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efcb7400640 1 -- 172.21.15.40:0/2790661812 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a99de0 msgr2=0x561866aa3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efcb7400640 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a99de0 0x561866aa3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efcb7400640 1 -- 172.21.15.40:0/2790661812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561866994510 con 0x561866a9a7d0 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efcbca00640 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9f940 0x561866aab4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efcb7400640 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 0x561866aa44a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efca4012720 tx=0x7efca4012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2237c6086ea08e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.041+0000 7efca1600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.042+0000 7efca1600640 1 -- 172.21.15.40:0/2790661812 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca400f8c0 con 0x561866a9a7d0 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.042+0000 7efca1600640 1 -- 172.21.15.40:0/2790661812 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7efca400fa60 con 0x561866a9a7d0 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.042+0000 7efcc19bc740 1 -- 172.21.15.40:0/2790661812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcb0036030 con 0x561866a9a7d0 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.042+0000 7efca1600640 1 -- 172.21.15.40:0/2790661812 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efca400fca0 con 0x561866a9a7d0 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.042+0000 7efcc19bc740 10 client.15453 fetch_fsmap learned FSMap version 24 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.042+0000 7efcc19bc740 1 -- 172.21.15.40:0/2790661812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561866710210 con 0x561866a9a7d0 2024-04-27T00:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.042+0000 7efca1600640 1 -- 172.21.15.40:0/2790661812 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcb0036030 con 0x561866a9a7d0 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.043+0000 7efca1600640 1 -- 172.21.15.40:0/2790661812 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7efca4010430 con 0x561866a9a7d0 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.043+0000 7efcc19bc740 10 client.15453 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.043+0000 7efcc19bc740 10 client.15453 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.043+0000 7efcc19bc740 20 client.15453 populate_metadata read hostname 'smithi040' 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.043+0000 7efcc19bc740 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561866aafae0 0x561866ab1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.043+0000 7efcc19bc740 4 client.15453 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.043+0000 7efcc19bc740 1 -- 172.21.15.40:0/2790661812 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561866a0bf90 con 0x561866aafae0 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.043+0000 7efcbca00640 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561866aafae0 0x561866ab1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.044+0000 7efcbca00640 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561866aafae0 0x561866ab1f30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1155039d836cf775 server_cookie=4e5b974204e1d60d in_seq=0 out_seq=0 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.044+0000 7efca1600640 10 client.15453 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.044+0000 7efca1600640 1 -- 172.21.15.40:0/2790661812 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561866a0bf90 con 0x561866aafae0 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.044+0000 7efca1600640 10 client.15453 handle_command_reply: tid=0 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efcc19bc740 1 client.15453 shutdown 2024-04-27T00:04:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efcc19bc740 1 -- 172.21.15.40:0/2790661812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 msgr2=0x561866aa44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efcc19bc740 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 0x561866aa44a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efca4012720 tx=0x7efca4012be0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efcc19bc740 1 -- 172.21.15.40:0/2790661812 shutdown_connections 2024-04-27T00:04:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efcc19bc740 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561866aafae0 0x561866ab1f30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efcc19bc740 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9f940 0x561866aab4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efcc19bc740 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 0x561866aa44a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efcc19bc740 1 --2- 172.21.15.40:0/2790661812 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a99de0 0x561866aa3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efcc19bc740 1 -- 172.21.15.40:0/2790661812 >> 172.21.15.40:0/2790661812 conn(0x561866a95450 msgr2=0x561866a94650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efca1600640 0 client.15453 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efcc19bc740 1 -- 172.21.15.40:0/2790661812 shutdown_connections 2024-04-27T00:04:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efcc19bc740 1 -- 172.21.15.40:0/2790661812 wait complete. 2024-04-27T00:04:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.045+0000 7efcc19bc740 20 client.15453 trim_cache size 0 max 16384 2024-04-27T00:04:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.062+0000 7efcc19bc740 1 Processor -- start 2024-04-27T00:04:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.062+0000 7efcc19bc740 1 -- start start 2024-04-27T00:04:14.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.062+0000 7efcc19bc740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a99de0 0x561866a95830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.062+0000 7efcc19bc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9a7d0 0x561866aab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.062+0000 7efcc19bc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9f940 0x561866a9b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.062+0000 7efcc19bc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81afce0 con 0x561866a9f940 2024-04-27T00:04:14.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.062+0000 7efcc19bc740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81ad5b0 con 0x561866a99de0 2024-04-27T00:04:14.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.062+0000 7efcc19bc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81ab400 con 0x561866a9a7d0 2024-04-27T00:04:14.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.062+0000 7efcbca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9f940 0x561866a9b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.063+0000 7efcb7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9a7d0 0x561866aab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.063+0000 7efcbca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9f940 0x561866a9b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36780/0 (socket says 172.21.15.40:36780) 2024-04-27T00:04:14.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.063+0000 7efcbca00640 1 -- 172.21.15.40:0/177005488 learned_addr learned my addr 172.21.15.40:0/177005488 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:14.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.063+0000 7efcbca00640 1 --2- 172.21.15.40:0/177005488 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a99de0 0x561866a95830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.063+0000 7efcb7400640 1 -- 172.21.15.40:0/177005488 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a99de0 msgr2=0x561866a95830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.063+0000 7efcb7400640 1 --2- 172.21.15.40:0/177005488 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a99de0 0x561866a95830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.063+0000 7efcb7400640 1 -- 172.21.15.40:0/177005488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9f940 msgr2=0x561866a9b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.063+0000 7efcb7400640 1 --2- 172.21.15.40:0/177005488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9f940 0x561866a9b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.063+0000 7efcb7400640 1 -- 172.21.15.40:0/177005488 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618667dc620 con 0x561866a9a7d0 2024-04-27T00:04:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.063+0000 7efcbca00640 1 --2- 172.21.15.40:0/177005488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9f940 0x561866a9b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:04:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.063+0000 7efcb7400640 1 --2- 172.21.15.40:0/177005488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9a7d0 0x561866aab4a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efca40126f0 tx=0x7efca4011ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f8235845d06f9d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:14.066 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.064+0000 7efca3e00640 1 -- 172.21.15.40:0/177005488 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca4010b10 con 0x561866a9a7d0 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.064+0000 7efca3e00640 1 -- 172.21.15.40:0/177005488 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efca4010430 con 0x561866a9a7d0 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.064+0000 7efca3e00640 1 -- 172.21.15.40:0/177005488 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca4005650 con 0x561866a9a7d0 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.064+0000 7efcc19bc740 1 -- 172.21.15.40:0/177005488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9a7d0 msgr2=0x561866aab4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.064+0000 7efcc19bc740 1 --2- 172.21.15.40:0/177005488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9a7d0 0x561866aab4a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efca40126f0 tx=0x7efca4011ef0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.065+0000 7efcc19bc740 1 -- 172.21.15.40:0/177005488 shutdown_connections 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.065+0000 7efcc19bc740 1 --2- 172.21.15.40:0/177005488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9f940 0x561866a9b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.065+0000 7efcc19bc740 1 --2- 172.21.15.40:0/177005488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a9a7d0 0x561866aab4a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.065+0000 7efcc19bc740 1 --2- 172.21.15.40:0/177005488 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a99de0 0x561866a95830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.065+0000 7efcc19bc740 1 -- 172.21.15.40:0/177005488 >> 172.21.15.40:0/177005488 conn(0x561866a95450 msgr2=0x561866a9efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.065+0000 7efcc19bc740 1 -- 172.21.15.40:0/177005488 shutdown_connections 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.065+0000 7efcc19bc740 1 -- 172.21.15.40:0/177005488 wait complete. 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.065+0000 7efcc19bc740 1 -- start start 2024-04-27T00:04:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.065+0000 7efcc19bc740 1 Processor -- start 2024-04-27T00:04:14.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.066+0000 7efcc19bc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a99de0 0x561866aa84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.066+0000 7efcc19bc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 0x561866aa89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.066+0000 7efcc19bc740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a9f940 0x561866b140a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.066+0000 7efcc19bc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81aba70 con 0x561866a9a7d0 2024-04-27T00:04:14.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.066+0000 7efcc19bc740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81acee0 con 0x561866a9f940 2024-04-27T00:04:14.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.066+0000 7efcc19bc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81ac140 con 0x561866a99de0 2024-04-27T00:04:14.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.066+0000 7efcc19bc740 10 client.0 _ll_register_callbacks cb 0x7efca4009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:04:14.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.066+0000 7efcb7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 0x561866aa89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.066+0000 7efcb7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 0x561866aa89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36782/0 (socket says 172.21.15.40:36782) 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.066+0000 7efcb7400640 1 -- 172.21.15.40:0/2651016744 learned_addr learned my addr 172.21.15.40:0/2651016744 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.066+0000 7efcbca00640 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a99de0 0x561866aa84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.066+0000 7efcbca00640 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a9f940 0x561866b140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.067+0000 7efcb7400640 1 -- 172.21.15.40:0/2651016744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a99de0 msgr2=0x561866aa84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.067+0000 7efcb7400640 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a99de0 0x561866aa84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.067+0000 7efcb7400640 1 -- 172.21.15.40:0/2651016744 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a9f940 msgr2=0x561866b140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.067+0000 7efcb7400640 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a9f940 0x561866b140a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.067+0000 7efcb7400640 1 -- 172.21.15.40:0/2651016744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618669b8cb0 con 0x561866a9a7d0 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.067+0000 7efcbca00640 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a9f940 0x561866b140a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.067+0000 7efcb7400640 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 0x561866aa89e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efca4010a10 tx=0x7efca40059b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d5864ee973a745 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.067+0000 7efca1600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:04:14.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.067+0000 7efca1600640 1 -- 172.21.15.40:0/2651016744 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca402d030 con 0x561866a9a7d0 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.067+0000 7efca1600640 1 -- 172.21.15.40:0/2651016744 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7efca4018070 con 0x561866a9a7d0 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.067+0000 7efcc19bc740 1 -- 172.21.15.40:0/2651016744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efca4010430 con 0x561866a9a7d0 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.068+0000 7efca1600640 1 -- 172.21.15.40:0/2651016744 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efca4037050 con 0x561866a9a7d0 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.068+0000 7efcc19bc740 10 client.15459 fetch_fsmap learned FSMap version 24 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.068+0000 7efcc19bc740 1 -- 172.21.15.40:0/2651016744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561866b15680 con 0x561866a9a7d0 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.068+0000 7efca1600640 1 -- 172.21.15.40:0/2651016744 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efca4010430 con 0x561866a9a7d0 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.068+0000 7efca1600640 1 -- 172.21.15.40:0/2651016744 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7efca40036f0 con 0x561866a9a7d0 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.068+0000 7efcc19bc740 10 client.15459 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.068+0000 7efcc19bc740 10 client.15459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.068+0000 7efcc19bc740 20 client.15459 populate_metadata read hostname 'smithi040' 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.068+0000 7efcc19bc740 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561866b16050 0x561866b18470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.068+0000 7efcc19bc740 4 client.15459 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.068+0000 7efcc19bc740 1 -- 172.21.15.40:0/2651016744 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561866a05e00 con 0x561866b16050 2024-04-27T00:04:14.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.069+0000 7efcbca00640 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561866b16050 0x561866b18470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.069+0000 7efcbca00640 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561866b16050 0x561866b18470 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=879e722148363731 server_cookie=12160ad901ec81ee in_seq=0 out_seq=0 2024-04-27T00:04:14.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.069+0000 7efca1600640 10 client.15459 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.083+0000 7efca1600640 1 -- 172.21.15.40:0/2651016744 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561866a05e00 con 0x561866b16050 2024-04-27T00:04:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.083+0000 7efca1600640 10 client.15459 handle_command_reply: tid=0 2024-04-27T00:04:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.083+0000 7efcc19bc740 1 client.15459 shutdown 2024-04-27T00:04:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.084+0000 7efcc19bc740 1 -- 172.21.15.40:0/2651016744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 msgr2=0x561866aa89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.084+0000 7efcc19bc740 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 0x561866aa89e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efca4010a10 tx=0x7efca40059b0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.084+0000 7efcc19bc740 1 -- 172.21.15.40:0/2651016744 shutdown_connections 2024-04-27T00:04:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.084+0000 7efcc19bc740 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561866b16050 0x561866b18470 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.084+0000 7efcc19bc740 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561866a9f940 0x561866b140a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.084+0000 7efcc19bc740 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561866a9a7d0 0x561866aa89e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.084+0000 7efcc19bc740 1 --2- 172.21.15.40:0/2651016744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561866a99de0 0x561866aa84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.084+0000 7efcc19bc740 1 -- 172.21.15.40:0/2651016744 >> 172.21.15.40:0/2651016744 conn(0x561866a95450 msgr2=0x561866a9eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.084+0000 7efca1600640 0 client.15459 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.084+0000 7efcc19bc740 1 -- 172.21.15.40:0/2651016744 shutdown_connections 2024-04-27T00:04:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.084+0000 7efcc19bc740 1 -- 172.21.15.40:0/2651016744 wait complete. 2024-04-27T00:04:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.084+0000 7efcc19bc740 20 client.15459 trim_cache size 0 max 16384 2024-04-27T00:04:14.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:04:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:04:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:04:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:04:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.085+0000 7efcbd400640 1 -- 172.21.15.40:0/3161389642 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7efc9c0a3c30 msgr2=0x7efc9c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.085+0000 7efcbd400640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7efc9c0a3c30 0x7efc9c0a60f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x561866a9abd0 tx=0x7efca80d4870 comp rx=0 tx=0).stop 2024-04-27T00:04:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.085+0000 7efcbd400640 1 -- 172.21.15.40:0/3161389642 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcb81049d0 msgr2=0x7efcb819ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:14.087 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.085+0000 7efcbd400640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcb81049d0 0x7efcb819ab80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efca800e9c0 tx=0x7efca800ee80 comp rx=0 tx=0).stop 2024-04-27T00:04:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.085+0000 7efcbd400640 1 -- 172.21.15.40:0/3161389642 shutdown_connections 2024-04-27T00:04:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.085+0000 7efcbd400640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb8106580 0x7efcb81a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.085+0000 7efcbd400640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7efc9c0a3c30 0x7efc9c0a60f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.085+0000 7efcbd400640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcb8105bc0 0x7efcb819b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.085+0000 7efcbd400640 1 --2- 172.21.15.40:0/3161389642 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7efcb81049d0 0x7efcb819ab80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.085+0000 7efcbd400640 1 -- 172.21.15.40:0/3161389642 >> 172.21.15.40:0/3161389642 conn(0x7efcb80fff40 msgr2=0x7efcb8108ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.086+0000 7efcbd400640 1 -- 172.21.15.40:0/3161389642 shutdown_connections 2024-04-27T00:04:14.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:14.086+0000 7efcbd400640 1 -- 172.21.15.40:0/3161389642 wait complete. 2024-04-27T00:04:14.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:13 smithi040 ceph-mon[28095]: pgmap v488: 129 pgs: 129 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:14.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:13 smithi080 ceph-mon[32362]: pgmap v488: 129 pgs: 129 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:14.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:13 smithi103 ceph-mon[32952]: pgmap v488: 129 pgs: 129 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:16.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:15 smithi040 ceph-mon[28095]: pgmap v489: 129 pgs: 129 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:04:16.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:15 smithi080 ceph-mon[32362]: pgmap v489: 129 pgs: 129 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:04:16.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:15 smithi103 ceph-mon[32952]: pgmap v489: 129 pgs: 129 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:04:18.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:17 smithi040 ceph-mon[28095]: pgmap v490: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:18.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:17 smithi080 ceph-mon[32362]: pgmap v490: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:18.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:17 smithi103 ceph-mon[32952]: pgmap v490: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:20.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:19 smithi040 ceph-mon[28095]: pgmap v491: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:20.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:19 smithi080 ceph-mon[32362]: pgmap v491: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:20.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:19 smithi103 ceph-mon[32952]: pgmap v491: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:22.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:21 smithi040 ceph-mon[28095]: pgmap v492: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-27T00:04:22.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:21 smithi080 ceph-mon[32362]: pgmap v492: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-27T00:04:22.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:21 smithi103 ceph-mon[32952]: pgmap v492: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-27T00:04:23.931 INFO:tasks.workunit.client.0.smithi040.stdout:1/600: open fd = 3 2024-04-27T00:04:23.931 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.931 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:2/600: open fd = 3 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:3/600: open fd = 3 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:4/600: open fd = 3 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.932 INFO:tasks.workunit.client.0.smithi040.stdout:5/600: open fd = 3 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:6/600: open fd = 3 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:7/600: open fd = 3 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:8/600: open fd = 3 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.933 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:9/600: open fd = 3 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:10/600: open fd = 3 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:11/600: open fd = 3 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:12/600: open fd = 3 2024-04-27T00:04:23.934 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.935 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.935 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.935 INFO:tasks.workunit.client.0.smithi040.stdout:13/600: open fd = 3 2024-04-27T00:04:23.935 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.935 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.935 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.935 INFO:tasks.workunit.client.0.smithi040.stdout:14/600: open fd = 3 2024-04-27T00:04:23.935 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.935 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.935 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.935 INFO:tasks.workunit.client.0.smithi040.stdout:15/600: open fd = 3 2024-04-27T00:04:23.935 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:16/600: open fd = 3 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:17/600: open fd = 3 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:18/600: open fd = 3 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.936 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:19/600: open fd = 3 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:20/600: open fd = 3 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:21/600: open fd = 3 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:22/600: open fd = 3 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.937 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:23/600: open fd = 3 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:24/600: open fd = 3 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:25/600: open fd = 3 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.938 INFO:tasks.workunit.client.0.smithi040.stdout:26/600: open fd = 3 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:27/600: open fd = 3 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:28/600: open fd = 3 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:29/600: open fd = 3 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.939 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:30/600: open fd = 3 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:31/600: open fd = 3 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:32/600: open fd = 3 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:33/600: open fd = 3 2024-04-27T00:04:23.940 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:34/600: open fd = 3 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:35/600: open fd = 3 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:36/600: open fd = 3 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.941 INFO:tasks.workunit.client.0.smithi040.stdout:37/600: open fd = 3 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:38/600: open fd = 3 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:39/600: open fd = 3 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:40/600: open fd = 3 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.942 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:41/600: open fd = 3 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:42/600: open fd = 3 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:43/600: open fd = 3 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:44/600: open fd = 3 2024-04-27T00:04:23.943 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:45/600: open fd = 3 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:46/600: open fd = 3 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:47/600: open fd = 3 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.944 INFO:tasks.workunit.client.0.smithi040.stdout:48/600: open fd = 3 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:49/600: open fd = 3 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:50/600: open fd = 3 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:51/600: open fd = 3 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.945 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:52/600: open fd = 3 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:53/600: open fd = 3 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:54/600: open fd = 3 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:55/600: open fd = 3 2024-04-27T00:04:23.946 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:56/600: open fd = 3 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:57/600: open fd = 3 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:58/600: open fd = 3 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.947 INFO:tasks.workunit.client.0.smithi040.stdout:59/600: open fd = 3 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:60/600: open fd = 3 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:61/600: open fd = 3 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:62/600: open fd = 3 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.948 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:23.949 INFO:tasks.workunit.client.0.smithi040.stdout:63/600: open fd = 3 2024-04-27T00:04:23.949 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.949 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:04:23.949 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:04:24.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:23 smithi040 ceph-mon[28095]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:24.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:23 smithi080 ceph-mon[32362]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:24.348 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:23 smithi103 ceph-mon[32952]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:26.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:25 smithi040 ceph-mon[28095]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:26.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:25 smithi080 ceph-mon[32362]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:26.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:25 smithi103 ceph-mon[32952]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.242+0000 7faecd200640 1 -- 172.21.15.40:0/2746680412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faec80772f0 msgr2=0x7faec8077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.242+0000 7faecd200640 1 --2- 172.21.15.40:0/2746680412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faec80772f0 0x7faec8077710 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faeb8009900 tx=0x7faeb802a3f0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.242+0000 7faecd200640 1 -- 172.21.15.40:0/2746680412 shutdown_connections 2024-04-27T00:04:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.242+0000 7faecd200640 1 --2- 172.21.15.40:0/2746680412 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faec8078ed0 0x7faec8079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.242+0000 7faecd200640 1 --2- 172.21.15.40:0/2746680412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faec8078530 0x7faec8078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.242+0000 7faecd200640 1 --2- 172.21.15.40:0/2746680412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faec80772f0 0x7faec8077710 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.242+0000 7faecd200640 1 -- 172.21.15.40:0/2746680412 >> 172.21.15.40:0/2746680412 conn(0x7faec80fff60 msgr2=0x7faec81023a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.242+0000 7faecd200640 1 -- 172.21.15.40:0/2746680412 shutdown_connections 2024-04-27T00:04:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.242+0000 7faecd200640 1 -- 172.21.15.40:0/2746680412 wait complete. 2024-04-27T00:04:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.243+0000 7faecd200640 1 Processor -- start 2024-04-27T00:04:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.243+0000 7faecd200640 1 -- start start 2024-04-27T00:04:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.244+0000 7faecd200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faec80772f0 0x7faec819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.244+0000 7faecd200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faec8078530 0x7faec819f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.244+0000 7faecd200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faec8078ed0 0x7faec81a65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.244+0000 7faecd200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faec8114970 con 0x7faec8078530 2024-04-27T00:04:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.244+0000 7faecd200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7faec81147f0 con 0x7faec8078ed0 2024-04-27T00:04:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.244+0000 7faecd200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faec8114af0 con 0x7faec80772f0 2024-04-27T00:04:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.244+0000 7faec7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faec80772f0 0x7faec819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.244+0000 7faec7e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faec8078ed0 0x7faec81a65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.244+0000 7faebfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faec8078530 0x7faec819f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.244+0000 7faec7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faec80772f0 0x7faec819eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43734/0 (socket says 172.21.15.40:43734) 2024-04-27T00:04:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.244+0000 7faec7400640 1 -- 172.21.15.40:0/1196270925 learned_addr learned my addr 172.21.15.40:0/1196270925 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.244+0000 7faec7e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faec8078ed0 0x7faec81a65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:33528/0 (socket says 172.21.15.40:33528) 2024-04-27T00:04:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.245+0000 7faec7e00640 1 -- 172.21.15.40:0/1196270925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faec80772f0 msgr2=0x7faec819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.245+0000 7faec7e00640 1 --2- 172.21.15.40:0/1196270925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faec80772f0 0x7faec819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.245+0000 7faec7e00640 1 -- 172.21.15.40:0/1196270925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faec8078530 msgr2=0x7faec819f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.245+0000 7faec7e00640 1 --2- 172.21.15.40:0/1196270925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faec8078530 0x7faec819f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.245+0000 7faec7e00640 1 -- 172.21.15.40:0/1196270925 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec81a6c10 con 0x7faec8078ed0 2024-04-27T00:04:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.245+0000 7faec7e00640 1 --2- 172.21.15.40:0/1196270925 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faec8078ed0 0x7faec81a65a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faec0009fd0 tx=0x7faec000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.245+0000 7faec4c00640 1 -- 172.21.15.40:0/1196270925 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec000ce20 con 0x7faec8078ed0 2024-04-27T00:04:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.245+0000 7faec4c00640 1 -- 172.21.15.40:0/1196270925 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faec0005020 con 0x7faec8078ed0 2024-04-27T00:04:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.245+0000 7faecd200640 1 -- 172.21.15.40:0/1196270925 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faec81a6f00 con 0x7faec8078ed0 2024-04-27T00:04:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.245+0000 7faec4c00640 1 -- 172.21.15.40:0/1196270925 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec0010690 con 0x7faec8078ed0 2024-04-27T00:04:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.246+0000 7faecd200640 1 -- 172.21.15.40:0/1196270925 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faec81a7370 con 0x7faec8078ed0 2024-04-27T00:04:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.249+0000 7faec4c00640 1 -- 172.21.15.40:0/1196270925 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faec00054d0 con 0x7faec8078ed0 2024-04-27T00:04:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.249+0000 7faed15a7740 1 Processor -- start 2024-04-27T00:04:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.249+0000 7faed15a7740 1 -- start start 2024-04-27T00:04:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.249+0000 7faed15a7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c2de0 0x55a9846c3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.249+0000 7faed15a7740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c37d0 0x55a9846c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faed15a7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c8940 0x55a9846cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faed15a7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a984627380 con 0x55a9846c2de0 2024-04-27T00:04:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faed15a7740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55a9845c3c30 con 0x55a9846c37d0 2024-04-27T00:04:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faed15a7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a984623340 con 0x55a9846c8940 2024-04-27T00:04:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faebfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c2de0 0x55a9846c3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faebfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c2de0 0x55a9846c3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41920/0 (socket says 172.21.15.40:41920) 2024-04-27T00:04:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faec7400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c37d0 0x55a9846c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faebfe00640 1 -- 172.21.15.40:0/2567457838 learned_addr learned my addr 172.21.15.40:0/2567457838 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faebfe00640 1 --2- 172.21.15.40:0/2567457838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c8940 0x55a9846cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.262 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faec7400640 1 -- 172.21.15.40:0/2567457838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c8940 msgr2=0x55a9846cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faec7400640 1 --2- 172.21.15.40:0/2567457838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c8940 0x55a9846cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faec7400640 1 -- 172.21.15.40:0/2567457838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c2de0 msgr2=0x55a9846c3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faec7400640 1 --2- 172.21.15.40:0/2567457838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c2de0 0x55a9846c3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faec7400640 1 -- 172.21.15.40:0/2567457838 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9845e5940 con 0x55a9846c37d0 2024-04-27T00:04:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.250+0000 7faebfe00640 1 --2- 172.21.15.40:0/2567457838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c8940 0x55a9846cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:04:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.251+0000 7faec4c00640 1 --2- 172.21.15.40:0/1196270925 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7faeac0a3c30 0x7faeac0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.251+0000 7faec7400640 1 --2- 172.21.15.40:0/2567457838 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c37d0 0x55a9846c82d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7faeb8004770 tx=0x7faeb8004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e59733fabd6884a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.251+0000 7faec4c00640 1 -- 172.21.15.40:0/1196270925 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7faec00d0ff0 con 0x7faec8078ed0 2024-04-27T00:04:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.251+0000 7faebcc00640 1 -- 172.21.15.40:0/2567457838 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb8035030 con 0x55a9846c37d0 2024-04-27T00:04:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.251+0000 7faebfe00640 1 --2- 172.21.15.40:0/1196270925 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7faeac0a3c30 0x7faeac0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.251+0000 7faebcc00640 1 -- 172.21.15.40:0/2567457838 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeb8040070 con 0x55a9846c37d0 2024-04-27T00:04:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.251+0000 7faebcc00640 1 -- 172.21.15.40:0/2567457838 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb8031940 con 0x55a9846c37d0 2024-04-27T00:04:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.252+0000 7faed15a7740 1 -- 172.21.15.40:0/2567457838 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c37d0 msgr2=0x55a9846c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.252+0000 7faed15a7740 1 --2- 172.21.15.40:0/2567457838 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c37d0 0x55a9846c82d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7faeb8004770 tx=0x7faeb8004040 comp rx=0 tx=0).stop 2024-04-27T00:04:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.252+0000 7faebfe00640 1 --2- 172.21.15.40:0/1196270925 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7faeac0a3c30 0x7faeac0a60f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009fd0 tx=0x7faeb0015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.252+0000 7faed15a7740 1 -- 172.21.15.40:0/2567457838 shutdown_connections 2024-04-27T00:04:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.252+0000 7faed15a7740 1 --2- 172.21.15.40:0/2567457838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c8940 0x55a9846cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.252+0000 7faed15a7740 1 --2- 172.21.15.40:0/2567457838 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c37d0 0x55a9846c82d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.252+0000 7faed15a7740 1 --2- 172.21.15.40:0/2567457838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c2de0 0x55a9846c3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.252+0000 7faed15a7740 1 -- 172.21.15.40:0/2567457838 >> 172.21.15.40:0/2567457838 conn(0x55a9846be450 msgr2=0x55a9846be850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.252+0000 7faed15a7740 1 -- 172.21.15.40:0/2567457838 shutdown_connections 2024-04-27T00:04:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.252+0000 7faed15a7740 1 -- 172.21.15.40:0/2567457838 wait complete. 2024-04-27T00:04:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.252+0000 7faed15a7740 1 -- start start 2024-04-27T00:04:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faed15a7740 1 Processor -- start 2024-04-27T00:04:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faed15a7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c2de0 0x55a9846cd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faed15a7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 0x55a9846cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faed15a7740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c8940 0x55a9846d4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faed15a7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a9846cb890 con 0x55a9846c37d0 2024-04-27T00:04:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faed15a7740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55a9845aec60 con 0x55a9846c8940 2024-04-27T00:04:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faed15a7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a9846cba10 con 0x55a9846c2de0 2024-04-27T00:04:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faed15a7740 10 client.0 _ll_register_callbacks cb 0x55a9845dae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:04:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faebfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 0x55a9846cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faec7e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c8940 0x55a9846d4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faebfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 0x55a9846cd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41932/0 (socket says 172.21.15.40:41932) 2024-04-27T00:04:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faebfe00640 1 -- 172.21.15.40:0/1423869913 learned_addr learned my addr 172.21.15.40:0/1423869913 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.253+0000 7faec7400640 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c2de0 0x55a9846cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.254+0000 7faec7e00640 1 -- 172.21.15.40:0/1423869913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c2de0 msgr2=0x55a9846cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.254+0000 7faec7e00640 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c2de0 0x55a9846cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.254+0000 7faec7e00640 1 -- 172.21.15.40:0/1423869913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 msgr2=0x55a9846cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.254+0000 7faec7e00640 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 0x55a9846cd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.254+0000 7faec7e00640 1 -- 172.21.15.40:0/1423869913 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9845bd510 con 0x55a9846c8940 2024-04-27T00:04:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.254+0000 7faebfe00640 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 0x55a9846cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:04:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.254+0000 7faec7e00640 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c8940 0x55a9846d4590 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faec00ce600 tx=0x7faec009cf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58c8d2aaeddbc819 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.254+0000 7faeb5600640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:04:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.254+0000 7faeb5600640 1 -- 172.21.15.40:0/1423869913 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec00a57b0 con 0x55a9846c8940 2024-04-27T00:04:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.254+0000 7faed15a7740 1 -- 172.21.15.40:0/1423869913 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faeb8040070 con 0x55a9846c8940 2024-04-27T00:04:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.254+0000 7faeb5600640 1 -- 172.21.15.40:0/1423869913 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7faec009c220 con 0x55a9846c8940 2024-04-27T00:04:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.255+0000 7faeb5600640 1 -- 172.21.15.40:0/1423869913 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faec00ce910 con 0x55a9846c8940 2024-04-27T00:04:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.255+0000 7faed15a7740 10 client.25589 fetch_fsmap learned FSMap version 24 2024-04-27T00:04:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.255+0000 7faed15a7740 1 -- 172.21.15.40:0/1423869913 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a984339210 con 0x55a9846c8940 2024-04-27T00:04:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.255+0000 7faeb5600640 1 -- 172.21.15.40:0/1423869913 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeb8040070 con 0x55a9846c8940 2024-04-27T00:04:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.255+0000 7faeb5600640 1 -- 172.21.15.40:0/1423869913 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7faec009bd20 con 0x55a9846c8940 2024-04-27T00:04:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.255+0000 7faed15a7740 10 client.25589 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:04:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.255+0000 7faed15a7740 10 client.25589 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:04:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.255+0000 7faed15a7740 20 client.25589 populate_metadata read hostname 'smithi040' 2024-04-27T00:04:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.255+0000 7faed15a7740 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a9846d8aa0 0x55a9846daed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.255+0000 7faed15a7740 4 client.25589 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:04:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.255+0000 7faed15a7740 1 -- 172.21.15.40:0/1423869913 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a984634f90 con 0x55a9846d8aa0 2024-04-27T00:04:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.256+0000 7faec7400640 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a9846d8aa0 0x55a9846daed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.256+0000 7faec7400640 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a9846d8aa0 0x55a9846daed0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65bdb9917ecec92a server_cookie=c31e506a69ecd545 in_seq=0 out_seq=0 2024-04-27T00:04:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.256+0000 7faeb5600640 10 client.25589 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faeb5600640 1 -- 172.21.15.40:0/1423869913 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a984634f90 con 0x55a9846d8aa0 2024-04-27T00:04:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faeb5600640 10 client.25589 handle_command_reply: tid=0 2024-04-27T00:04:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faed15a7740 1 client.25589 shutdown 2024-04-27T00:04:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faed15a7740 1 -- 172.21.15.40:0/1423869913 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c8940 msgr2=0x55a9846d4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faed15a7740 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c8940 0x55a9846d4590 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faec00ce600 tx=0x7faec009cf10 comp rx=0 tx=0).stop 2024-04-27T00:04:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faed15a7740 1 -- 172.21.15.40:0/1423869913 shutdown_connections 2024-04-27T00:04:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faed15a7740 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a9846d8aa0 0x55a9846daed0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faed15a7740 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c8940 0x55a9846d4590 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faeb5600640 0 client.25589 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faed15a7740 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 0x55a9846cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faed15a7740 1 --2- 172.21.15.40:0/1423869913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c2de0 0x55a9846cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faed15a7740 1 -- 172.21.15.40:0/1423869913 >> 172.21.15.40:0/1423869913 conn(0x55a9846be450 msgr2=0x55a9846bd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faed15a7740 1 -- 172.21.15.40:0/1423869913 shutdown_connections 2024-04-27T00:04:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.257+0000 7faed15a7740 1 -- 172.21.15.40:0/1423869913 wait complete. 2024-04-27T00:04:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.258+0000 7faed15a7740 20 client.25589 trim_cache size 0 max 16384 2024-04-27T00:04:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.274+0000 7faed15a7740 1 Processor -- start 2024-04-27T00:04:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.274+0000 7faed15a7740 1 -- start start 2024-04-27T00:04:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.274+0000 7faed15a7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846be450 0x55a9846c14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.274+0000 7faed15a7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c2de0 0x55a9846d4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.274+0000 7faed15a7740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c37d0 0x55a9846c1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.274+0000 7faed15a7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faec81b40e0 con 0x55a9846c2de0 2024-04-27T00:04:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.274+0000 7faed15a7740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7faec81b19b0 con 0x55a9846c37d0 2024-04-27T00:04:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.274+0000 7faed15a7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faec81af800 con 0x55a9846be450 2024-04-27T00:04:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.275+0000 7faec7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846be450 0x55a9846c14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.275+0000 7faec7e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c37d0 0x55a9846c1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.275+0000 7faec7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846be450 0x55a9846c14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43756/0 (socket says 172.21.15.40:43756) 2024-04-27T00:04:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.275+0000 7faec7400640 1 -- 172.21.15.40:0/3860540851 learned_addr learned my addr 172.21.15.40:0/3860540851 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.275+0000 7faebfe00640 1 --2- 172.21.15.40:0/3860540851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c2de0 0x55a9846d4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.275+0000 7faec7400640 1 -- 172.21.15.40:0/3860540851 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c37d0 msgr2=0x55a9846c1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.275+0000 7faec7400640 1 --2- 172.21.15.40:0/3860540851 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c37d0 0x55a9846c1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.275+0000 7faec7400640 1 -- 172.21.15.40:0/3860540851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c2de0 msgr2=0x55a9846d4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.275+0000 7faec7400640 1 --2- 172.21.15.40:0/3860540851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c2de0 0x55a9846d4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.275+0000 7faec7400640 1 -- 172.21.15.40:0/3860540851 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a984405620 con 0x55a9846be450 2024-04-27T00:04:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.275+0000 7faebfe00640 1 --2- 172.21.15.40:0/3860540851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c2de0 0x55a9846d4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:04:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.276+0000 7faec7400640 1 --2- 172.21.15.40:0/3860540851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846be450 0x55a9846c14e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faeb8006010 tx=0x7faeb8047040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bfff70033641d06 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:27.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.276+0000 7faeb7e00640 1 -- 172.21.15.40:0/3860540851 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb8035030 con 0x55a9846be450 2024-04-27T00:04:27.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.276+0000 7faeb7e00640 1 -- 172.21.15.40:0/3860540851 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeb8040070 con 0x55a9846be450 2024-04-27T00:04:27.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.276+0000 7faeb7e00640 1 -- 172.21.15.40:0/3860540851 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb80315a0 con 0x55a9846be450 2024-04-27T00:04:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.277+0000 7faed15a7740 1 -- 172.21.15.40:0/3860540851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846be450 msgr2=0x55a9846c14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.277+0000 7faed15a7740 1 --2- 172.21.15.40:0/3860540851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846be450 0x55a9846c14e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faeb8006010 tx=0x7faeb8047040 comp rx=0 tx=0).stop 2024-04-27T00:04:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.277+0000 7faed15a7740 1 -- 172.21.15.40:0/3860540851 shutdown_connections 2024-04-27T00:04:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.277+0000 7faed15a7740 1 --2- 172.21.15.40:0/3860540851 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846c37d0 0x55a9846c1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.277+0000 7faed15a7740 1 --2- 172.21.15.40:0/3860540851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c2de0 0x55a9846d4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.277+0000 7faed15a7740 1 --2- 172.21.15.40:0/3860540851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846be450 0x55a9846c14e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.277+0000 7faed15a7740 1 -- 172.21.15.40:0/3860540851 >> 172.21.15.40:0/3860540851 conn(0x55a9846c8440 msgr2=0x55a9846c4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.277+0000 7faed15a7740 1 -- 172.21.15.40:0/3860540851 shutdown_connections 2024-04-27T00:04:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.277+0000 7faed15a7740 1 -- 172.21.15.40:0/3860540851 wait complete. 2024-04-27T00:04:27.282 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.277+0000 7faed15a7740 1 -- start start 2024-04-27T00:04:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.277+0000 7faed15a7740 1 Processor -- start 2024-04-27T00:04:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faed15a7740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846be450 0x55a9846d1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faed15a7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c2de0 0x55a9846d21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faed15a7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 0x55a9846d31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faed15a7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faec81afe70 con 0x55a9846c37d0 2024-04-27T00:04:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faed15a7740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7faec81b12e0 con 0x55a9846be450 2024-04-27T00:04:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faed15a7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faec81b0540 con 0x55a9846c2de0 2024-04-27T00:04:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faed15a7740 10 client.0 _ll_register_callbacks cb 0x7faec0099070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:04:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faec7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 0x55a9846d31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faebfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c2de0 0x55a9846d21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faebfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c2de0 0x55a9846d21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43758/0 (socket says 172.21.15.40:43758) 2024-04-27T00:04:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faebfe00640 1 -- 172.21.15.40:0/1121201612 learned_addr learned my addr 172.21.15.40:0/1121201612 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faec7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 0x55a9846d31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41946/0 (socket says 172.21.15.40:41946) 2024-04-27T00:04:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faec7400640 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846be450 0x55a9846d1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faec7e00640 1 -- 172.21.15.40:0/1121201612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c2de0 msgr2=0x55a9846d21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faec7e00640 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c2de0 0x55a9846d21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.278+0000 7faec7e00640 1 -- 172.21.15.40:0/1121201612 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846be450 msgr2=0x55a9846d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.279+0000 7faec7e00640 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846be450 0x55a9846d1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.279+0000 7faec7e00640 1 -- 172.21.15.40:0/1121201612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9845e1cb0 con 0x55a9846c37d0 2024-04-27T00:04:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.279+0000 7faec7400640 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846be450 0x55a9846d1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:04:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.279+0000 7faec7e00640 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 0x55a9846d31b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faec009cad0 tx=0x7faec0005870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e5bba06f70207 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.279+0000 7faeb5600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:04:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.279+0000 7faeb5600640 1 -- 172.21.15.40:0/1121201612 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec007f070 con 0x55a9846c37d0 2024-04-27T00:04:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.279+0000 7faeb5600640 1 -- 172.21.15.40:0/1121201612 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7faec00a23b0 con 0x55a9846c37d0 2024-04-27T00:04:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.279+0000 7faed15a7740 1 -- 172.21.15.40:0/1121201612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faeb8040070 con 0x55a9846c37d0 2024-04-27T00:04:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.279+0000 7faeb5600640 1 -- 172.21.15.40:0/1121201612 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faec009cba0 con 0x55a9846c37d0 2024-04-27T00:04:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.279+0000 7faed15a7740 10 client.15477 fetch_fsmap learned FSMap version 24 2024-04-27T00:04:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.279+0000 7faed15a7740 1 -- 172.21.15.40:0/1121201612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a98473f050 con 0x55a9846c37d0 2024-04-27T00:04:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.280+0000 7faeb5600640 1 -- 172.21.15.40:0/1121201612 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeb8040070 con 0x55a9846c37d0 2024-04-27T00:04:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.280+0000 7faeb5600640 1 -- 172.21.15.40:0/1121201612 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7faec0082020 con 0x55a9846c37d0 2024-04-27T00:04:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.280+0000 7faed15a7740 10 client.15477 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:04:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.280+0000 7faed15a7740 10 client.15477 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:04:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.280+0000 7faed15a7740 20 client.15477 populate_metadata read hostname 'smithi040' 2024-04-27T00:04:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.280+0000 7faed15a7740 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a9846c8940 0x55a9847417a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.280+0000 7faed15a7740 4 client.15477 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:04:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.280+0000 7faed15a7740 1 -- 172.21.15.40:0/1121201612 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a98462ee00 con 0x55a9846c8940 2024-04-27T00:04:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.280+0000 7faec7400640 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a9846c8940 0x55a9847417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.281+0000 7faec7400640 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a9846c8940 0x55a9847417a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d33e54dbf57f4a0 server_cookie=75847b3a435c4f33 in_seq=0 out_seq=0 2024-04-27T00:04:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.281+0000 7faeb5600640 10 client.15477 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.297+0000 7faeb5600640 1 -- 172.21.15.40:0/1121201612 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a98462ee00 con 0x55a9846c8940 2024-04-27T00:04:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.297+0000 7faeb5600640 10 client.15477 handle_command_reply: tid=0 2024-04-27T00:04:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.297+0000 7faed15a7740 1 client.15477 shutdown 2024-04-27T00:04:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.298+0000 7faed15a7740 1 -- 172.21.15.40:0/1121201612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 msgr2=0x55a9846d31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.298+0000 7faed15a7740 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 0x55a9846d31b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faec009cad0 tx=0x7faec0005870 comp rx=0 tx=0).stop 2024-04-27T00:04:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.298+0000 7faed15a7740 1 -- 172.21.15.40:0/1121201612 shutdown_connections 2024-04-27T00:04:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.298+0000 7faed15a7740 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a9846c8940 0x55a9847417a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.298+0000 7faed15a7740 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a9846c37d0 0x55a9846d31b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.298+0000 7faed15a7740 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9846c2de0 0x55a9846d21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.298+0000 7faed15a7740 1 --2- 172.21.15.40:0/1121201612 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a9846be450 0x55a9846d1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.298+0000 7faeb5600640 0 client.15477 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.298+0000 7faed15a7740 1 -- 172.21.15.40:0/1121201612 >> 172.21.15.40:0/1121201612 conn(0x55a9846c8440 msgr2=0x55a9846c14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.298+0000 7faed15a7740 1 -- 172.21.15.40:0/1121201612 shutdown_connections 2024-04-27T00:04:27.300 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.298+0000 7faed15a7740 1 -- 172.21.15.40:0/1121201612 wait complete. 2024-04-27T00:04:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.298+0000 7faed15a7740 20 client.15477 trim_cache size 0 max 16384 2024-04-27T00:04:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:04:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:04:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:04:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:04:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.299+0000 7faecd200640 1 -- 172.21.15.40:0/1196270925 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7faeac0a3c30 msgr2=0x7faeac0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.299+0000 7faecd200640 1 --2- 172.21.15.40:0/1196270925 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7faeac0a3c30 0x7faeac0a60f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009fd0 tx=0x7faeb0015040 comp rx=0 tx=0).stop 2024-04-27T00:04:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.299+0000 7faecd200640 1 -- 172.21.15.40:0/1196270925 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faec8078ed0 msgr2=0x7faec81a65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.299+0000 7faecd200640 1 --2- 172.21.15.40:0/1196270925 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faec8078ed0 0x7faec81a65a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faec0009fd0 tx=0x7faec000ef10 comp rx=0 tx=0).stop 2024-04-27T00:04:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.300+0000 7faecd200640 1 -- 172.21.15.40:0/1196270925 shutdown_connections 2024-04-27T00:04:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.300+0000 7faecd200640 1 --2- 172.21.15.40:0/1196270925 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7faec8078ed0 0x7faec81a65a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.300+0000 7faecd200640 1 --2- 172.21.15.40:0/1196270925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faec8078530 0x7faec819f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.300+0000 7faecd200640 1 --2- 172.21.15.40:0/1196270925 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7faeac0a3c30 0x7faeac0a60f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.300+0000 7faecd200640 1 --2- 172.21.15.40:0/1196270925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faec80772f0 0x7faec819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.300+0000 7faecd200640 1 -- 172.21.15.40:0/1196270925 >> 172.21.15.40:0/1196270925 conn(0x7faec80fff60 msgr2=0x7faec81032c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.300+0000 7faecd200640 1 -- 172.21.15.40:0/1196270925 shutdown_connections 2024-04-27T00:04:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:27.300+0000 7faecd200640 1 -- 172.21.15.40:0/1196270925 wait complete. 2024-04-27T00:04:28.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:27 smithi040 ceph-mon[28095]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:28.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:27 smithi080 ceph-mon[32362]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:28.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:27 smithi103 ceph-mon[32952]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:30.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:29 smithi080 ceph-mon[32362]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:04:30.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:29 smithi103 ceph-mon[32952]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:04:30.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:29 smithi040 ceph-mon[28095]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:04:32.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:31 smithi080 ceph-mon[32362]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:04:32.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:31 smithi103 ceph-mon[32952]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:04:32.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:31 smithi040 ceph-mon[28095]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:04:34.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:33 smithi080 ceph-mon[32362]: pgmap v498: 129 pgs: 129 active+clean; 1.5 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:34.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:33 smithi103 ceph-mon[32952]: pgmap v498: 129 pgs: 129 active+clean; 1.5 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:34.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:33 smithi040 ceph-mon[28095]: pgmap v498: 129 pgs: 129 active+clean; 1.5 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:36.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:35 smithi080 ceph-mon[32362]: pgmap v499: 129 pgs: 129 active+clean; 1.5 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:36.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:35 smithi103 ceph-mon[32952]: pgmap v499: 129 pgs: 129 active+clean; 1.5 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:36.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:35 smithi040 ceph-mon[28095]: pgmap v499: 129 pgs: 129 active+clean; 1.5 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:38.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:37 smithi080 ceph-mon[32362]: pgmap v500: 129 pgs: 129 active+clean; 1.6 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:38.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:37 smithi103 ceph-mon[32952]: pgmap v500: 129 pgs: 129 active+clean; 1.6 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:38.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:37 smithi040 ceph-mon[28095]: pgmap v500: 129 pgs: 129 active+clean; 1.6 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:40.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:39 smithi080 ceph-mon[32362]: pgmap v501: 129 pgs: 129 active+clean; 1.6 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:40.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:39 smithi103 ceph-mon[32952]: pgmap v501: 129 pgs: 129 active+clean; 1.6 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:40.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.452+0000 7f9427e00640 1 -- 172.21.15.40:0/3485402101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9420108fc0 msgr2=0x7f94201093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:40.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.452+0000 7f9427e00640 1 --2- 172.21.15.40:0/3485402101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9420108fc0 0x7f94201093e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9414009900 tx=0x7f941402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.452+0000 7f9427e00640 1 -- 172.21.15.40:0/3485402101 shutdown_connections 2024-04-27T00:04:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.452+0000 7f9427e00640 1 --2- 172.21.15.40:0/3485402101 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f942010ab70 0x7f9420111400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.452+0000 7f9427e00640 1 --2- 172.21.15.40:0/3485402101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f942010a1b0 0x7f942010a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.452+0000 7f9427e00640 1 --2- 172.21.15.40:0/3485402101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9420108fc0 0x7f94201093e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.452+0000 7f9427e00640 1 -- 172.21.15.40:0/3485402101 >> 172.21.15.40:0/3485402101 conn(0x7f9420077f80 msgr2=0x7f942007a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.452+0000 7f9427e00640 1 -- 172.21.15.40:0/3485402101 shutdown_connections 2024-04-27T00:04:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.452+0000 7f9427e00640 1 -- 172.21.15.40:0/3485402101 wait complete. 2024-04-27T00:04:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.453+0000 7f9427e00640 1 Processor -- start 2024-04-27T00:04:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.454+0000 7f9427e00640 1 -- start start 2024-04-27T00:04:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.454+0000 7f9427e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9420108fc0 0x7f942019f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.454+0000 7f9427e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f942010a1b0 0x7f942019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.454+0000 7f9427e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f942010ab70 0x7f94201a6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.454+0000 7f9427e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9420114b50 con 0x7f942010a1b0 2024-04-27T00:04:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.454+0000 7f9427e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f94201149d0 con 0x7f9420108fc0 2024-04-27T00:04:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.454+0000 7f9427e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9420114cd0 con 0x7f942010ab70 2024-04-27T00:04:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.454+0000 7f9425400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9420108fc0 0x7f942019f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.454+0000 7f9425400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9420108fc0 0x7f942019f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:44208/0 (socket says 172.21.15.40:44208) 2024-04-27T00:04:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.454+0000 7f9425400640 1 -- 172.21.15.40:0/367801490 learned_addr learned my addr 172.21.15.40:0/367801490 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.454+0000 7f9425e00640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f942010ab70 0x7f94201a6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.454+0000 7f9424a00640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f942010a1b0 0x7f942019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.455+0000 7f9425400640 1 -- 172.21.15.40:0/367801490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f942010ab70 msgr2=0x7f94201a6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.455+0000 7f9425400640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f942010ab70 0x7f94201a6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.455+0000 7f9425400640 1 -- 172.21.15.40:0/367801490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f942010a1b0 msgr2=0x7f942019f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.455+0000 7f9425400640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f942010a1b0 0x7f942019f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.455+0000 7f9425400640 1 -- 172.21.15.40:0/367801490 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94201a6d90 con 0x7f9420108fc0 2024-04-27T00:04:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.455+0000 7f9425400640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9420108fc0 0x7f942019f170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9414031500 tx=0x7f9414003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.455+0000 7f941e000640 1 -- 172.21.15.40:0/367801490 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941403f070 con 0x7f9420108fc0 2024-04-27T00:04:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.456+0000 7f9427e00640 1 -- 172.21.15.40:0/367801490 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94201a7020 con 0x7f9420108fc0 2024-04-27T00:04:40.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.456+0000 7f941e000640 1 -- 172.21.15.40:0/367801490 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9414031c50 con 0x7f9420108fc0 2024-04-27T00:04:40.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.456+0000 7f941e000640 1 -- 172.21.15.40:0/367801490 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414035030 con 0x7f9420108fc0 2024-04-27T00:04:40.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.456+0000 7f9427e00640 1 -- 172.21.15.40:0/367801490 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94201a7560 con 0x7f9420108fc0 2024-04-27T00:04:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.459+0000 7f942adae740 1 Processor -- start 2024-04-27T00:04:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.459+0000 7f942adae740 1 -- start start 2024-04-27T00:04:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.459+0000 7f942adae740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a300bd0 0x56351a300ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.459+0000 7f942adae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a3015c0 0x56351a3060c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.459+0000 7f942adae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a306730 0x56351a308bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.459+0000 7f942adae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56351a265380 con 0x56351a306730 2024-04-27T00:04:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.459+0000 7f942adae740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x56351a201c30 con 0x56351a300bd0 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.459+0000 7f942adae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56351a261340 con 0x56351a3015c0 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f941e000640 1 -- 172.21.15.40:0/367801490 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9414042070 con 0x7f9420108fc0 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f9424a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a306730 0x56351a308bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f9425e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a300bd0 0x56351a300ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f9424a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a306730 0x56351a308bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42014/0 (socket says 172.21.15.40:42014) 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f9424a00640 1 -- 172.21.15.40:0/3118870529 learned_addr learned my addr 172.21.15.40:0/3118870529 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f9424a00640 1 -- 172.21.15.40:0/3118870529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a3015c0 msgr2=0x56351a3060c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f9425400640 1 --2- 172.21.15.40:0/3118870529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a3015c0 0x56351a3060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f9424a00640 1 --2- 172.21.15.40:0/3118870529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a3015c0 0x56351a3060c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f9424a00640 1 -- 172.21.15.40:0/3118870529 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a300bd0 msgr2=0x56351a300ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f9424a00640 1 --2- 172.21.15.40:0/3118870529 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a300bd0 0x56351a300ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f9424a00640 1 -- 172.21.15.40:0/3118870529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56351a223940 con 0x56351a306730 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f9425400640 1 --2- 172.21.15.40:0/3118870529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a3015c0 0x56351a3060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T00:04:40.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.460+0000 7f9425e00640 1 --2- 172.21.15.40:0/3118870529 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a300bd0 0x56351a300ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:04:40.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.461+0000 7f9424a00640 1 --2- 172.21.15.40:0/3118870529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a306730 0x56351a308bc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f940c009ca0 tx=0x7f940c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c47532c742dfdc87 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:40.465 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.461+0000 7f9412600640 1 -- 172.21.15.40:0/3118870529 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c01e070 con 0x56351a306730 2024-04-27T00:04:40.465 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.461+0000 7f9412600640 1 -- 172.21.15.40:0/3118870529 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f940c019ae0 con 0x56351a306730 2024-04-27T00:04:40.465 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.461+0000 7f9412600640 1 -- 172.21.15.40:0/3118870529 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c00e950 con 0x56351a306730 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.461+0000 7f941e000640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f94080a3d70 0x7f94080a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.461+0000 7f941e000640 1 -- 172.21.15.40:0/367801490 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f94140f0fd0 con 0x7f9420108fc0 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.462+0000 7f942adae740 1 -- 172.21.15.40:0/3118870529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a306730 msgr2=0x56351a308bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.462+0000 7f942adae740 1 --2- 172.21.15.40:0/3118870529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a306730 0x56351a308bc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f940c009ca0 tx=0x7f940c011810 comp rx=0 tx=0).stop 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.462+0000 7f9425e00640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f94080a3d70 0x7f94080a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.462+0000 7f942adae740 1 -- 172.21.15.40:0/3118870529 shutdown_connections 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.462+0000 7f942adae740 1 --2- 172.21.15.40:0/3118870529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a306730 0x56351a308bc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.462+0000 7f942adae740 1 --2- 172.21.15.40:0/3118870529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a3015c0 0x56351a3060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.462+0000 7f942adae740 1 --2- 172.21.15.40:0/3118870529 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a300bd0 0x56351a300ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.462+0000 7f942adae740 1 -- 172.21.15.40:0/3118870529 >> 172.21.15.40:0/3118870529 conn(0x56351a2fc240 msgr2=0x56351a2fc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.462+0000 7f942adae740 1 -- 172.21.15.40:0/3118870529 shutdown_connections 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.462+0000 7f942adae740 1 -- 172.21.15.40:0/3118870529 wait complete. 2024-04-27T00:04:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.462+0000 7f9425e00640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f94080a3d70 0x7f94080a6230 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x56351a301e20 tx=0x7f9418009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.462+0000 7f942adae740 1 -- start start 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f942adae740 1 Processor -- start 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f942adae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a300bd0 0x56351a30ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f942adae740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a3015c0 0x56351a30b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f942adae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a306730 0x56351a312390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f942adae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56351a3096e0 con 0x56351a306730 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f942adae740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x56351a1ecc60 con 0x56351a3015c0 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f942adae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56351a309860 con 0x56351a300bd0 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f942adae740 10 client.0 _ll_register_callbacks cb 0x56351a218e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f9425400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a300bd0 0x56351a30ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f9425e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a306730 0x56351a312390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f9425400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a300bd0 0x56351a30ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:57432/0 (socket says 172.21.15.40:57432) 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f9425400640 1 -- 172.21.15.40:0/2992702087 learned_addr learned my addr 172.21.15.40:0/2992702087 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.463+0000 7f9424a00640 1 --2- 172.21.15.40:0/2992702087 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a3015c0 0x56351a30b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.464+0000 7f9425400640 1 -- 172.21.15.40:0/2992702087 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a3015c0 msgr2=0x56351a30b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.464+0000 7f9425400640 1 --2- 172.21.15.40:0/2992702087 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a3015c0 0x56351a30b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.464+0000 7f9425400640 1 -- 172.21.15.40:0/2992702087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a306730 msgr2=0x56351a312390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.464+0000 7f9425400640 1 --2- 172.21.15.40:0/2992702087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a306730 0x56351a312390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.464+0000 7f9425400640 1 -- 172.21.15.40:0/2992702087 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56351a1fb510 con 0x56351a300bd0 2024-04-27T00:04:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.464+0000 7f9425400640 1 --2- 172.21.15.40:0/2992702087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a300bd0 0x56351a30ae10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f94140ee5b0 tx=0x7f941402a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4705c8fe7dbdc06 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.464+0000 7f9406a00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:04:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.464+0000 7f9406a00640 1 -- 172.21.15.40:0/2992702087 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414049050 con 0x56351a300bd0 2024-04-27T00:04:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.465+0000 7f9406a00640 1 -- 172.21.15.40:0/2992702087 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f94140ee7d0 con 0x56351a300bd0 2024-04-27T00:04:40.468 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.465+0000 7f942adae740 1 -- 172.21.15.40:0/2992702087 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f940c019ae0 con 0x56351a300bd0 2024-04-27T00:04:40.468 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:39 smithi040 ceph-mon[28095]: pgmap v501: 129 pgs: 129 active+clean; 1.6 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.465+0000 7f9406a00640 1 -- 172.21.15.40:0/2992702087 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f94140bd450 con 0x56351a300bd0 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.465+0000 7f942adae740 10 client.25630 fetch_fsmap learned FSMap version 24 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.465+0000 7f942adae740 1 -- 172.21.15.40:0/2992702087 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563519f77210 con 0x56351a300bd0 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.465+0000 7f9406a00640 1 -- 172.21.15.40:0/2992702087 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f940c019ae0 con 0x56351a300bd0 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.465+0000 7f9406a00640 1 -- 172.21.15.40:0/2992702087 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f94140d60d0 con 0x56351a300bd0 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.466+0000 7f942adae740 10 client.25630 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.466+0000 7f942adae740 10 client.25630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.466+0000 7f942adae740 20 client.25630 populate_metadata read hostname 'smithi040' 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.466+0000 7f942adae740 1 --2- 172.21.15.40:0/2992702087 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56351a3168e0 0x56351a318d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.466+0000 7f942adae740 4 client.25630 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.466+0000 7f942adae740 1 -- 172.21.15.40:0/2992702087 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56351a272f90 con 0x56351a3168e0 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.466+0000 7f9424a00640 1 --2- 172.21.15.40:0/2992702087 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56351a3168e0 0x56351a318d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.467+0000 7f9424a00640 1 --2- 172.21.15.40:0/2992702087 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56351a3168e0 0x56351a318d70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b49d8fc0561dd59a server_cookie=3180713b8912e3c6 in_seq=0 out_seq=0 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.467+0000 7f9406a00640 10 client.25630 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f9406a00640 1 -- 172.21.15.40:0/2992702087 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56351a272f90 con 0x56351a3168e0 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f9406a00640 10 client.25630 handle_command_reply: tid=0 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f942adae740 1 client.25630 shutdown 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f942adae740 1 -- 172.21.15.40:0/2992702087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a300bd0 msgr2=0x56351a30ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f942adae740 1 --2- 172.21.15.40:0/2992702087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a300bd0 0x56351a30ae10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f94140ee5b0 tx=0x7f941402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f942adae740 1 -- 172.21.15.40:0/2992702087 shutdown_connections 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f942adae740 1 --2- 172.21.15.40:0/2992702087 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56351a3168e0 0x56351a318d70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f942adae740 1 --2- 172.21.15.40:0/2992702087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a306730 0x56351a312390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f942adae740 1 --2- 172.21.15.40:0/2992702087 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a3015c0 0x56351a30b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f942adae740 1 --2- 172.21.15.40:0/2992702087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a300bd0 0x56351a30ae10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f9406a00640 0 client.25630 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f942adae740 1 -- 172.21.15.40:0/2992702087 >> 172.21.15.40:0/2992702087 conn(0x56351a2fc240 msgr2=0x56351a2fb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f942adae740 1 -- 172.21.15.40:0/2992702087 shutdown_connections 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f942adae740 1 -- 172.21.15.40:0/2992702087 wait complete. 2024-04-27T00:04:40.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.468+0000 7f942adae740 20 client.25630 trim_cache size 0 max 16384 2024-04-27T00:04:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.485+0000 7f942adae740 1 Processor -- start 2024-04-27T00:04:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.485+0000 7f942adae740 1 -- start start 2024-04-27T00:04:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.485+0000 7f942adae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a300bd0 0x56351a2ff1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.485+0000 7f942adae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a3015c0 0x56351a312350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.485+0000 7f942adae740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a306730 0x56351a2ff730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.485+0000 7f942adae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94201b4280 con 0x56351a300bd0 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.485+0000 7f942adae740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f94201b1b50 con 0x56351a306730 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.485+0000 7f942adae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f94201af9a0 con 0x56351a3015c0 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.485+0000 7f9425e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a306730 0x56351a2ff730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.486+0000 7f9425400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a300bd0 0x56351a2ff1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.486+0000 7f9425e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a306730 0x56351a2ff730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:44242/0 (socket says 172.21.15.40:44242) 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.486+0000 7f9425400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a300bd0 0x56351a2ff1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42032/0 (socket says 172.21.15.40:42032) 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.486+0000 7f9425e00640 1 -- 172.21.15.40:0/220995077 learned_addr learned my addr 172.21.15.40:0/220995077 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.486+0000 7f9424a00640 1 --2- 172.21.15.40:0/220995077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a3015c0 0x56351a312350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.486+0000 7f9425400640 1 -- 172.21.15.40:0/220995077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a3015c0 msgr2=0x56351a312350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.486+0000 7f9425400640 1 --2- 172.21.15.40:0/220995077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a3015c0 0x56351a312350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.486+0000 7f9425400640 1 -- 172.21.15.40:0/220995077 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a306730 msgr2=0x56351a2ff730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.486+0000 7f9425400640 1 --2- 172.21.15.40:0/220995077 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a306730 0x56351a2ff730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.486+0000 7f9425400640 1 -- 172.21.15.40:0/220995077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56351a043620 con 0x56351a300bd0 2024-04-27T00:04:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.486+0000 7f9425400640 1 --2- 172.21.15.40:0/220995077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a300bd0 0x56351a2ff1f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9414004010 tx=0x7f94140c5b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca12b021f9bd6f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.486+0000 7f9411c00640 1 -- 172.21.15.40:0/220995077 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414049070 con 0x56351a300bd0 2024-04-27T00:04:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.487+0000 7f9411c00640 1 -- 172.21.15.40:0/220995077 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94140bcd50 con 0x56351a300bd0 2024-04-27T00:04:40.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.487+0000 7f9411c00640 1 -- 172.21.15.40:0/220995077 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94140ee7d0 con 0x56351a300bd0 2024-04-27T00:04:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.487+0000 7f942adae740 1 -- 172.21.15.40:0/220995077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a300bd0 msgr2=0x56351a2ff1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.487+0000 7f942adae740 1 --2- 172.21.15.40:0/220995077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a300bd0 0x56351a2ff1f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9414004010 tx=0x7f94140c5b40 comp rx=0 tx=0).stop 2024-04-27T00:04:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.487+0000 7f942adae740 1 -- 172.21.15.40:0/220995077 shutdown_connections 2024-04-27T00:04:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.487+0000 7f942adae740 1 --2- 172.21.15.40:0/220995077 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a306730 0x56351a2ff730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.487+0000 7f942adae740 1 --2- 172.21.15.40:0/220995077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a3015c0 0x56351a312350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.487+0000 7f942adae740 1 --2- 172.21.15.40:0/220995077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a300bd0 0x56351a2ff1f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.487+0000 7f942adae740 1 -- 172.21.15.40:0/220995077 >> 172.21.15.40:0/220995077 conn(0x56351a2fc240 msgr2=0x56351a2fc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.487+0000 7f942adae740 1 -- 172.21.15.40:0/220995077 shutdown_connections 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.487+0000 7f942adae740 1 -- 172.21.15.40:0/220995077 wait complete. 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.488+0000 7f942adae740 1 -- start start 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.488+0000 7f942adae740 1 Processor -- start 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.488+0000 7f942adae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a300bd0 0x56351a30f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.488+0000 7f942adae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a3015c0 0x56351a30f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.488+0000 7f942adae740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a306730 0x56351a310950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.488+0000 7f942adae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94201b0010 con 0x56351a3015c0 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.488+0000 7f942adae740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f94201b1480 con 0x56351a306730 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.488+0000 7f942adae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f94201b06e0 con 0x56351a300bd0 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.488+0000 7f942adae740 10 client.0 _ll_register_callbacks cb 0x7f9414005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.489+0000 7f9424a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a3015c0 0x56351a30f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.489+0000 7f9424a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a3015c0 0x56351a30f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42042/0 (socket says 172.21.15.40:42042) 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.489+0000 7f9424a00640 1 -- 172.21.15.40:0/3922312848 learned_addr learned my addr 172.21.15.40:0/3922312848 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.489+0000 7f9425400640 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a300bd0 0x56351a30f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.489+0000 7f9425e00640 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a306730 0x56351a310950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.489+0000 7f9424a00640 1 -- 172.21.15.40:0/3922312848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a300bd0 msgr2=0x56351a30f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.489+0000 7f9424a00640 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a300bd0 0x56351a30f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.489+0000 7f9424a00640 1 -- 172.21.15.40:0/3922312848 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a306730 msgr2=0x56351a310950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.489+0000 7f9424a00640 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a306730 0x56351a310950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.489+0000 7f9424a00640 1 -- 172.21.15.40:0/3922312848 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56351a21fcb0 con 0x56351a3015c0 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.489+0000 7f9424a00640 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a3015c0 0x56351a30f9b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f940c011d10 tx=0x7f940c005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d83652a84d9e1c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.489+0000 7f9406a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.490+0000 7f9406a00640 1 -- 172.21.15.40:0/3922312848 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c01e040 con 0x56351a3015c0 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.490+0000 7f9406a00640 1 -- 172.21.15.40:0/3922312848 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f940c019a80 con 0x56351a3015c0 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.490+0000 7f942adae740 1 -- 172.21.15.40:0/3922312848 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94140bcd50 con 0x56351a3015c0 2024-04-27T00:04:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.490+0000 7f9406a00640 1 -- 172.21.15.40:0/3922312848 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f940c03c050 con 0x56351a3015c0 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.490+0000 7f942adae740 10 client.15495 fetch_fsmap learned FSMap version 24 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.490+0000 7f942adae740 1 -- 172.21.15.40:0/3922312848 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56351a3120f0 con 0x56351a3015c0 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.490+0000 7f9406a00640 1 -- 172.21.15.40:0/3922312848 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94140bcd50 con 0x56351a3015c0 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.491+0000 7f9406a00640 1 -- 172.21.15.40:0/3922312848 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f940c004680 con 0x56351a3015c0 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.491+0000 7f942adae740 10 client.15495 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.491+0000 7f942adae740 10 client.15495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.491+0000 7f942adae740 20 client.15495 populate_metadata read hostname 'smithi040' 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.491+0000 7f942adae740 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56351a37cee0 0x56351a37f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.491+0000 7f942adae740 4 client.15495 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.491+0000 7f942adae740 1 -- 172.21.15.40:0/3922312848 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56351a26ce00 con 0x56351a37cee0 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.491+0000 7f9425400640 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56351a37cee0 0x56351a37f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.491+0000 7f9425400640 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56351a37cee0 0x56351a37f2d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f85ca2d7324186a server_cookie=c99b7faa594a446d in_seq=0 out_seq=0 2024-04-27T00:04:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.491+0000 7f9406a00640 10 client.15495 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.505+0000 7f9406a00640 1 -- 172.21.15.40:0/3922312848 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56351a26ce00 con 0x56351a37cee0 2024-04-27T00:04:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f9406a00640 10 client.15495 handle_command_reply: tid=0 2024-04-27T00:04:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f942adae740 1 client.15495 shutdown 2024-04-27T00:04:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f942adae740 1 -- 172.21.15.40:0/3922312848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a3015c0 msgr2=0x56351a30f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f942adae740 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a3015c0 0x56351a30f9b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f940c011d10 tx=0x7f940c005db0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f942adae740 1 -- 172.21.15.40:0/3922312848 shutdown_connections 2024-04-27T00:04:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f942adae740 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56351a37cee0 0x56351a37f2d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f942adae740 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56351a306730 0x56351a310950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f942adae740 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56351a3015c0 0x56351a30f9b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f9406a00640 0 client.15495 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f942adae740 1 --2- 172.21.15.40:0/3922312848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56351a300bd0 0x56351a30f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f942adae740 1 -- 172.21.15.40:0/3922312848 >> 172.21.15.40:0/3922312848 conn(0x56351a2fc240 msgr2=0x56351a2ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f942adae740 1 -- 172.21.15.40:0/3922312848 shutdown_connections 2024-04-27T00:04:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f942adae740 1 -- 172.21.15.40:0/3922312848 wait complete. 2024-04-27T00:04:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.506+0000 7f942adae740 20 client.15495 trim_cache size 0 max 16384 2024-04-27T00:04:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:04:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:04:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:04:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:04:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.507+0000 7f9427e00640 1 -- 172.21.15.40:0/367801490 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f94080a3d70 msgr2=0x7f94080a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.507+0000 7f9427e00640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f94080a3d70 0x7f94080a6230 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x56351a301e20 tx=0x7f9418009500 comp rx=0 tx=0).stop 2024-04-27T00:04:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.507+0000 7f9427e00640 1 -- 172.21.15.40:0/367801490 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9420108fc0 msgr2=0x7f942019f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.507+0000 7f9427e00640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9420108fc0 0x7f942019f170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9414031500 tx=0x7f9414003c10 comp rx=0 tx=0).stop 2024-04-27T00:04:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.507+0000 7f9427e00640 1 -- 172.21.15.40:0/367801490 shutdown_connections 2024-04-27T00:04:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.507+0000 7f9427e00640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f94080a3d70 0x7f94080a6230 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.507+0000 7f9427e00640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f942010ab70 0x7f94201a6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.507+0000 7f9427e00640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f942010a1b0 0x7f942019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.507+0000 7f9427e00640 1 --2- 172.21.15.40:0/367801490 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9420108fc0 0x7f942019f170 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.507+0000 7f9427e00640 1 -- 172.21.15.40:0/367801490 >> 172.21.15.40:0/367801490 conn(0x7f9420077f80 msgr2=0x7f942010d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.508+0000 7f9427e00640 1 -- 172.21.15.40:0/367801490 shutdown_connections 2024-04-27T00:04:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:40.508+0000 7f9427e00640 1 -- 172.21.15.40:0/367801490 wait complete. 2024-04-27T00:04:42.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:42 smithi080 ceph-mon[32362]: pgmap v502: 129 pgs: 129 active+clean; 1.6 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:42.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:42 smithi103 ceph-mon[32952]: pgmap v502: 129 pgs: 129 active+clean; 1.6 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:42.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:41 smithi040 ceph-mon[28095]: pgmap v502: 129 pgs: 129 active+clean; 1.6 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:43.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:43 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:04:43.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:43 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:04:43.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:43 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:04:44.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:44 smithi080 ceph-mon[32362]: pgmap v503: 129 pgs: 129 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:04:44.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:44 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:04:44.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:44 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:04:44.300 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:44 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:04:44.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:44 smithi103 ceph-mon[32952]: pgmap v503: 129 pgs: 129 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:04:44.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:04:44.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:04:44.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:44 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:04:44.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:44 smithi040 ceph-mon[28095]: pgmap v503: 129 pgs: 129 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:04:44.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:04:44.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:04:44.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:44 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:04:46.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:46 smithi080 ceph-mon[32362]: pgmap v504: 129 pgs: 129 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:46.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:46 smithi103 ceph-mon[32952]: pgmap v504: 129 pgs: 129 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:46.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:46 smithi040 ceph-mon[28095]: pgmap v504: 129 pgs: 129 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:04:48.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:48 smithi080 ceph-mon[32362]: pgmap v505: 129 pgs: 129 active+clean; 1.7 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:48.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:48 smithi103 ceph-mon[32952]: pgmap v505: 129 pgs: 129 active+clean; 1.7 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:48.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:48 smithi040 ceph-mon[28095]: pgmap v505: 129 pgs: 129 active+clean; 1.7 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:04:50.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:50 smithi080 ceph-mon[32362]: pgmap v506: 129 pgs: 129 active+clean; 1.7 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:04:50.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:50 smithi103 ceph-mon[32952]: pgmap v506: 129 pgs: 129 active+clean; 1.7 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:04:50.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:50 smithi040 ceph-mon[28095]: pgmap v506: 129 pgs: 129 active+clean; 1.7 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:04:52.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:52 smithi080 ceph-mon[32362]: pgmap v507: 129 pgs: 129 active+clean; 1.7 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:04:52.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:52 smithi103 ceph-mon[32952]: pgmap v507: 129 pgs: 129 active+clean; 1.7 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:04:52.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:52 smithi040 ceph-mon[28095]: pgmap v507: 129 pgs: 129 active+clean; 1.7 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:04:53.657 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.656+0000 7f76eb200640 1 -- 172.21.15.40:0/3172089402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76e4108de0 msgr2=0x7f76e4109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:53.657 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.656+0000 7f76eb200640 1 --2- 172.21.15.40:0/3172089402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76e4108de0 0x7f76e4109200 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f76d8009900 tx=0x7f76d802a3f0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.656+0000 7f76eb200640 1 -- 172.21.15.40:0/3172089402 shutdown_connections 2024-04-27T00:04:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.656+0000 7f76eb200640 1 --2- 172.21.15.40:0/3172089402 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f76e410a990 0x7f76e4111220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.656+0000 7f76eb200640 1 --2- 172.21.15.40:0/3172089402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76e4109fd0 0x7f76e410a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.656+0000 7f76eb200640 1 --2- 172.21.15.40:0/3172089402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76e4108de0 0x7f76e4109200 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.656+0000 7f76eb200640 1 -- 172.21.15.40:0/3172089402 >> 172.21.15.40:0/3172089402 conn(0x7f76e4077dc0 msgr2=0x7f76e407a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.656+0000 7f76eb200640 1 -- 172.21.15.40:0/3172089402 shutdown_connections 2024-04-27T00:04:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.656+0000 7f76eb200640 1 -- 172.21.15.40:0/3172089402 wait complete. 2024-04-27T00:04:53.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.657+0000 7f76eb200640 1 Processor -- start 2024-04-27T00:04:53.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.657+0000 7f76eb200640 1 -- start start 2024-04-27T00:04:53.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.657+0000 7f76eb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76e4108de0 0x7f76e419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.657+0000 7f76eb200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f76e4109fd0 0x7f76e419f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.657+0000 7f76eb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76e410a990 0x7f76e41a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.657+0000 7f76eb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f76e4114950 con 0x7f76e410a990 2024-04-27T00:04:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.657+0000 7f76eb200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41147d0 con 0x7f76e4109fd0 2024-04-27T00:04:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.657+0000 7f76eb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76e4114ad0 con 0x7f76e4108de0 2024-04-27T00:04:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.658+0000 7f76e9400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f76e4109fd0 0x7f76e419f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.658+0000 7f76e9400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f76e4109fd0 0x7f76e419f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:60942/0 (socket says 172.21.15.40:60942) 2024-04-27T00:04:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.658+0000 7f76ea800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76e410a990 0x7f76e41a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.658+0000 7f76e9400640 1 -- 172.21.15.40:0/731052878 learned_addr learned my addr 172.21.15.40:0/731052878 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.658+0000 7f76e9e00640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76e4108de0 0x7f76e419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.658+0000 7f76e9400640 1 -- 172.21.15.40:0/731052878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76e4108de0 msgr2=0x7f76e419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.658+0000 7f76e9400640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76e4108de0 0x7f76e419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.658+0000 7f76e9400640 1 -- 172.21.15.40:0/731052878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76e410a990 msgr2=0x7f76e41a65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.658+0000 7f76e9400640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76e410a990 0x7f76e41a65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.658+0000 7f76e9400640 1 -- 172.21.15.40:0/731052878 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76e41a6c20 con 0x7f76e4109fd0 2024-04-27T00:04:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.659+0000 7f76e9400640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f76e4109fd0 0x7f76e419f540 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f76d400e9f0 tx=0x7f76d400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.659+0000 7f76dea00640 1 -- 172.21.15.40:0/731052878 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d400cd80 con 0x7f76e4109fd0 2024-04-27T00:04:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.659+0000 7f76dea00640 1 -- 172.21.15.40:0/731052878 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76d40054d0 con 0x7f76e4109fd0 2024-04-27T00:04:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.659+0000 7f76dea00640 1 -- 172.21.15.40:0/731052878 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d4010690 con 0x7f76e4109fd0 2024-04-27T00:04:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.659+0000 7f76eb200640 1 -- 172.21.15.40:0/731052878 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76e41a6f10 con 0x7f76e4109fd0 2024-04-27T00:04:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.659+0000 7f76eb200640 1 -- 172.21.15.40:0/731052878 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76e41a73d0 con 0x7f76e4109fd0 2024-04-27T00:04:53.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.663+0000 7f76ef5ff740 1 Processor -- start 2024-04-27T00:04:53.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.663+0000 7f76ef5ff740 1 -- start start 2024-04-27T00:04:53.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76ef5ff740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f0fde0 0x55a579f10200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76ef5ff740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 0x55a579f152d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76ef5ff740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f17dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76ef5ff740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a579e74380 con 0x55a579f107d0 2024-04-27T00:04:53.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76ef5ff740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55a579e10c30 con 0x55a579f15940 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76ef5ff740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a579e70340 con 0x55a579f0fde0 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76dea00640 1 -- 172.21.15.40:0/731052878 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f76d4004e90 con 0x7f76e4109fd0 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76e9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 0x55a579f152d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76ea800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f0fde0 0x55a579f10200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76ea800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f0fde0 0x55a579f10200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:39998/0 (socket says 172.21.15.40:39998) 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76e9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 0x55a579f152d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54804/0 (socket says 172.21.15.40:54804) 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76ea800640 1 -- 172.21.15.40:0/1736900539 learned_addr learned my addr 172.21.15.40:0/1736900539 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76e9400640 1 --2- 172.21.15.40:0/1736900539 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f17dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.664+0000 7f76ea800640 1 -- 172.21.15.40:0/1736900539 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 msgr2=0x55a579f17dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.665+0000 7f76ea800640 1 --2- 172.21.15.40:0/1736900539 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f17dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.665+0000 7f76ea800640 1 -- 172.21.15.40:0/1736900539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 msgr2=0x55a579f152d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.665+0000 7f76ea800640 1 --2- 172.21.15.40:0/1736900539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 0x55a579f152d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.665+0000 7f76ea800640 1 -- 172.21.15.40:0/1736900539 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a579e32940 con 0x55a579f0fde0 2024-04-27T00:04:53.668 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.665+0000 7f76e9e00640 1 --2- 172.21.15.40:0/1736900539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 0x55a579f152d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:04:53.668 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.665+0000 7f76e9400640 1 --2- 172.21.15.40:0/1736900539 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f17dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T00:04:53.668 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.665+0000 7f76ea800640 1 --2- 172.21.15.40:0/1736900539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f0fde0 0x55a579f10200 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009fd0 tx=0x7f76e00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c5c7ec3e79d4c2d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:53.668 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.665+0000 7f76d3000640 1 -- 172.21.15.40:0/1736900539 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0004d20 con 0x55a579f0fde0 2024-04-27T00:04:53.668 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.665+0000 7f76d3000640 1 -- 172.21.15.40:0/1736900539 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e0004050 con 0x55a579f0fde0 2024-04-27T00:04:53.668 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.665+0000 7f76d3000640 1 -- 172.21.15.40:0/1736900539 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0005240 con 0x55a579f0fde0 2024-04-27T00:04:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.666+0000 7f76dea00640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f76cc0a3cb0 0x7f76cc0a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.666+0000 7f76ef5ff740 1 -- 172.21.15.40:0/1736900539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f0fde0 msgr2=0x55a579f10200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.666+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/1736900539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f0fde0 0x55a579f10200 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009fd0 tx=0x7f76e00118d0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.666+0000 7f76ef5ff740 1 -- 172.21.15.40:0/1736900539 shutdown_connections 2024-04-27T00:04:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.666+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/1736900539 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f17dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.666+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/1736900539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 0x55a579f152d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.666+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/1736900539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f0fde0 0x55a579f10200 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.666+0000 7f76ef5ff740 1 -- 172.21.15.40:0/1736900539 >> 172.21.15.40:0/1736900539 conn(0x55a579f0b450 msgr2=0x55a579f0b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.666+0000 7f76dea00640 1 -- 172.21.15.40:0/731052878 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f76d40d0c90 con 0x7f76e4109fd0 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.666+0000 7f76ef5ff740 1 -- 172.21.15.40:0/1736900539 shutdown_connections 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.666+0000 7f76ef5ff740 1 -- 172.21.15.40:0/1736900539 wait complete. 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.667+0000 7f76e9e00640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f76cc0a3cb0 0x7f76cc0a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.667+0000 7f76ef5ff740 1 -- start start 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.667+0000 7f76ef5ff740 1 Processor -- start 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.667+0000 7f76e9e00640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f76cc0a3cb0 0x7f76cc0a6170 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f76d8005ab0 tx=0x7f76d8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76ef5ff740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f0fde0 0x55a579f19f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76ef5ff740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 0x55a579f1a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76ef5ff740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f21500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76ef5ff740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a579f188f0 con 0x55a579f0fde0 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76ef5ff740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55a579dfbc60 con 0x55a579f15940 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76ef5ff740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a579f18a70 con 0x55a579f107d0 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76ef5ff740 10 client.0 _ll_register_callbacks cb 0x55a579e27e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:04:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76e9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f0fde0 0x55a579f19f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76e9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 0x55a579f1a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76e9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f0fde0 0x55a579f19f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54818/0 (socket says 172.21.15.40:54818) 2024-04-27T00:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76e9e00640 1 -- 172.21.15.40:0/2319633093 learned_addr learned my addr 172.21.15.40:0/2319633093 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76ea800640 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f21500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76e9400640 1 -- 172.21.15.40:0/2319633093 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 msgr2=0x55a579f21500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76e9400640 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f21500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76e9400640 1 -- 172.21.15.40:0/2319633093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f0fde0 msgr2=0x55a579f19f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76e9400640 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f0fde0 0x55a579f19f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76e9400640 1 -- 172.21.15.40:0/2319633093 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a579e0a510 con 0x55a579f107d0 2024-04-27T00:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.668+0000 7f76ea800640 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f21500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:04:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.669+0000 7f76e9400640 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 0x55a579f1a4c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f76d40ce2a0 tx=0x7f76d409c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e6b06e1ef76fa5d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.669+0000 7f76c7400640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.669+0000 7f76c7400640 1 -- 172.21.15.40:0/2319633093 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d409ca70 con 0x55a579f107d0 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.669+0000 7f76ef5ff740 1 -- 172.21.15.40:0/2319633093 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76e0004050 con 0x55a579f107d0 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.669+0000 7f76c7400640 1 -- 172.21.15.40:0/2319633093 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f76d40ce4f0 con 0x55a579f107d0 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.669+0000 7f76c7400640 1 -- 172.21.15.40:0/2319633093 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f76d40a0e20 con 0x55a579f107d0 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.669+0000 7f76ef5ff740 10 client.25648 fetch_fsmap learned FSMap version 24 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.669+0000 7f76ef5ff740 1 -- 172.21.15.40:0/2319633093 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a579b86210 con 0x55a579f107d0 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.670+0000 7f76c7400640 1 -- 172.21.15.40:0/2319633093 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e0004050 con 0x55a579f107d0 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.670+0000 7f76c7400640 1 -- 172.21.15.40:0/2319633093 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f76d40a5250 con 0x55a579f107d0 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.670+0000 7f76ef5ff740 10 client.25648 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.670+0000 7f76ef5ff740 10 client.25648 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.670+0000 7f76ef5ff740 20 client.25648 populate_metadata read hostname 'smithi040' 2024-04-27T00:04:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.670+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a579f25ac0 0x55a579f27ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.670+0000 7f76ef5ff740 4 client.25648 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:04:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.670+0000 7f76ef5ff740 1 -- 172.21.15.40:0/2319633093 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a579e81f90 con 0x55a579f25ac0 2024-04-27T00:04:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.670+0000 7f76e9e00640 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a579f25ac0 0x55a579f27ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.671+0000 7f76e9e00640 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a579f25ac0 0x55a579f27ef0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72ed29d814f95e91 server_cookie=5d07bba7d9449c96 in_seq=0 out_seq=0 2024-04-27T00:04:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.671+0000 7f76c7400640 10 client.25648 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76c7400640 1 -- 172.21.15.40:0/2319633093 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a579e81f90 con 0x55a579f25ac0 2024-04-27T00:04:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76c7400640 10 client.25648 handle_command_reply: tid=0 2024-04-27T00:04:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76ef5ff740 1 client.25648 shutdown 2024-04-27T00:04:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76ef5ff740 1 -- 172.21.15.40:0/2319633093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 msgr2=0x55a579f1a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 0x55a579f1a4c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f76d40ce2a0 tx=0x7f76d409c670 comp rx=0 tx=0).stop 2024-04-27T00:04:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76ef5ff740 1 -- 172.21.15.40:0/2319633093 shutdown_connections 2024-04-27T00:04:53.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a579f25ac0 0x55a579f27ef0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f21500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 0x55a579f1a4c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76c7400640 0 client.25648 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/2319633093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f0fde0 0x55a579f19f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76ef5ff740 1 -- 172.21.15.40:0/2319633093 >> 172.21.15.40:0/2319633093 conn(0x55a579f0b450 msgr2=0x55a579f0a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76ef5ff740 1 -- 172.21.15.40:0/2319633093 shutdown_connections 2024-04-27T00:04:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76ef5ff740 1 -- 172.21.15.40:0/2319633093 wait complete. 2024-04-27T00:04:53.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.672+0000 7f76ef5ff740 20 client.25648 trim_cache size 0 max 16384 2024-04-27T00:04:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.689+0000 7f76ef5ff740 1 Processor -- start 2024-04-27T00:04:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.689+0000 7f76ef5ff740 1 -- start start 2024-04-27T00:04:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.689+0000 7f76ef5ff740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f0fde0 0x55a579f0add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.689+0000 7f76ef5ff740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 0x55a579f214c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.689+0000 7f76ef5ff740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f0ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.689+0000 7f76ef5ff740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41b4110 con 0x55a579f107d0 2024-04-27T00:04:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.689+0000 7f76ef5ff740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41b19e0 con 0x55a579f15940 2024-04-27T00:04:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.689+0000 7f76ef5ff740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41af830 con 0x55a579f0fde0 2024-04-27T00:04:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.690+0000 7f76e9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 0x55a579f214c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.690+0000 7f76e9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 0x55a579f214c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54826/0 (socket says 172.21.15.40:54826) 2024-04-27T00:04:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.690+0000 7f76e9400640 1 -- 172.21.15.40:0/738112700 learned_addr learned my addr 172.21.15.40:0/738112700 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.690+0000 7f76ea800640 1 --2- 172.21.15.40:0/738112700 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f0ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.690+0000 7f76e9e00640 1 --2- 172.21.15.40:0/738112700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f0fde0 0x55a579f0add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.690+0000 7f76e9400640 1 -- 172.21.15.40:0/738112700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f0fde0 msgr2=0x55a579f0add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.690+0000 7f76e9400640 1 --2- 172.21.15.40:0/738112700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f0fde0 0x55a579f0add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.690+0000 7f76e9400640 1 -- 172.21.15.40:0/738112700 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 msgr2=0x55a579f0ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.690+0000 7f76e9400640 1 --2- 172.21.15.40:0/738112700 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f0ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.690+0000 7f76e9400640 1 -- 172.21.15.40:0/738112700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a579c52620 con 0x55a579f107d0 2024-04-27T00:04:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.691+0000 7f76e9400640 1 --2- 172.21.15.40:0/738112700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 0x55a579f214c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f76d40ce320 tx=0x7f76d409c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ff26aa448cce16 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.691+0000 7f76d2600640 1 -- 172.21.15.40:0/738112700 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d4005180 con 0x55a579f107d0 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.691+0000 7f76d2600640 1 -- 172.21.15.40:0/738112700 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76d409ca70 con 0x55a579f107d0 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.691+0000 7f76d2600640 1 -- 172.21.15.40:0/738112700 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d40ce4f0 con 0x55a579f107d0 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.691+0000 7f76ef5ff740 1 -- 172.21.15.40:0/738112700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 msgr2=0x55a579f214c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.692+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/738112700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 0x55a579f214c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f76d40ce320 tx=0x7f76d409c6e0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.692+0000 7f76ef5ff740 1 -- 172.21.15.40:0/738112700 shutdown_connections 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.692+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/738112700 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f15940 0x55a579f0ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.692+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/738112700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f107d0 0x55a579f214c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.692+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/738112700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f0fde0 0x55a579f0add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.692+0000 7f76ef5ff740 1 -- 172.21.15.40:0/738112700 >> 172.21.15.40:0/738112700 conn(0x55a579f0b450 msgr2=0x55a579f0e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.692+0000 7f76ef5ff740 1 -- 172.21.15.40:0/738112700 shutdown_connections 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.692+0000 7f76ef5ff740 1 -- 172.21.15.40:0/738112700 wait complete. 2024-04-27T00:04:53.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.692+0000 7f76ef5ff740 1 -- start start 2024-04-27T00:04:53.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.692+0000 7f76ef5ff740 1 Processor -- start 2024-04-27T00:04:53.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76ef5ff740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f0fde0 0x55a579f1ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76ef5ff740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 0x55a579f1f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76ef5ff740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f15940 0x55a579f20240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76ef5ff740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41afea0 con 0x55a579f15940 2024-04-27T00:04:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76ef5ff740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41b1310 con 0x55a579f0fde0 2024-04-27T00:04:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76ef5ff740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76e41b0570 con 0x55a579f107d0 2024-04-27T00:04:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76ef5ff740 10 client.0 _ll_register_callbacks cb 0x7f76d400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:04:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76e9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 0x55a579f1f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76ea800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f15940 0x55a579f20240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76e9e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f0fde0 0x55a579f1ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76e9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 0x55a579f1f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:40036/0 (socket says 172.21.15.40:40036) 2024-04-27T00:04:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76e9400640 1 -- 172.21.15.40:0/4222197599 learned_addr learned my addr 172.21.15.40:0/4222197599 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:04:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76e9400640 1 -- 172.21.15.40:0/4222197599 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f0fde0 msgr2=0x55a579f1ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76e9400640 1 --2- 172.21.15.40:0/4222197599 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f0fde0 0x55a579f1ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76e9400640 1 -- 172.21.15.40:0/4222197599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f15940 msgr2=0x55a579f20240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:04:53.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76e9400640 1 --2- 172.21.15.40:0/4222197599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f15940 0x55a579f20240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.693+0000 7f76e9400640 1 -- 172.21.15.40:0/4222197599 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a579e2ecb0 con 0x55a579f107d0 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.694+0000 7f76e9400640 1 --2- 172.21.15.40:0/4222197599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 0x55a579f1f280 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f76d4098ed0 tx=0x7f76d40ce260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44472a80275315a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.694+0000 7f76c7400640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.694+0000 7f76c7400640 1 -- 172.21.15.40:0/4222197599 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d408c060 con 0x55a579f107d0 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.694+0000 7f76c7400640 1 -- 172.21.15.40:0/4222197599 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f76d40990e0 con 0x55a579f107d0 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.694+0000 7f76ef5ff740 1 -- 172.21.15.40:0/4222197599 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76d409ca70 con 0x55a579f107d0 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.694+0000 7f76c7400640 1 -- 172.21.15.40:0/4222197599 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f76d4005180 con 0x55a579f107d0 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.694+0000 7f76ef5ff740 10 client.25654 fetch_fsmap learned FSMap version 24 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.694+0000 7f76ef5ff740 1 -- 172.21.15.40:0/4222197599 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a579f8c070 con 0x55a579f107d0 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.695+0000 7f76c7400640 1 -- 172.21.15.40:0/4222197599 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76d409ca70 con 0x55a579f107d0 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.695+0000 7f76c7400640 1 -- 172.21.15.40:0/4222197599 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f76d40987c0 con 0x55a579f107d0 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.695+0000 7f76ef5ff740 10 client.25654 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:04:53.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.695+0000 7f76ef5ff740 10 client.25654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:04:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.695+0000 7f76ef5ff740 20 client.25654 populate_metadata read hostname 'smithi040' 2024-04-27T00:04:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.695+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/4222197599 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a579f8c710 0x55a579f8eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:04:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.695+0000 7f76ef5ff740 4 client.25654 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:04:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.695+0000 7f76ef5ff740 1 -- 172.21.15.40:0/4222197599 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a579e7be00 con 0x55a579f8c710 2024-04-27T00:04:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.695+0000 7f76e9e00640 1 --2- 172.21.15.40:0/4222197599 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a579f8c710 0x55a579f8eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:04:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.696+0000 7f76e9e00640 1 --2- 172.21.15.40:0/4222197599 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a579f8c710 0x55a579f8eb30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3a427aecdc78cd server_cookie=ba75968312055751 in_seq=0 out_seq=0 2024-04-27T00:04:53.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.696+0000 7f76c7400640 10 client.25654 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.710+0000 7f76c7400640 1 -- 172.21.15.40:0/4222197599 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a579e7be00 con 0x55a579f8c710 2024-04-27T00:04:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.710+0000 7f76c7400640 10 client.25654 handle_command_reply: tid=0 2024-04-27T00:04:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.710+0000 7f76ef5ff740 1 client.25654 shutdown 2024-04-27T00:04:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.710+0000 7f76ef5ff740 1 -- 172.21.15.40:0/4222197599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 msgr2=0x55a579f1f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.710+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/4222197599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 0x55a579f1f280 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f76d4098ed0 tx=0x7f76d40ce260 comp rx=0 tx=0).stop 2024-04-27T00:04:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.710+0000 7f76ef5ff740 1 -- 172.21.15.40:0/4222197599 shutdown_connections 2024-04-27T00:04:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.710+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/4222197599 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55a579f8c710 0x55a579f8eb30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.710+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/4222197599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a579f15940 0x55a579f20240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.710+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/4222197599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a579f107d0 0x55a579f1f280 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.710+0000 7f76ef5ff740 1 --2- 172.21.15.40:0/4222197599 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55a579f0fde0 0x55a579f1ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.710+0000 7f76ef5ff740 1 -- 172.21.15.40:0/4222197599 >> 172.21.15.40:0/4222197599 conn(0x55a579f0b450 msgr2=0x55a579f09dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:53.712 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.710+0000 7f76c7400640 0 client.25654 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:04:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.711+0000 7f76ef5ff740 1 -- 172.21.15.40:0/4222197599 shutdown_connections 2024-04-27T00:04:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.711+0000 7f76ef5ff740 1 -- 172.21.15.40:0/4222197599 wait complete. 2024-04-27T00:04:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.711+0000 7f76ef5ff740 20 client.25654 trim_cache size 0 max 16384 2024-04-27T00:04:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:04:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:04:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:04:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:04:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.712+0000 7f76eb200640 1 -- 172.21.15.40:0/731052878 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f76cc0a3cb0 msgr2=0x7f76cc0a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.712+0000 7f76eb200640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f76cc0a3cb0 0x7f76cc0a6170 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f76d8005ab0 tx=0x7f76d8005a20 comp rx=0 tx=0).stop 2024-04-27T00:04:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.712+0000 7f76eb200640 1 -- 172.21.15.40:0/731052878 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f76e4109fd0 msgr2=0x7f76e419f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:04:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.712+0000 7f76eb200640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f76e4109fd0 0x7f76e419f540 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f76d400e9f0 tx=0x7f76d400edf0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.712+0000 7f76eb200640 1 -- 172.21.15.40:0/731052878 shutdown_connections 2024-04-27T00:04:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.712+0000 7f76eb200640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76e410a990 0x7f76e41a65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.714 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.712+0000 7f76eb200640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f76e4109fd0 0x7f76e419f540 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.712+0000 7f76eb200640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f76cc0a3cb0 0x7f76cc0a6170 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.712+0000 7f76eb200640 1 --2- 172.21.15.40:0/731052878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76e4108de0 0x7f76e419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:04:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.712+0000 7f76eb200640 1 -- 172.21.15.40:0/731052878 >> 172.21.15.40:0/731052878 conn(0x7f76e4077dc0 msgr2=0x7f76e410d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:04:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.712+0000 7f76eb200640 1 -- 172.21.15.40:0/731052878 shutdown_connections 2024-04-27T00:04:53.715 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:04:53.712+0000 7f76eb200640 1 -- 172.21.15.40:0/731052878 wait complete. 2024-04-27T00:04:54.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:54 smithi080 ceph-mon[32362]: pgmap v508: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:54.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:54 smithi103 ceph-mon[32952]: pgmap v508: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:54.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:54 smithi040 ceph-mon[28095]: pgmap v508: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:56.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:56 smithi080 ceph-mon[32362]: pgmap v509: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:04:56.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:56 smithi103 ceph-mon[32952]: pgmap v509: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:04:56.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:56 smithi040 ceph-mon[28095]: pgmap v509: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:04:58.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:04:58 smithi103 ceph-mon[32952]: pgmap v510: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:58.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:04:58 smithi040 ceph-mon[28095]: pgmap v510: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:04:58.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:04:58 smithi080 ceph-mon[32362]: pgmap v510: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:00.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:00 smithi103 ceph-mon[32952]: pgmap v511: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:00.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:00 smithi040 ceph-mon[28095]: pgmap v511: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:00.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:00 smithi080 ceph-mon[32362]: pgmap v511: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:02.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:02 smithi103 ceph-mon[32952]: pgmap v512: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:02.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:02 smithi040 ceph-mon[28095]: pgmap v512: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:02.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:02 smithi080 ceph-mon[32362]: pgmap v512: 129 pgs: 129 active+clean; 1.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:04.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:04 smithi103 ceph-mon[32952]: pgmap v513: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:05:04.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:04 smithi040 ceph-mon[28095]: pgmap v513: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:05:04.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:04 smithi080 ceph-mon[32362]: pgmap v513: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:05:06.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:06 smithi103 ceph-mon[32952]: pgmap v514: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:05:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:06 smithi040 ceph-mon[28095]: pgmap v514: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:05:06.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:06 smithi080 ceph-mon[32362]: pgmap v514: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:05:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.864+0000 7f57ab800640 1 -- 172.21.15.40:0/1219790428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a4101270 msgr2=0x7f57a4101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.864+0000 7f57ab800640 1 --2- 172.21.15.40:0/1219790428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a4101270 0x7f57a4101690 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5798009860 tx=0x7f579802a3d0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.865+0000 7f57ab800640 1 -- 172.21.15.40:0/1219790428 shutdown_connections 2024-04-27T00:05:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.865+0000 7f57ab800640 1 --2- 172.21.15.40:0/1219790428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a4102570 0x7f57a41029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.865+0000 7f57ab800640 1 --2- 172.21.15.40:0/1219790428 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57a4101bd0 0x7f57a4102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.865+0000 7f57ab800640 1 --2- 172.21.15.40:0/1219790428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a4101270 0x7f57a4101690 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.865+0000 7f57ab800640 1 -- 172.21.15.40:0/1219790428 >> 172.21.15.40:0/1219790428 conn(0x7f57a40fcdf0 msgr2=0x7f57a40ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.865+0000 7f57ab800640 1 -- 172.21.15.40:0/1219790428 shutdown_connections 2024-04-27T00:05:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.865+0000 7f57ab800640 1 -- 172.21.15.40:0/1219790428 wait complete. 2024-04-27T00:05:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.866+0000 7f57ab800640 1 Processor -- start 2024-04-27T00:05:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.866+0000 7f57ab800640 1 -- start start 2024-04-27T00:05:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.866+0000 7f57ab800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a4101270 0x7f57a419acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.866+0000 7f57ab800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57a4101bd0 0x7f57a419b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.866+0000 7f57ab800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a4102570 0x7f57a41a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.866+0000 7f57ab800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57a41106f0 con 0x7f57a4101270 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.866+0000 7f57ab800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f57a4110570 con 0x7f57a4101bd0 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.866+0000 7f57ab800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57a4110870 con 0x7f57a4102570 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.867+0000 7f57a3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57a4101bd0 0x7f57a419b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.867+0000 7f57a8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a4101270 0x7f57a419acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.867+0000 7f57a3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57a4101bd0 0x7f57a419b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:39066/0 (socket says 172.21.15.40:39066) 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.867+0000 7f57a3e00640 1 -- 172.21.15.40:0/3347732837 learned_addr learned my addr 172.21.15.40:0/3347732837 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.867+0000 7f57a9800640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a4102570 0x7f57a41a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.867+0000 7f57a3e00640 1 -- 172.21.15.40:0/3347732837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a4102570 msgr2=0x7f57a41a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.867+0000 7f57a3e00640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a4102570 0x7f57a41a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.867+0000 7f57a3e00640 1 -- 172.21.15.40:0/3347732837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a4101270 msgr2=0x7f57a419acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.867+0000 7f57a3e00640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a4101270 0x7f57a419acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.867+0000 7f57a3e00640 1 -- 172.21.15.40:0/3347732837 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a41a2900 con 0x7f57a4101bd0 2024-04-27T00:05:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.868+0000 7f57a3e00640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57a4101bd0 0x7f57a419b230 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f579000e9c0 tx=0x7f579000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.868+0000 7f57a1600640 1 -- 172.21.15.40:0/3347732837 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579000cde0 con 0x7f57a4101bd0 2024-04-27T00:05:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.868+0000 7f57ab800640 1 -- 172.21.15.40:0/3347732837 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57a41a2bf0 con 0x7f57a4101bd0 2024-04-27T00:05:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.868+0000 7f57a1600640 1 -- 172.21.15.40:0/3347732837 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57900035f0 con 0x7f57a4101bd0 2024-04-27T00:05:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.868+0000 7f57a1600640 1 -- 172.21.15.40:0/3347732837 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5790010690 con 0x7f57a4101bd0 2024-04-27T00:05:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.868+0000 7f57ab800640 1 -- 172.21.15.40:0/3347732837 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57a41a3140 con 0x7f57a4101bd0 2024-04-27T00:05:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.870+0000 7f57ae615740 1 Processor -- start 2024-04-27T00:05:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.870+0000 7f57ae615740 1 -- start start 2024-04-27T00:05:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a1600640 1 -- 172.21.15.40:0/3347732837 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5790004000 con 0x7f57a4101bd0 2024-04-27T00:05:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57ae615740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a51bd0 0x55b241a51ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57ae615740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a525c0 0x55b241a570c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57ae615740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a57730 0x55b241a59bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57ae615740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b2419b6380 con 0x55b241a525c0 2024-04-27T00:05:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57ae615740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55b241952c30 con 0x55b241a57730 2024-04-27T00:05:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57ae615740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b2419b2340 con 0x55b241a51bd0 2024-04-27T00:05:06.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a9800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a51bd0 0x55b241a51ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a9800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a51bd0 0x55b241a51ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:58162/0 (socket says 172.21.15.40:58162) 2024-04-27T00:05:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a9800640 1 -- 172.21.15.40:0/3489441953 learned_addr learned my addr 172.21.15.40:0/3489441953 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a8e00640 1 --2- 172.21.15.40:0/3489441953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a525c0 0x55b241a570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a3e00640 1 --2- 172.21.15.40:0/3489441953 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a57730 0x55b241a59bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a9800640 1 -- 172.21.15.40:0/3489441953 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a57730 msgr2=0x55b241a59bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a9800640 1 --2- 172.21.15.40:0/3489441953 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a57730 0x55b241a59bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a9800640 1 -- 172.21.15.40:0/3489441953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a525c0 msgr2=0x55b241a570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a9800640 1 --2- 172.21.15.40:0/3489441953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a525c0 0x55b241a570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a9800640 1 -- 172.21.15.40:0/3489441953 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b241974940 con 0x55b241a51bd0 2024-04-27T00:05:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a8e00640 1 --2- 172.21.15.40:0/3489441953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a525c0 0x55b241a570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:05:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a9800640 1 --2- 172.21.15.40:0/3489441953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a51bd0 0x55b241a51ff0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f579c009fd0 tx=0x7f579c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af1a29d400fa5415 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f5796000640 1 -- 172.21.15.40:0/3489441953 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c004d20 con 0x55b241a51bd0 2024-04-27T00:05:06.876 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f5796000640 1 -- 172.21.15.40:0/3489441953 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f579c004050 con 0x55b241a51bd0 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f5796000640 1 -- 172.21.15.40:0/3489441953 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c005280 con 0x55b241a51bd0 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.872+0000 7f57a1600640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f578c0a3cf0 0x7f578c0a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.874+0000 7f57a1600640 1 -- 172.21.15.40:0/3347732837 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f57900d0d10 con 0x7f57a4101bd0 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.874+0000 7f57a8e00640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f578c0a3cf0 0x7f578c0a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.874+0000 7f57ae615740 1 -- 172.21.15.40:0/3489441953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a51bd0 msgr2=0x55b241a51ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.874+0000 7f57ae615740 1 --2- 172.21.15.40:0/3489441953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a51bd0 0x55b241a51ff0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f579c009fd0 tx=0x7f579c0118d0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.874+0000 7f57ae615740 1 -- 172.21.15.40:0/3489441953 shutdown_connections 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.874+0000 7f57ae615740 1 --2- 172.21.15.40:0/3489441953 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a57730 0x55b241a59bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.874+0000 7f57ae615740 1 --2- 172.21.15.40:0/3489441953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a525c0 0x55b241a570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.874+0000 7f57ae615740 1 --2- 172.21.15.40:0/3489441953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a51bd0 0x55b241a51ff0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.874+0000 7f57ae615740 1 -- 172.21.15.40:0/3489441953 >> 172.21.15.40:0/3489441953 conn(0x55b241a4d240 msgr2=0x55b241a4d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.874+0000 7f57ae615740 1 -- 172.21.15.40:0/3489441953 shutdown_connections 2024-04-27T00:05:06.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.874+0000 7f57ae615740 1 -- 172.21.15.40:0/3489441953 wait complete. 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.874+0000 7f57a8e00640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f578c0a3cf0 0x7f578c0a61b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f579802a9c0 tx=0x7f5798030980 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57ae615740 1 -- start start 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57ae615740 1 Processor -- start 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57ae615740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a51bd0 0x55b241a5bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57ae615740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 0x55b241a5c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57ae615740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a57730 0x55b241a63360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57ae615740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b241a5a6e0 con 0x55b241a51bd0 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57ae615740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55b24193dc60 con 0x55b241a57730 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57ae615740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b241a5a860 con 0x55b241a525c0 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57ae615740 10 client.0 _ll_register_callbacks cb 0x55b241969e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57a8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a51bd0 0x55b241a5bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57a3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 0x55b241a5c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57a3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 0x55b241a5c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:58174/0 (socket says 172.21.15.40:58174) 2024-04-27T00:05:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57a3e00640 1 -- 172.21.15.40:0/1859232183 learned_addr learned my addr 172.21.15.40:0/1859232183 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57a9800640 1 --2- 172.21.15.40:0/1859232183 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a57730 0x55b241a63360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57a3e00640 1 -- 172.21.15.40:0/1859232183 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a57730 msgr2=0x55b241a63360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57a3e00640 1 --2- 172.21.15.40:0/1859232183 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a57730 0x55b241a63360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57a3e00640 1 -- 172.21.15.40:0/1859232183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a51bd0 msgr2=0x55b241a5bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57a3e00640 1 --2- 172.21.15.40:0/1859232183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a51bd0 0x55b241a5bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.879 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57a3e00640 1 -- 172.21.15.40:0/1859232183 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b24194c510 con 0x55b241a525c0 2024-04-27T00:05:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f57a3e00640 1 --2- 172.21.15.40:0/1859232183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 0x55b241a5c320 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5790098e60 tx=0x7f5790003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bab7b518fac8f0f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.875+0000 7f578aa00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:05:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.877+0000 7f578aa00640 1 -- 172.21.15.40:0/1859232183 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579009abd0 con 0x55b241a525c0 2024-04-27T00:05:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.877+0000 7f578aa00640 1 -- 172.21.15.40:0/1859232183 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f57900094f0 con 0x55b241a525c0 2024-04-27T00:05:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.877+0000 7f57ae615740 1 -- 172.21.15.40:0/1859232183 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f579c004050 con 0x55b241a525c0 2024-04-27T00:05:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.877+0000 7f578aa00640 1 -- 172.21.15.40:0/1859232183 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f57900042f0 con 0x55b241a525c0 2024-04-27T00:05:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.877+0000 7f57ae615740 10 client.25666 fetch_fsmap learned FSMap version 24 2024-04-27T00:05:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.877+0000 7f57ae615740 1 -- 172.21.15.40:0/1859232183 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b2416c8210 con 0x55b241a525c0 2024-04-27T00:05:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.877+0000 7f578aa00640 1 -- 172.21.15.40:0/1859232183 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f579c004050 con 0x55b241a525c0 2024-04-27T00:05:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.878+0000 7f578aa00640 1 -- 172.21.15.40:0/1859232183 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f57900d5040 con 0x55b241a525c0 2024-04-27T00:05:06.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.878+0000 7f57ae615740 10 client.25666 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.878+0000 7f57ae615740 10 client.25666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.878+0000 7f57ae615740 20 client.25666 populate_metadata read hostname 'smithi040' 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.878+0000 7f57ae615740 1 --2- 172.21.15.40:0/1859232183 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b241a67900 0x55b241a69d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.878+0000 7f57ae615740 4 client.25666 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.878+0000 7f57ae615740 1 -- 172.21.15.40:0/1859232183 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2419c3f90 con 0x55b241a67900 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.878+0000 7f57a8e00640 1 --2- 172.21.15.40:0/1859232183 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b241a67900 0x55b241a69d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.879+0000 7f57a8e00640 1 --2- 172.21.15.40:0/1859232183 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b241a67900 0x55b241a69d70 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c2e302c2801c575 server_cookie=16340d3c0dd1dfc5 in_seq=0 out_seq=0 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.879+0000 7f578aa00640 10 client.25666 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.879+0000 7f578aa00640 1 -- 172.21.15.40:0/1859232183 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b2419c3f90 con 0x55b241a67900 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.879+0000 7f578aa00640 10 client.25666 handle_command_reply: tid=0 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f57ae615740 1 client.25666 shutdown 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f57ae615740 1 -- 172.21.15.40:0/1859232183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 msgr2=0x55b241a5c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f57ae615740 1 --2- 172.21.15.40:0/1859232183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 0x55b241a5c320 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5790098e60 tx=0x7f5790003cd0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f57ae615740 1 -- 172.21.15.40:0/1859232183 shutdown_connections 2024-04-27T00:05:06.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f57ae615740 1 --2- 172.21.15.40:0/1859232183 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b241a67900 0x55b241a69d70 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f57ae615740 1 --2- 172.21.15.40:0/1859232183 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a57730 0x55b241a63360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f57ae615740 1 --2- 172.21.15.40:0/1859232183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 0x55b241a5c320 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f57ae615740 1 --2- 172.21.15.40:0/1859232183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a51bd0 0x55b241a5bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f57ae615740 1 -- 172.21.15.40:0/1859232183 >> 172.21.15.40:0/1859232183 conn(0x55b241a4d240 msgr2=0x55b241a4c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f578aa00640 0 client.25666 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f57ae615740 1 -- 172.21.15.40:0/1859232183 shutdown_connections 2024-04-27T00:05:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f57ae615740 1 -- 172.21.15.40:0/1859232183 wait complete. 2024-04-27T00:05:06.884 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.880+0000 7f57ae615740 20 client.25666 trim_cache size 0 max 16384 2024-04-27T00:05:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.896+0000 7f57ae615740 1 Processor -- start 2024-04-27T00:05:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.896+0000 7f57ae615740 1 -- start start 2024-04-27T00:05:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.896+0000 7f57ae615740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a51bd0 0x55b241a5bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.896+0000 7f57ae615740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a525c0 0x55b241a63320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.896+0000 7f57ae615740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a57730 0x55b241a5c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.896+0000 7f57ae615740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57a41afed0 con 0x55b241a51bd0 2024-04-27T00:05:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.896+0000 7f57ae615740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f57a41ad7a0 con 0x55b241a525c0 2024-04-27T00:05:06.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.896+0000 7f57ae615740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57a41ab5f0 con 0x55b241a57730 2024-04-27T00:05:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.896+0000 7f57a8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a51bd0 0x55b241a5bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.896+0000 7f57a8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a51bd0 0x55b241a5bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33328/0 (socket says 172.21.15.40:33328) 2024-04-27T00:05:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.896+0000 7f57a8e00640 1 -- 172.21.15.40:0/771873138 learned_addr learned my addr 172.21.15.40:0/771873138 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.898+0000 7f57a3e00640 1 --2- 172.21.15.40:0/771873138 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a525c0 0x55b241a63320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.898+0000 7f57a9800640 1 --2- 172.21.15.40:0/771873138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a57730 0x55b241a5c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.898+0000 7f57a8e00640 1 -- 172.21.15.40:0/771873138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a57730 msgr2=0x55b241a5c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.898+0000 7f57a8e00640 1 --2- 172.21.15.40:0/771873138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a57730 0x55b241a5c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.898+0000 7f57a8e00640 1 -- 172.21.15.40:0/771873138 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a525c0 msgr2=0x55b241a63320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.898+0000 7f57a8e00640 1 --2- 172.21.15.40:0/771873138 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a525c0 0x55b241a63320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.898+0000 7f57a8e00640 1 -- 172.21.15.40:0/771873138 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b241794620 con 0x55b241a51bd0 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.898+0000 7f57a3e00640 1 --2- 172.21.15.40:0/771873138 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a525c0 0x55b241a63320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.898+0000 7f57a8e00640 1 --2- 172.21.15.40:0/771873138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a51bd0 0x55b241a5bde0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5798009780 tx=0x7f5798041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52defed790df4c65 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.899+0000 7f5795600640 1 -- 172.21.15.40:0/771873138 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5798005510 con 0x55b241a51bd0 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.899+0000 7f5795600640 1 -- 172.21.15.40:0/771873138 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5798053450 con 0x55b241a51bd0 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.899+0000 7f5795600640 1 -- 172.21.15.40:0/771873138 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5798004920 con 0x55b241a51bd0 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.899+0000 7f57ae615740 1 -- 172.21.15.40:0/771873138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a51bd0 msgr2=0x55b241a5bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.899+0000 7f57ae615740 1 --2- 172.21.15.40:0/771873138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a51bd0 0x55b241a5bde0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5798009780 tx=0x7f5798041000 comp rx=0 tx=0).stop 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.899+0000 7f57ae615740 1 -- 172.21.15.40:0/771873138 shutdown_connections 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.899+0000 7f57ae615740 1 --2- 172.21.15.40:0/771873138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a57730 0x55b241a5c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.899+0000 7f57ae615740 1 --2- 172.21.15.40:0/771873138 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a525c0 0x55b241a63320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.899+0000 7f57ae615740 1 --2- 172.21.15.40:0/771873138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a51bd0 0x55b241a5bde0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.899+0000 7f57ae615740 1 -- 172.21.15.40:0/771873138 >> 172.21.15.40:0/771873138 conn(0x55b241a4d240 msgr2=0x55b241a4d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.899+0000 7f57ae615740 1 -- 172.21.15.40:0/771873138 shutdown_connections 2024-04-27T00:05:06.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.900+0000 7f57ae615740 1 -- 172.21.15.40:0/771873138 wait complete. 2024-04-27T00:05:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.900+0000 7f57ae615740 1 -- start start 2024-04-27T00:05:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.900+0000 7f57ae615740 1 Processor -- start 2024-04-27T00:05:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.900+0000 7f57ae615740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a51bd0 0x55b241a603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.900+0000 7f57ae615740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 0x55b241a60910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.900+0000 7f57ae615740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a57730 0x55b241acbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.900+0000 7f57ae615740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57a41abc60 con 0x55b241a57730 2024-04-27T00:05:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57ae615740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f57a41ad0d0 con 0x55b241a51bd0 2024-04-27T00:05:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57ae615740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57a41ac330 con 0x55b241a525c0 2024-04-27T00:05:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57ae615740 10 client.0 _ll_register_callbacks cb 0x7f57900046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57a9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a57730 0x55b241acbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57a9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a57730 0x55b241acbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33330/0 (socket says 172.21.15.40:33330) 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57a8e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a51bd0 0x55b241a603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57a9800640 1 -- 172.21.15.40:0/1237225732 learned_addr learned my addr 172.21.15.40:0/1237225732 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57a3e00640 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 0x55b241a60910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57a9800640 1 -- 172.21.15.40:0/1237225732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 msgr2=0x55b241a60910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57a9800640 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 0x55b241a60910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57a9800640 1 -- 172.21.15.40:0/1237225732 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a51bd0 msgr2=0x55b241a603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57a9800640 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a51bd0 0x55b241a603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57a9800640 1 -- 172.21.15.40:0/1237225732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b241970cb0 con 0x55b241a57730 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57a8e00640 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a51bd0 0x55b241a603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.901+0000 7f57a3e00640 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 0x55b241a60910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.902+0000 7f57a9800640 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a57730 0x55b241acbee0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f579c009f70 tx=0x7f579c003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441274db9f23665f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.902+0000 7f578aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.902+0000 7f578aa00640 1 -- 172.21.15.40:0/1237225732 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c005250 con 0x55b241a57730 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.902+0000 7f57ae615740 1 -- 172.21.15.40:0/1237225732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5798053450 con 0x55b241a57730 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.902+0000 7f578aa00640 1 -- 172.21.15.40:0/1237225732 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f579c0053f0 con 0x55b241a57730 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.902+0000 7f578aa00640 1 -- 172.21.15.40:0/1237225732 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f579c035050 con 0x55b241a57730 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.902+0000 7f57ae615740 10 client.15531 fetch_fsmap learned FSMap version 24 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.902+0000 7f57ae615740 1 -- 172.21.15.40:0/1237225732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b241acd5e0 con 0x55b241a57730 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.903+0000 7f578aa00640 1 -- 172.21.15.40:0/1237225732 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5798053450 con 0x55b241a57730 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.903+0000 7f578aa00640 1 -- 172.21.15.40:0/1237225732 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f579c019430 con 0x55b241a57730 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.903+0000 7f57ae615740 10 client.15531 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.903+0000 7f57ae615740 10 client.15531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.903+0000 7f57ae615740 20 client.15531 populate_metadata read hostname 'smithi040' 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.903+0000 7f57ae615740 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b241acddd0 0x55b241ad01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.903+0000 7f57ae615740 4 client.15531 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:05:06.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.903+0000 7f57ae615740 1 -- 172.21.15.40:0/1237225732 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b2419bde00 con 0x55b241acddd0 2024-04-27T00:05:06.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.903+0000 7f57a8e00640 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b241acddd0 0x55b241ad01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:06.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.904+0000 7f57a8e00640 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b241acddd0 0x55b241ad01f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54598ed6b5fcac45 server_cookie=420a449a5813ecda in_seq=0 out_seq=0 2024-04-27T00:05:06.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.904+0000 7f578aa00640 10 client.15531 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f578aa00640 1 -- 172.21.15.40:0/1237225732 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b2419bde00 con 0x55b241acddd0 2024-04-27T00:05:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f578aa00640 10 client.15531 handle_command_reply: tid=0 2024-04-27T00:05:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ae615740 1 client.15531 shutdown 2024-04-27T00:05:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ae615740 1 -- 172.21.15.40:0/1237225732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a57730 msgr2=0x55b241acbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:06.918 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ae615740 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a57730 0x55b241acbee0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f579c009f70 tx=0x7f579c003da0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ae615740 1 -- 172.21.15.40:0/1237225732 shutdown_connections 2024-04-27T00:05:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ae615740 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55b241acddd0 0x55b241ad01f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ae615740 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b241a57730 0x55b241acbee0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ae615740 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b241a525c0 0x55b241a60910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ae615740 1 --2- 172.21.15.40:0/1237225732 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55b241a51bd0 0x55b241a603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f578aa00640 0 client.15531 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ae615740 1 -- 172.21.15.40:0/1237225732 >> 172.21.15.40:0/1237225732 conn(0x55b241a4d240 msgr2=0x55b241a5bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ae615740 1 -- 172.21.15.40:0/1237225732 shutdown_connections 2024-04-27T00:05:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ae615740 1 -- 172.21.15.40:0/1237225732 wait complete. 2024-04-27T00:05:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ae615740 20 client.15531 trim_cache size 0 max 16384 2024-04-27T00:05:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:05:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:05:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:05:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:05:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ab800640 1 -- 172.21.15.40:0/3347732837 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f578c0a3cf0 msgr2=0x7f578c0a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ab800640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f578c0a3cf0 0x7f578c0a61b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f579802a9c0 tx=0x7f5798030980 comp rx=0 tx=0).stop 2024-04-27T00:05:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ab800640 1 -- 172.21.15.40:0/3347732837 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57a4101bd0 msgr2=0x7f57a419b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ab800640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57a4101bd0 0x7f57a419b230 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f579000e9c0 tx=0x7f579000ee80 comp rx=0 tx=0).stop 2024-04-27T00:05:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ab800640 1 -- 172.21.15.40:0/3347732837 shutdown_connections 2024-04-27T00:05:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ab800640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f578c0a3cf0 0x7f578c0a61b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ab800640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a4102570 0x7f57a41a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ab800640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f57a4101bd0 0x7f57a419b230 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ab800640 1 --2- 172.21.15.40:0/3347732837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a4101270 0x7f57a419acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.918+0000 7f57ab800640 1 -- 172.21.15.40:0/3347732837 >> 172.21.15.40:0/3347732837 conn(0x7f57a40fcdf0 msgr2=0x7f57a40fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.920+0000 7f57ab800640 1 -- 172.21.15.40:0/3347732837 shutdown_connections 2024-04-27T00:05:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:06.920+0000 7f57ab800640 1 -- 172.21.15.40:0/3347732837 wait complete. 2024-04-27T00:05:08.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:08 smithi103 ceph-mon[32952]: pgmap v515: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:05:08.351 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:08 smithi080 ceph-mon[32362]: pgmap v515: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:05:08.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:08 smithi040 ceph-mon[28095]: pgmap v515: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:05:10.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:10 smithi103 ceph-mon[32952]: pgmap v516: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-27T00:05:10.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:10 smithi040 ceph-mon[28095]: pgmap v516: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-27T00:05:10.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:10 smithi080 ceph-mon[32362]: pgmap v516: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-27T00:05:12.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:12 smithi103 ceph-mon[32952]: pgmap v517: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-27T00:05:12.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:12 smithi040 ceph-mon[28095]: pgmap v517: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-27T00:05:12.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:12 smithi080 ceph-mon[32362]: pgmap v517: 129 pgs: 129 active+clean; 1.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-27T00:05:14.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:14 smithi103 ceph-mon[32952]: pgmap v518: 129 pgs: 129 active+clean; 2.0 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:05:14.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:14 smithi040 ceph-mon[28095]: pgmap v518: 129 pgs: 129 active+clean; 2.0 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:05:14.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:14 smithi080 ceph-mon[32362]: pgmap v518: 129 pgs: 129 active+clean; 2.0 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:05:16.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:16 smithi040 ceph-mon[28095]: pgmap v519: 129 pgs: 129 active+clean; 2.0 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:05:16.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:16 smithi080 ceph-mon[32362]: pgmap v519: 129 pgs: 129 active+clean; 2.0 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:05:16.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:16 smithi103 ceph-mon[32952]: pgmap v519: 129 pgs: 129 active+clean; 2.0 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:05:18.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:18 smithi040 ceph-mon[28095]: pgmap v520: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:05:18.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:18 smithi080 ceph-mon[32362]: pgmap v520: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:05:18.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:18 smithi103 ceph-mon[32952]: pgmap v520: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:05:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.067+0000 7f7059a00640 1 -- 172.21.15.40:0/1068998298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7054108550 msgr2=0x7f705410a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.067+0000 7f7059a00640 1 --2- 172.21.15.40:0/1068998298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7054108550 0x7f705410a940 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f704c009900 tx=0x7f704c02a3f0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.067+0000 7f7059a00640 1 -- 172.21.15.40:0/1068998298 shutdown_connections 2024-04-27T00:05:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.067+0000 7f7059a00640 1 --2- 172.21.15.40:0/1068998298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7054108550 0x7f705410a940 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.067+0000 7f7059a00640 1 --2- 172.21.15.40:0/1068998298 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7054105c20 0x7f7054108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.067+0000 7f7059a00640 1 --2- 172.21.15.40:0/1068998298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70541032f0 0x7f70541056e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.067+0000 7f7059a00640 1 -- 172.21.15.40:0/1068998298 >> 172.21.15.40:0/1068998298 conn(0x7f70540fcc20 msgr2=0x7f70540ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.067+0000 7f7059a00640 1 -- 172.21.15.40:0/1068998298 shutdown_connections 2024-04-27T00:05:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.068+0000 7f7059a00640 1 -- 172.21.15.40:0/1068998298 wait complete. 2024-04-27T00:05:20.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.068+0000 7f7059a00640 1 Processor -- start 2024-04-27T00:05:20.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.069+0000 7f7059a00640 1 -- start start 2024-04-27T00:05:20.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.069+0000 7f7059a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f70541032f0 0x7f705419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.069+0000 7f7059a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7054105c20 0x7f705419b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.069+0000 7f7059a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7054108550 0x7f70541a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.069+0000 7f7059a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f705410e340 con 0x7f7054108550 2024-04-27T00:05:20.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.069+0000 7f7059a00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f705410e1c0 con 0x7f70541032f0 2024-04-27T00:05:20.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.069+0000 7f7059a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f705410e4c0 con 0x7f7054105c20 2024-04-27T00:05:20.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.069+0000 7f7059000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7054108550 0x7f70541a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.069+0000 7f7053e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f70541032f0 0x7f705419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.069+0000 7f7053400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7054105c20 0x7f705419b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.069+0000 7f7053e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f70541032f0 0x7f705419abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:48668/0 (socket says 172.21.15.40:48668) 2024-04-27T00:05:20.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.069+0000 7f7053e00640 1 -- 172.21.15.40:0/3642343694 learned_addr learned my addr 172.21.15.40:0/3642343694 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:20.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.070+0000 7f7053e00640 1 -- 172.21.15.40:0/3642343694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7054105c20 msgr2=0x7f705419b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:20.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.070+0000 7f7053e00640 1 --2- 172.21.15.40:0/3642343694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7054105c20 0x7f705419b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.070+0000 7f7053e00640 1 -- 172.21.15.40:0/3642343694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7054108550 msgr2=0x7f70541a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:20.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.070+0000 7f7053e00640 1 --2- 172.21.15.40:0/3642343694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7054108550 0x7f70541a2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.070+0000 7f7053e00640 1 -- 172.21.15.40:0/3642343694 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70541a27d0 con 0x7f70541032f0 2024-04-27T00:05:20.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.070+0000 7f7053e00640 1 --2- 172.21.15.40:0/3642343694 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f70541032f0 0x7f705419abb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f704400e9f0 tx=0x7f704400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:20.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.071+0000 7f7050c00640 1 -- 172.21.15.40:0/3642343694 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704400cd80 con 0x7f70541032f0 2024-04-27T00:05:20.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.071+0000 7f7050c00640 1 -- 172.21.15.40:0/3642343694 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70440108e0 con 0x7f70541032f0 2024-04-27T00:05:20.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.071+0000 7f7059a00640 1 -- 172.21.15.40:0/3642343694 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70541a2ac0 con 0x7f70541032f0 2024-04-27T00:05:20.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.071+0000 7f7050c00640 1 -- 172.21.15.40:0/3642343694 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044018980 con 0x7f70541032f0 2024-04-27T00:05:20.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.071+0000 7f7059a00640 1 -- 172.21.15.40:0/3642343694 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70541a3010 con 0x7f70541032f0 2024-04-27T00:05:20.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.075+0000 7f705dd88740 1 Processor -- start 2024-04-27T00:05:20.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.075+0000 7f705dd88740 1 -- start start 2024-04-27T00:05:20.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.075+0000 7f705dd88740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea30de0 0x558b0ea31200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.075+0000 7f705dd88740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea317d0 0x558b0ea362d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f705dd88740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea36940 0x558b0ea38dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f705dd88740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558b0e995380 con 0x558b0ea317d0 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f705dd88740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x558b0e931c30 con 0x558b0ea36940 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f705dd88740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558b0e991340 con 0x558b0ea30de0 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f7050c00640 1 -- 172.21.15.40:0/3642343694 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7044010430 con 0x7f70541032f0 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f7059000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea30de0 0x558b0ea31200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f7059000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea30de0 0x558b0ea31200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46400/0 (socket says 172.21.15.40:46400) 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f7053400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea36940 0x558b0ea38dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f7059000640 1 -- 172.21.15.40:0/1974975610 learned_addr learned my addr 172.21.15.40:0/1974975610 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f7053e00640 1 --2- 172.21.15.40:0/1974975610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea317d0 0x558b0ea362d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f7053400640 1 -- 172.21.15.40:0/1974975610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea30de0 msgr2=0x558b0ea31200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f7053400640 1 --2- 172.21.15.40:0/1974975610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea30de0 0x558b0ea31200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f7053400640 1 -- 172.21.15.40:0/1974975610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea317d0 msgr2=0x558b0ea362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:20.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f7053400640 1 --2- 172.21.15.40:0/1974975610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea317d0 0x558b0ea362d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.076+0000 7f7053400640 1 -- 172.21.15.40:0/1974975610 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b0e953940 con 0x558b0ea36940 2024-04-27T00:05:20.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.077+0000 7f7059000640 1 --2- 172.21.15.40:0/1974975610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea30de0 0x558b0ea31200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:05:20.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.077+0000 7f7053e00640 1 --2- 172.21.15.40:0/1974975610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea317d0 0x558b0ea362d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:05:20.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.077+0000 7f7053400640 1 --2- 172.21.15.40:0/1974975610 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea36940 0x558b0ea38dd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f705406aa80 tx=0x7f703c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53c62a4b9eee38da server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:20.082 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.077+0000 7f7049600640 1 -- 172.21.15.40:0/1974975610 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c003c30 con 0x558b0ea36940 2024-04-27T00:05:20.082 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.077+0000 7f7049600640 1 -- 172.21.15.40:0/1974975610 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f703c003dd0 con 0x558b0ea36940 2024-04-27T00:05:20.082 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.077+0000 7f7049600640 1 -- 172.21.15.40:0/1974975610 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c00fa80 con 0x558b0ea36940 2024-04-27T00:05:20.082 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.077+0000 7f7050c00640 1 --2- 172.21.15.40:0/3642343694 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f70340a3c30 0x7f70340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.082 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.078+0000 7f7050c00640 1 -- 172.21.15.40:0/3642343694 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f70440d0c00 con 0x7f70541032f0 2024-04-27T00:05:20.082 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.078+0000 7f7059000640 1 --2- 172.21.15.40:0/3642343694 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f70340a3c30 0x7f70340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.078+0000 7f705dd88740 1 -- 172.21.15.40:0/1974975610 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea36940 msgr2=0x558b0ea38dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.078+0000 7f705dd88740 1 --2- 172.21.15.40:0/1974975610 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea36940 0x558b0ea38dd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f705406aa80 tx=0x7f703c0118d0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.078+0000 7f705dd88740 1 -- 172.21.15.40:0/1974975610 shutdown_connections 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.078+0000 7f705dd88740 1 --2- 172.21.15.40:0/1974975610 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea36940 0x558b0ea38dd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.078+0000 7f705dd88740 1 --2- 172.21.15.40:0/1974975610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea317d0 0x558b0ea362d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.078+0000 7f705dd88740 1 --2- 172.21.15.40:0/1974975610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea30de0 0x558b0ea31200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.078+0000 7f705dd88740 1 -- 172.21.15.40:0/1974975610 >> 172.21.15.40:0/1974975610 conn(0x558b0ea2c450 msgr2=0x558b0ea2c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.078+0000 7f705dd88740 1 -- 172.21.15.40:0/1974975610 shutdown_connections 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.078+0000 7f705dd88740 1 -- 172.21.15.40:0/1974975610 wait complete. 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.079+0000 7f7059000640 1 --2- 172.21.15.40:0/3642343694 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f70340a3c30 0x7f70340a60f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f704c031500 tx=0x7f704c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.079+0000 7f705dd88740 1 -- start start 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.079+0000 7f705dd88740 1 Processor -- start 2024-04-27T00:05:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.079+0000 7f705dd88740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea30de0 0x558b0ea3b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.079+0000 7f705dd88740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea317d0 0x558b0ea3b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.079+0000 7f705dd88740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea42530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.079+0000 7f705dd88740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558b0ea398f0 con 0x558b0ea317d0 2024-04-27T00:05:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.079+0000 7f705dd88740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x558b0e91cc60 con 0x558b0ea30de0 2024-04-27T00:05:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.079+0000 7f705dd88740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558b0ea39a70 con 0x558b0ea36940 2024-04-27T00:05:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.079+0000 7f705dd88740 10 client.0 _ll_register_callbacks cb 0x558b0e948e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:05:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7053400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea317d0 0x558b0ea3b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7053400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea317d0 0x558b0ea3b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49334/0 (socket says 172.21.15.40:49334) 2024-04-27T00:05:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7053e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea30de0 0x558b0ea3b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7053400640 1 -- 172.21.15.40:0/2076064939 learned_addr learned my addr 172.21.15.40:0/2076064939 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7059000640 1 --2- 172.21.15.40:0/2076064939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea42530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7053e00640 1 -- 172.21.15.40:0/2076064939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 msgr2=0x558b0ea42530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7053e00640 1 --2- 172.21.15.40:0/2076064939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea42530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7053e00640 1 -- 172.21.15.40:0/2076064939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea317d0 msgr2=0x558b0ea3b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7053e00640 1 --2- 172.21.15.40:0/2076064939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea317d0 0x558b0ea3b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7053e00640 1 -- 172.21.15.40:0/2076064939 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b0e92b510 con 0x558b0ea30de0 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7053e00640 1 --2- 172.21.15.40:0/2076064939 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea30de0 0x558b0ea3b020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f70440ce210 tx=0x7f704400cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3088e52f94da4350 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7042000640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.080+0000 7f7042000640 1 -- 172.21.15.40:0/2076064939 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704409ca90 con 0x558b0ea30de0 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.081+0000 7f7042000640 1 -- 172.21.15.40:0/2076064939 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f70440bbba0 con 0x558b0ea30de0 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.081+0000 7f705dd88740 1 -- 172.21.15.40:0/2076064939 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f703c003dd0 con 0x558b0ea30de0 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.081+0000 7f7042000640 1 -- 172.21.15.40:0/2076064939 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f70440a0d90 con 0x558b0ea30de0 2024-04-27T00:05:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.081+0000 7f705dd88740 10 client.25658 fetch_fsmap learned FSMap version 24 2024-04-27T00:05:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.081+0000 7f705dd88740 1 -- 172.21.15.40:0/2076064939 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558b0e6a7210 con 0x558b0ea30de0 2024-04-27T00:05:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.081+0000 7f7042000640 1 -- 172.21.15.40:0/2076064939 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f703c003dd0 con 0x558b0ea30de0 2024-04-27T00:05:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.081+0000 7f7042000640 1 -- 172.21.15.40:0/2076064939 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f70440d6070 con 0x558b0ea30de0 2024-04-27T00:05:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.081+0000 7f705dd88740 10 client.25658 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:05:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.081+0000 7f705dd88740 10 client.25658 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:05:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.081+0000 7f705dd88740 20 client.25658 populate_metadata read hostname 'smithi040' 2024-04-27T00:05:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.082+0000 7f705dd88740 1 --2- 172.21.15.40:0/2076064939 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558b0ea46aa0 0x558b0ea48f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.082+0000 7f705dd88740 4 client.25658 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:05:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.082+0000 7f705dd88740 1 -- 172.21.15.40:0/2076064939 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b0e9a2f90 con 0x558b0ea46aa0 2024-04-27T00:05:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.082+0000 7f7053400640 1 --2- 172.21.15.40:0/2076064939 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558b0ea46aa0 0x558b0ea48f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.082+0000 7f7053400640 1 --2- 172.21.15.40:0/2076064939 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558b0ea46aa0 0x558b0ea48f30 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cffe143923eb21f server_cookie=9e376f881a1b84cb in_seq=0 out_seq=0 2024-04-27T00:05:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.082+0000 7f7042000640 10 client.25658 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.083+0000 7f7042000640 1 -- 172.21.15.40:0/2076064939 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b0e9a2f90 con 0x558b0ea46aa0 2024-04-27T00:05:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.083+0000 7f7042000640 10 client.25658 handle_command_reply: tid=0 2024-04-27T00:05:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.083+0000 7f705dd88740 1 client.25658 shutdown 2024-04-27T00:05:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.083+0000 7f705dd88740 1 -- 172.21.15.40:0/2076064939 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea30de0 msgr2=0x558b0ea3b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.083+0000 7f705dd88740 1 --2- 172.21.15.40:0/2076064939 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea30de0 0x558b0ea3b020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f70440ce210 tx=0x7f704400cf20 comp rx=0 tx=0).stop 2024-04-27T00:05:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.083+0000 7f705dd88740 1 -- 172.21.15.40:0/2076064939 shutdown_connections 2024-04-27T00:05:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.083+0000 7f705dd88740 1 --2- 172.21.15.40:0/2076064939 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558b0ea46aa0 0x558b0ea48f30 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.083+0000 7f705dd88740 1 --2- 172.21.15.40:0/2076064939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea42530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.083+0000 7f705dd88740 1 --2- 172.21.15.40:0/2076064939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea317d0 0x558b0ea3b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.083+0000 7f705dd88740 1 --2- 172.21.15.40:0/2076064939 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea30de0 0x558b0ea3b020 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.083+0000 7f7042000640 0 client.25658 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.083+0000 7f705dd88740 1 -- 172.21.15.40:0/2076064939 >> 172.21.15.40:0/2076064939 conn(0x558b0ea2c450 msgr2=0x558b0ea2b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:20.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.084+0000 7f705dd88740 1 -- 172.21.15.40:0/2076064939 shutdown_connections 2024-04-27T00:05:20.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.084+0000 7f705dd88740 1 -- 172.21.15.40:0/2076064939 wait complete. 2024-04-27T00:05:20.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.084+0000 7f705dd88740 20 client.25658 trim_cache size 0 max 16384 2024-04-27T00:05:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.100+0000 7f705dd88740 1 Processor -- start 2024-04-27T00:05:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.100+0000 7f705dd88740 1 -- start start 2024-04-27T00:05:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.101+0000 7f705dd88740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea30de0 0x558b0ea329f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.101+0000 7f705dd88740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea317d0 0x558b0ea424f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.101+0000 7f705dd88740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea32f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.101+0000 7f705dd88740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f70541afea0 con 0x558b0ea30de0 2024-04-27T00:05:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.101+0000 7f705dd88740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f70541ad770 con 0x558b0ea317d0 2024-04-27T00:05:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.101+0000 7f705dd88740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70541ab5c0 con 0x558b0ea36940 2024-04-27T00:05:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.101+0000 7f7059000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea32f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.101+0000 7f7059000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea32f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46424/0 (socket says 172.21.15.40:46424) 2024-04-27T00:05:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.101+0000 7f7059000640 1 -- 172.21.15.40:0/3333823400 learned_addr learned my addr 172.21.15.40:0/3333823400 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.101+0000 7f7053400640 1 --2- 172.21.15.40:0/3333823400 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea317d0 0x558b0ea424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.101+0000 7f7053e00640 1 --2- 172.21.15.40:0/3333823400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea30de0 0x558b0ea329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.102+0000 7f7059000640 1 -- 172.21.15.40:0/3333823400 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea317d0 msgr2=0x558b0ea424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.102+0000 7f7059000640 1 --2- 172.21.15.40:0/3333823400 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea317d0 0x558b0ea424f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.102+0000 7f7059000640 1 -- 172.21.15.40:0/3333823400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea30de0 msgr2=0x558b0ea329f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.102+0000 7f7059000640 1 --2- 172.21.15.40:0/3333823400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea30de0 0x558b0ea329f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.102+0000 7f7059000640 1 -- 172.21.15.40:0/3333823400 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b0e773620 con 0x558b0ea36940 2024-04-27T00:05:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.102+0000 7f7053e00640 1 --2- 172.21.15.40:0/3333823400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea30de0 0x558b0ea329f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:05:20.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.102+0000 7f7059000640 1 --2- 172.21.15.40:0/3333823400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea32f30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f704c02a9e0 tx=0x7f704c040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c598a6cbbb7db47f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.102+0000 7f7048c00640 1 -- 172.21.15.40:0/3333823400 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704c035040 con 0x558b0ea36940 2024-04-27T00:05:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.103+0000 7f7048c00640 1 -- 172.21.15.40:0/3333823400 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f704c031a50 con 0x558b0ea36940 2024-04-27T00:05:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.103+0000 7f7048c00640 1 -- 172.21.15.40:0/3333823400 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704c03da50 con 0x558b0ea36940 2024-04-27T00:05:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.103+0000 7f705dd88740 1 -- 172.21.15.40:0/3333823400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 msgr2=0x558b0ea32f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.103+0000 7f705dd88740 1 --2- 172.21.15.40:0/3333823400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea32f30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f704c02a9e0 tx=0x7f704c040000 comp rx=0 tx=0).stop 2024-04-27T00:05:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.103+0000 7f705dd88740 1 -- 172.21.15.40:0/3333823400 shutdown_connections 2024-04-27T00:05:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.103+0000 7f705dd88740 1 --2- 172.21.15.40:0/3333823400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea32f30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.103+0000 7f705dd88740 1 --2- 172.21.15.40:0/3333823400 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea317d0 0x558b0ea424f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.103+0000 7f705dd88740 1 --2- 172.21.15.40:0/3333823400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea30de0 0x558b0ea329f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.103+0000 7f705dd88740 1 -- 172.21.15.40:0/3333823400 >> 172.21.15.40:0/3333823400 conn(0x558b0ea2c450 msgr2=0x558b0ea2c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.103+0000 7f705dd88740 1 -- 172.21.15.40:0/3333823400 shutdown_connections 2024-04-27T00:05:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.103+0000 7f705dd88740 1 -- 172.21.15.40:0/3333823400 wait complete. 2024-04-27T00:05:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.104+0000 7f705dd88740 1 -- start start 2024-04-27T00:05:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.104+0000 7f705dd88740 1 Processor -- start 2024-04-27T00:05:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.104+0000 7f705dd88740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea30de0 0x558b0ea42210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.104+0000 7f705dd88740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea317d0 0x558b0ea3ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f705dd88740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea3f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f705dd88740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f70541abc30 con 0x558b0ea30de0 2024-04-27T00:05:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f705dd88740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f70541ad0a0 con 0x558b0ea317d0 2024-04-27T00:05:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f705dd88740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70541ac300 con 0x558b0ea36940 2024-04-27T00:05:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f705dd88740 10 client.0 _ll_register_callbacks cb 0x7f7044009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:05:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f7053400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea317d0 0x558b0ea3ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f7053400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea317d0 0x558b0ea3ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:48712/0 (socket says 172.21.15.40:48712) 2024-04-27T00:05:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f7053400640 1 -- 172.21.15.40:0/1594051169 learned_addr learned my addr 172.21.15.40:0/1594051169 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f7059000640 1 --2- 172.21.15.40:0/1594051169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea3f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f7053400640 1 -- 172.21.15.40:0/1594051169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 msgr2=0x558b0ea3f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f7053400640 1 --2- 172.21.15.40:0/1594051169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea3f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f7053400640 1 -- 172.21.15.40:0/1594051169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea30de0 msgr2=0x558b0ea42210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-27T00:05:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f7053400640 1 --2- 172.21.15.40:0/1594051169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea30de0 0x558b0ea42210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.105+0000 7f7053400640 1 -- 172.21.15.40:0/1594051169 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b0e94fcb0 con 0x558b0ea317d0 2024-04-27T00:05:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.106+0000 7f7053400640 1 --2- 172.21.15.40:0/1594051169 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea317d0 0x558b0ea3ecb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f703c009f70 tx=0x7f703c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94172232a533243 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:20.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.106+0000 7f7042000640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:05:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.108+0000 7f7042000640 1 -- 172.21.15.40:0/1594051169 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c003d30 con 0x558b0ea317d0 2024-04-27T00:05:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.108+0000 7f7042000640 1 -- 172.21.15.40:0/1594051169 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f703c0096c0 con 0x558b0ea317d0 2024-04-27T00:05:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.108+0000 7f705dd88740 1 -- 172.21.15.40:0/1594051169 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f704c031a50 con 0x558b0ea317d0 2024-04-27T00:05:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.109+0000 7f7042000640 1 -- 172.21.15.40:0/1594051169 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f703c0291d0 con 0x558b0ea317d0 2024-04-27T00:05:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.109+0000 7f705dd88740 10 client.25664 fetch_fsmap learned FSMap version 24 2024-04-27T00:05:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.109+0000 7f705dd88740 1 -- 172.21.15.40:0/1594051169 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558b0eaac040 con 0x558b0ea317d0 2024-04-27T00:05:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.109+0000 7f7042000640 1 -- 172.21.15.40:0/1594051169 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f704c031a50 con 0x558b0ea317d0 2024-04-27T00:05:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.109+0000 7f7042000640 1 -- 172.21.15.40:0/1594051169 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f703c018070 con 0x558b0ea317d0 2024-04-27T00:05:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.109+0000 7f705dd88740 10 client.25664 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:05:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.109+0000 7f705dd88740 10 client.25664 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:05:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.109+0000 7f705dd88740 20 client.25664 populate_metadata read hostname 'smithi040' 2024-04-27T00:05:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.109+0000 7f705dd88740 1 --2- 172.21.15.40:0/1594051169 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558b0eaac800 0x558b0eaaec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.109+0000 7f705dd88740 4 client.25664 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:05:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.109+0000 7f705dd88740 1 -- 172.21.15.40:0/1594051169 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558b0e99ce00 con 0x558b0eaac800 2024-04-27T00:05:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.110+0000 7f7053e00640 1 --2- 172.21.15.40:0/1594051169 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558b0eaac800 0x558b0eaaec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.110+0000 7f7053e00640 1 --2- 172.21.15.40:0/1594051169 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558b0eaac800 0x558b0eaaec20 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=885ead3ede0766fd server_cookie=50903a3b977ea0ef in_seq=0 out_seq=0 2024-04-27T00:05:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.110+0000 7f7042000640 10 client.25664 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:20.125 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f7042000640 1 -- 172.21.15.40:0/1594051169 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558b0e99ce00 con 0x558b0eaac800 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f7042000640 10 client.25664 handle_command_reply: tid=0 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f705dd88740 1 client.25664 shutdown 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f705dd88740 1 -- 172.21.15.40:0/1594051169 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea317d0 msgr2=0x558b0ea3ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f705dd88740 1 --2- 172.21.15.40:0/1594051169 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea317d0 0x558b0ea3ecb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f703c009f70 tx=0x7f703c009c00 comp rx=0 tx=0).stop 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f705dd88740 1 -- 172.21.15.40:0/1594051169 shutdown_connections 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f705dd88740 1 --2- 172.21.15.40:0/1594051169 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558b0eaac800 0x558b0eaaec20 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f705dd88740 1 --2- 172.21.15.40:0/1594051169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b0ea36940 0x558b0ea3f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f705dd88740 1 --2- 172.21.15.40:0/1594051169 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558b0ea317d0 0x558b0ea3ecb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f705dd88740 1 --2- 172.21.15.40:0/1594051169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558b0ea30de0 0x558b0ea42210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f705dd88740 1 -- 172.21.15.40:0/1594051169 >> 172.21.15.40:0/1594051169 conn(0x558b0ea2c450 msgr2=0x558b0ea329f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f7042000640 0 client.25664 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.125+0000 7f705dd88740 1 -- 172.21.15.40:0/1594051169 shutdown_connections 2024-04-27T00:05:20.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.126+0000 7f705dd88740 1 -- 172.21.15.40:0/1594051169 wait complete. 2024-04-27T00:05:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.126+0000 7f705dd88740 20 client.25664 trim_cache size 0 max 16384 2024-04-27T00:05:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:05:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:05:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:05:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:05:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.127+0000 7f7059a00640 1 -- 172.21.15.40:0/3642343694 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f70340a3c30 msgr2=0x7f70340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.127+0000 7f7059a00640 1 --2- 172.21.15.40:0/3642343694 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f70340a3c30 0x7f70340a60f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f704c031500 tx=0x7f704c005a20 comp rx=0 tx=0).stop 2024-04-27T00:05:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.127+0000 7f7059a00640 1 -- 172.21.15.40:0/3642343694 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f70541032f0 msgr2=0x7f705419abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.127+0000 7f7059a00640 1 --2- 172.21.15.40:0/3642343694 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f70541032f0 0x7f705419abb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f704400e9f0 tx=0x7f704400eeb0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.127+0000 7f7059a00640 1 -- 172.21.15.40:0/3642343694 shutdown_connections 2024-04-27T00:05:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.127+0000 7f7059a00640 1 --2- 172.21.15.40:0/3642343694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7054108550 0x7f70541a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.127+0000 7f7059a00640 1 --2- 172.21.15.40:0/3642343694 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f70340a3c30 0x7f70340a60f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.127+0000 7f7059a00640 1 --2- 172.21.15.40:0/3642343694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7054105c20 0x7f705419b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.127+0000 7f7059a00640 1 --2- 172.21.15.40:0/3642343694 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f70541032f0 0x7f705419abb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.127+0000 7f7059a00640 1 -- 172.21.15.40:0/3642343694 >> 172.21.15.40:0/3642343694 conn(0x7f70540fcc20 msgr2=0x7f7054102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.127+0000 7f7059a00640 1 -- 172.21.15.40:0/3642343694 shutdown_connections 2024-04-27T00:05:20.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:20.127+0000 7f7059a00640 1 -- 172.21.15.40:0/3642343694 wait complete. 2024-04-27T00:05:20.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:20 smithi040 ceph-mon[28095]: pgmap v521: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:20.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:20 smithi080 ceph-mon[32362]: pgmap v521: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:20.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:20 smithi103 ceph-mon[32952]: pgmap v521: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:22.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:22 smithi040 ceph-mon[28095]: pgmap v522: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:22.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:22 smithi080 ceph-mon[32362]: pgmap v522: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:22.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:22 smithi103 ceph-mon[32952]: pgmap v522: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:24.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:24 smithi040 ceph-mon[28095]: pgmap v523: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:05:24.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:24 smithi080 ceph-mon[32362]: pgmap v523: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:05:24.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:24 smithi103 ceph-mon[32952]: pgmap v523: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:05:26.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:26 smithi040 ceph-mon[28095]: pgmap v524: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:26.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:26 smithi080 ceph-mon[32362]: pgmap v524: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:26.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:26 smithi103 ceph-mon[32952]: pgmap v524: 129 pgs: 129 active+clean; 2.1 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:27.451 INFO:tasks.workunit.client.0.smithi040.stdout:64/600: open fd = 3 2024-04-27T00:05:27.451 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.451 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.451 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.451 INFO:tasks.workunit.client.0.smithi040.stdout:65/600: open fd = 3 2024-04-27T00:05:27.451 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.451 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.451 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.451 INFO:tasks.workunit.client.0.smithi040.stdout:66/600: open fd = 3 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:67/600: open fd = 3 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:68/600: open fd = 3 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:69/600: open fd = 3 2024-04-27T00:05:27.452 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:70/600: open fd = 3 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:71/600: open fd = 3 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:72/600: open fd = 3 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.453 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:73/600: open fd = 3 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:74/600: open fd = 3 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:75/600: open fd = 3 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.454 INFO:tasks.workunit.client.0.smithi040.stdout:76/600: open fd = 3 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:77/600: open fd = 3 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:78/600: open fd = 3 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:79/600: open fd = 3 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.455 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:80/600: open fd = 3 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:81/600: open fd = 3 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:82/600: open fd = 3 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:83/600: open fd = 3 2024-04-27T00:05:27.456 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:84/600: open fd = 3 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:85/600: open fd = 3 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:86/600: open fd = 3 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.457 INFO:tasks.workunit.client.0.smithi040.stdout:87/600: open fd = 3 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:88/600: open fd = 3 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:89/600: open fd = 3 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:90/600: open fd = 3 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.458 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:91/600: open fd = 3 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:92/600: open fd = 3 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:93/600: open fd = 3 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.459 INFO:tasks.workunit.client.0.smithi040.stdout:94/600: open fd = 3 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:95/600: open fd = 3 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:96/600: open fd = 3 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:97/600: open fd = 3 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.460 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:98/600: open fd = 3 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:99/600: open fd = 3 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:100/600: open fd = 3 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:101/600: open fd = 3 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.461 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:102/600: open fd = 3 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:103/600: open fd = 3 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:104/600: open fd = 3 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.462 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:105/600: open fd = 3 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:106/600: open fd = 3 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:107/600: open fd = 3 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:108/600: open fd = 3 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:109/600: open fd = 3 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:110/600: open fd = 3 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:111/600: open fd = 3 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:112/600: open fd = 3 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.464 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:113/600: open fd = 3 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:114/600: open fd = 3 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:115/600: open fd = 3 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:116/600: open fd = 3 2024-04-27T00:05:27.465 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:117/600: open fd = 3 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:118/600: open fd = 3 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:119/600: open fd = 3 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.466 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:120/600: open fd = 3 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:121/600: open fd = 3 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:122/600: open fd = 3 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:123/600: open fd = 3 2024-04-27T00:05:27.467 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:124/600: open fd = 3 2024-04-27T00:05:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:125/600: open fd = 3 2024-04-27T00:05:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:05:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:126/600: open fd = 3 2024-04-27T00:05:27.468 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:05:28.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:27 smithi103 ceph-mon[32952]: pgmap v525: 129 pgs: 129 active+clean; 2.2 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:05:28.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:27 smithi040 ceph-mon[28095]: pgmap v525: 129 pgs: 129 active+clean; 2.2 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:05:28.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:27 smithi080 ceph-mon[32362]: pgmap v525: 129 pgs: 129 active+clean; 2.2 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:05:30.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:29 smithi040 ceph-mon[28095]: pgmap v526: 129 pgs: 129 active+clean; 2.2 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:30.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:29 smithi080 ceph-mon[32362]: pgmap v526: 129 pgs: 129 active+clean; 2.2 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:30.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:29 smithi103 ceph-mon[32952]: pgmap v526: 129 pgs: 129 active+clean; 2.2 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:32.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:31 smithi040 ceph-mon[28095]: pgmap v527: 129 pgs: 129 active+clean; 2.2 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:32.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:31 smithi080 ceph-mon[32362]: pgmap v527: 129 pgs: 129 active+clean; 2.2 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:32.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:31 smithi103 ceph-mon[32952]: pgmap v527: 129 pgs: 129 active+clean; 2.2 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.273+0000 7f3c5b200640 1 -- 172.21.15.40:0/2297521952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c5c1065a0 msgr2=0x7f3c5c10ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.273+0000 7f3c5b200640 1 --2- 172.21.15.40:0/2297521952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c5c1065a0 0x7f3c5c10ce30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009900 tx=0x7f3c5402a400 comp rx=0 tx=0).stop 2024-04-27T00:05:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.273+0000 7f3c5b200640 1 -- 172.21.15.40:0/2297521952 shutdown_connections 2024-04-27T00:05:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.273+0000 7f3c5b200640 1 --2- 172.21.15.40:0/2297521952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c5c1065a0 0x7f3c5c10ce30 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.273+0000 7f3c5b200640 1 --2- 172.21.15.40:0/2297521952 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c5c105be0 0x7f3c5c106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.273+0000 7f3c5b200640 1 --2- 172.21.15.40:0/2297521952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c5c1049f0 0x7f3c5c104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.273+0000 7f3c5b200640 1 -- 172.21.15.40:0/2297521952 >> 172.21.15.40:0/2297521952 conn(0x7f3c5c0fffa0 msgr2=0x7f3c5c1023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.273+0000 7f3c5b200640 1 -- 172.21.15.40:0/2297521952 shutdown_connections 2024-04-27T00:05:33.276 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.273+0000 7f3c5b200640 1 -- 172.21.15.40:0/2297521952 wait complete. 2024-04-27T00:05:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.274+0000 7f3c5b200640 1 Processor -- start 2024-04-27T00:05:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.275+0000 7f3c5b200640 1 -- start start 2024-04-27T00:05:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.275+0000 7f3c5b200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c5c1049f0 0x7f3c5c19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.275+0000 7f3c5b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c5c105be0 0x7f3c5c19b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.275+0000 7f3c5b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c5c1065a0 0x7f3c5c1a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.275+0000 7f3c5b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c110500 con 0x7f3c5c105be0 2024-04-27T00:05:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.275+0000 7f3c5b200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c110380 con 0x7f3c5c1049f0 2024-04-27T00:05:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.275+0000 7f3c5b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c110680 con 0x7f3c5c1065a0 2024-04-27T00:05:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.275+0000 7f3c51400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c5c105be0 0x7f3c5c19b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.275+0000 7f3c59e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c5c1049f0 0x7f3c5c19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.275+0000 7f3c59e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c5c1049f0 0x7f3c5c19aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:55458/0 (socket says 172.21.15.40:55458) 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.275+0000 7f3c59e00640 1 -- 172.21.15.40:0/552954989 learned_addr learned my addr 172.21.15.40:0/552954989 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.275+0000 7f3c5a800640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c5c1065a0 0x7f3c5c1a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.276+0000 7f3c59e00640 1 -- 172.21.15.40:0/552954989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c5c1065a0 msgr2=0x7f3c5c1a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.276+0000 7f3c59e00640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c5c1065a0 0x7f3c5c1a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.276+0000 7f3c59e00640 1 -- 172.21.15.40:0/552954989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c5c105be0 msgr2=0x7f3c5c19b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.276+0000 7f3c59e00640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c5c105be0 0x7f3c5c19b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.276+0000 7f3c59e00640 1 -- 172.21.15.40:0/552954989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5c1a27c0 con 0x7f3c5c1049f0 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.276+0000 7f3c59e00640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c5c1049f0 0x7f3c5c19aba0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00ea80 tx=0x7f3c4c00ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.276+0000 7f3c53400640 1 -- 172.21.15.40:0/552954989 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c00ce40 con 0x7f3c5c1049f0 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.276+0000 7f3c53400640 1 -- 172.21.15.40:0/552954989 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c4c011920 con 0x7f3c5c1049f0 2024-04-27T00:05:33.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.276+0000 7f3c5b200640 1 -- 172.21.15.40:0/552954989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5c1a2ab0 con 0x7f3c5c1049f0 2024-04-27T00:05:33.280 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.276+0000 7f3c53400640 1 -- 172.21.15.40:0/552954989 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c01aaa0 con 0x7f3c5c1049f0 2024-04-27T00:05:33.280 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.276+0000 7f3c5b200640 1 -- 172.21.15.40:0/552954989 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c5c1a3000 con 0x7f3c5c1049f0 2024-04-27T00:05:33.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.280+0000 7f3c639a0740 1 Processor -- start 2024-04-27T00:05:33.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.280+0000 7f3c53400640 1 -- 172.21.15.40:0/552954989 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3c4c01ac40 con 0x7f3c5c1049f0 2024-04-27T00:05:33.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.280+0000 7f3c639a0740 1 -- start start 2024-04-27T00:05:33.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.280+0000 7f3c639a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 0x55bdd8e98200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.280+0000 7f3c639a0740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e987d0 0x55bdd8e9d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.280+0000 7f3c639a0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e9d940 0x55bdd8e9fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.280+0000 7f3c639a0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bdd8dfc380 con 0x55bdd8e9d940 2024-04-27T00:05:33.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.280+0000 7f3c639a0740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55bdd8d98c30 con 0x55bdd8e987d0 2024-04-27T00:05:33.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.280+0000 7f3c639a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bdd8df8340 con 0x55bdd8e97de0 2024-04-27T00:05:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.280+0000 7f3c51400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e9d940 0x55bdd8e9fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.280+0000 7f3c5a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 0x55bdd8e98200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.281+0000 7f3c59e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e987d0 0x55bdd8e9d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.281+0000 7f3c5a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 0x55bdd8e98200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:41066/0 (socket says 172.21.15.40:41066) 2024-04-27T00:05:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.281+0000 7f3c51400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e9d940 0x55bdd8e9fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55718/0 (socket says 172.21.15.40:55718) 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.281+0000 7f3c5a800640 1 -- 172.21.15.40:0/2275868092 learned_addr learned my addr 172.21.15.40:0/2275868092 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.281+0000 7f3c5a800640 1 -- 172.21.15.40:0/2275868092 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e987d0 msgr2=0x55bdd8e9d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.281+0000 7f3c5a800640 1 --2- 172.21.15.40:0/2275868092 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e987d0 0x55bdd8e9d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.281+0000 7f3c5a800640 1 -- 172.21.15.40:0/2275868092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e9d940 msgr2=0x55bdd8e9fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.281+0000 7f3c5a800640 1 --2- 172.21.15.40:0/2275868092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e9d940 0x55bdd8e9fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.281+0000 7f3c5a800640 1 -- 172.21.15.40:0/2275868092 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdd8dba940 con 0x55bdd8e97de0 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.281+0000 7f3c51400640 1 --2- 172.21.15.40:0/2275868092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e9d940 0x55bdd8e9fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.281+0000 7f3c53400640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3c400a3c30 0x7f3c400a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.281+0000 7f3c5a800640 1 --2- 172.21.15.40:0/2275868092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 0x55bdd8e98200 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009900 tx=0x7f3c54031b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3815aa1ccc50c036 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.282+0000 7f3c53400640 1 -- 172.21.15.40:0/552954989 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f3c4c0d0ea0 con 0x7f3c5c1049f0 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.282+0000 7f3c59e00640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3c400a3c30 0x7f3c400a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.282+0000 7f3c4b000640 1 -- 172.21.15.40:0/2275868092 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c54035030 con 0x55bdd8e97de0 2024-04-27T00:05:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.282+0000 7f3c4b000640 1 -- 172.21.15.40:0/2275868092 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c54040070 con 0x55bdd8e97de0 2024-04-27T00:05:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.282+0000 7f3c4b000640 1 -- 172.21.15.40:0/2275868092 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c54003ab0 con 0x55bdd8e97de0 2024-04-27T00:05:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.282+0000 7f3c639a0740 1 -- 172.21.15.40:0/2275868092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 msgr2=0x55bdd8e98200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.282+0000 7f3c639a0740 1 --2- 172.21.15.40:0/2275868092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 0x55bdd8e98200 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009900 tx=0x7f3c54031b90 comp rx=0 tx=0).stop 2024-04-27T00:05:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.282+0000 7f3c59e00640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3c400a3c30 0x7f3c400a60f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55bdd8e98bd0 tx=0x7f3c4c0d4950 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.283+0000 7f3c639a0740 1 -- 172.21.15.40:0/2275868092 shutdown_connections 2024-04-27T00:05:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.283+0000 7f3c639a0740 1 --2- 172.21.15.40:0/2275868092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e9d940 0x55bdd8e9fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.283+0000 7f3c639a0740 1 --2- 172.21.15.40:0/2275868092 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e987d0 0x55bdd8e9d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.283+0000 7f3c639a0740 1 --2- 172.21.15.40:0/2275868092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 0x55bdd8e98200 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.283+0000 7f3c639a0740 1 -- 172.21.15.40:0/2275868092 >> 172.21.15.40:0/2275868092 conn(0x55bdd8e93450 msgr2=0x55bdd8e93850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.283+0000 7f3c639a0740 1 -- 172.21.15.40:0/2275868092 shutdown_connections 2024-04-27T00:05:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.283+0000 7f3c639a0740 1 -- 172.21.15.40:0/2275868092 wait complete. 2024-04-27T00:05:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.283+0000 7f3c639a0740 1 -- start start 2024-04-27T00:05:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.283+0000 7f3c639a0740 1 Processor -- start 2024-04-27T00:05:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c639a0740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e97de0 0x55bdd8ea2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c639a0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c639a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e9d940 0x55bdd8ea9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c639a0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bdd8ea0890 con 0x55bdd8e987d0 2024-04-27T00:05:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c639a0740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55bdd8d83c60 con 0x55bdd8e97de0 2024-04-27T00:05:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c639a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bdd8ea0a10 con 0x55bdd8e9d940 2024-04-27T00:05:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c639a0740 10 client.0 _ll_register_callbacks cb 0x55bdd8dafe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:05:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c51400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c5a800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e97de0 0x55bdd8ea2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c51400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55722/0 (socket says 172.21.15.40:55722) 2024-04-27T00:05:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c5a800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e97de0 0x55bdd8ea2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:55472/0 (socket says 172.21.15.40:55472) 2024-04-27T00:05:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c51400640 1 -- 172.21.15.40:0/2948499991 learned_addr learned my addr 172.21.15.40:0/2948499991 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:33.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.284+0000 7f3c5a800640 1 --2- 172.21.15.40:0/2948499991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e9d940 0x55bdd8ea9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.285+0000 7f3c51400640 1 -- 172.21.15.40:0/2948499991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e9d940 msgr2=0x55bdd8ea9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:33.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.285+0000 7f3c51400640 1 --2- 172.21.15.40:0/2948499991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e9d940 0x55bdd8ea9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.285+0000 7f3c51400640 1 -- 172.21.15.40:0/2948499991 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e97de0 msgr2=0x55bdd8ea2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:33.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.285+0000 7f3c51400640 1 --2- 172.21.15.40:0/2948499991 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e97de0 0x55bdd8ea2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.285+0000 7f3c51400640 1 -- 172.21.15.40:0/2948499991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdd8d92510 con 0x55bdd8e987d0 2024-04-27T00:05:33.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.285+0000 7f3c51400640 1 --2- 172.21.15.40:0/2948499991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea25c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400bad0 tx=0x7f3c4400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14e992c791dd605 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:33.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.285+0000 7f3c3f400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:05:33.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.285+0000 7f3c3f400640 1 -- 172.21.15.40:0/2948499991 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c44003e20 con 0x55bdd8e987d0 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.285+0000 7f3c3f400640 1 -- 172.21.15.40:0/2948499991 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f3c44011440 con 0x55bdd8e987d0 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.285+0000 7f3c639a0740 1 -- 172.21.15.40:0/2948499991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c54040070 con 0x55bdd8e987d0 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.286+0000 7f3c3f400640 1 -- 172.21.15.40:0/2948499991 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3c44011640 con 0x55bdd8e987d0 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.286+0000 7f3c639a0740 10 client.15558 fetch_fsmap learned FSMap version 24 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.286+0000 7f3c639a0740 1 -- 172.21.15.40:0/2948499991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bdd8b0e210 con 0x55bdd8e987d0 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.286+0000 7f3c3f400640 1 -- 172.21.15.40:0/2948499991 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c54040070 con 0x55bdd8e987d0 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.286+0000 7f3c3f400640 1 -- 172.21.15.40:0/2948499991 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3c4401b910 con 0x55bdd8e987d0 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.286+0000 7f3c639a0740 10 client.15558 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.286+0000 7f3c639a0740 10 client.15558 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.286+0000 7f3c639a0740 20 client.15558 populate_metadata read hostname 'smithi040' 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.286+0000 7f3c639a0740 1 --2- 172.21.15.40:0/2948499991 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55bdd8eadb60 0x55bdd8eaffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.286+0000 7f3c639a0740 4 client.15558 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:05:33.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.286+0000 7f3c639a0740 1 -- 172.21.15.40:0/2948499991 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdd8e09f90 con 0x55bdd8eadb60 2024-04-27T00:05:33.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.287+0000 7f3c5a800640 1 --2- 172.21.15.40:0/2948499991 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55bdd8eadb60 0x55bdd8eaffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.287+0000 7f3c5a800640 1 --2- 172.21.15.40:0/2948499991 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55bdd8eadb60 0x55bdd8eaffc0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c7a23f9bbb8b9b9 server_cookie=6038f6caf6c2a0fa in_seq=0 out_seq=0 2024-04-27T00:05:33.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.287+0000 7f3c3f400640 10 client.15558 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:33.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.288+0000 7f3c3f400640 1 -- 172.21.15.40:0/2948499991 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bdd8e09f90 con 0x55bdd8eadb60 2024-04-27T00:05:33.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.288+0000 7f3c3f400640 10 client.15558 handle_command_reply: tid=0 2024-04-27T00:05:33.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.288+0000 7f3c639a0740 1 client.15558 shutdown 2024-04-27T00:05:33.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.288+0000 7f3c639a0740 1 -- 172.21.15.40:0/2948499991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 msgr2=0x55bdd8ea25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:33.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.288+0000 7f3c639a0740 1 --2- 172.21.15.40:0/2948499991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea25c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400bad0 tx=0x7f3c4400bf90 comp rx=0 tx=0).stop 2024-04-27T00:05:33.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.288+0000 7f3c639a0740 1 -- 172.21.15.40:0/2948499991 shutdown_connections 2024-04-27T00:05:33.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.288+0000 7f3c639a0740 1 --2- 172.21.15.40:0/2948499991 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55bdd8eadb60 0x55bdd8eaffc0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.288+0000 7f3c639a0740 1 --2- 172.21.15.40:0/2948499991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e9d940 0x55bdd8ea9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.288+0000 7f3c639a0740 1 --2- 172.21.15.40:0/2948499991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea25c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.295 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.288+0000 7f3c639a0740 1 --2- 172.21.15.40:0/2948499991 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e97de0 0x55bdd8ea2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.295 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.288+0000 7f3c3f400640 0 client.15558 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:33.295 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.288+0000 7f3c639a0740 1 -- 172.21.15.40:0/2948499991 >> 172.21.15.40:0/2948499991 conn(0x55bdd8e93450 msgr2=0x55bdd8e92650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:33.295 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.289+0000 7f3c639a0740 1 -- 172.21.15.40:0/2948499991 shutdown_connections 2024-04-27T00:05:33.295 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.289+0000 7f3c639a0740 1 -- 172.21.15.40:0/2948499991 wait complete. 2024-04-27T00:05:33.295 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.289+0000 7f3c639a0740 20 client.15558 trim_cache size 0 max 16384 2024-04-27T00:05:33.306 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.305+0000 7f3c639a0740 1 Processor -- start 2024-04-27T00:05:33.306 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.305+0000 7f3c639a0740 1 -- start start 2024-04-27T00:05:33.306 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.305+0000 7f3c639a0740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e93450 0x55bdd8e99a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.307 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.305+0000 7f3c639a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 0x55bdd8e99f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.307 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.305+0000 7f3c639a0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.307 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.305+0000 7f3c639a0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1afd60 con 0x55bdd8e987d0 2024-04-27T00:05:33.307 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.306+0000 7f3c639a0740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1ad630 con 0x55bdd8e93450 2024-04-27T00:05:33.307 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.306+0000 7f3c639a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1ab480 con 0x55bdd8e97de0 2024-04-27T00:05:33.311 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.306+0000 7f3c51400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 0x55bdd8e99f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.311 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.306+0000 7f3c51400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 0x55bdd8e99f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:41078/0 (socket says 172.21.15.40:41078) 2024-04-27T00:05:33.311 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.306+0000 7f3c51400640 1 -- 172.21.15.40:0/250340974 learned_addr learned my addr 172.21.15.40:0/250340974 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:33.311 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.306+0000 7f3c5a800640 1 --2- 172.21.15.40:0/250340974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.311 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.306+0000 7f3c5a800640 1 --2- 172.21.15.40:0/250340974 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e93450 0x55bdd8e99a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.311 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.306+0000 7f3c51400640 1 -- 172.21.15.40:0/250340974 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e93450 msgr2=0x55bdd8e99a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:33.311 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.306+0000 7f3c51400640 1 --2- 172.21.15.40:0/250340974 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e93450 0x55bdd8e99a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.311 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.307+0000 7f3c51400640 1 -- 172.21.15.40:0/250340974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 msgr2=0x55bdd8ea9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:33.311 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.307+0000 7f3c51400640 1 --2- 172.21.15.40:0/250340974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.311 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.307+0000 7f3c51400640 1 -- 172.21.15.40:0/250340974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdd8bda620 con 0x55bdd8e97de0 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.307+0000 7f3c5a800640 1 --2- 172.21.15.40:0/250340974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.307+0000 7f3c51400640 1 --2- 172.21.15.40:0/250340974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 0x55bdd8e99f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400baa0 tx=0x7f3c440114c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77de60b1f1c4cca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.307+0000 7f3c4a600640 1 -- 172.21.15.40:0/250340974 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c44003ea0 con 0x55bdd8e97de0 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.307+0000 7f3c4a600640 1 -- 172.21.15.40:0/250340974 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c4400b400 con 0x55bdd8e97de0 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.307+0000 7f3c4a600640 1 -- 172.21.15.40:0/250340974 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4401b420 con 0x55bdd8e97de0 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.308+0000 7f3c639a0740 1 -- 172.21.15.40:0/250340974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 msgr2=0x55bdd8e99f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.308+0000 7f3c639a0740 1 --2- 172.21.15.40:0/250340974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 0x55bdd8e99f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400baa0 tx=0x7f3c440114c0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.308+0000 7f3c639a0740 1 -- 172.21.15.40:0/250340974 shutdown_connections 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.308+0000 7f3c639a0740 1 --2- 172.21.15.40:0/250340974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.308+0000 7f3c639a0740 1 --2- 172.21.15.40:0/250340974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e97de0 0x55bdd8e99f90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.308+0000 7f3c639a0740 1 --2- 172.21.15.40:0/250340974 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e93450 0x55bdd8e99a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.308+0000 7f3c639a0740 1 -- 172.21.15.40:0/250340974 >> 172.21.15.40:0/250340974 conn(0x55bdd8e9d410 msgr2=0x55bdd8e93830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:33.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.308+0000 7f3c639a0740 1 -- 172.21.15.40:0/250340974 shutdown_connections 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.308+0000 7f3c639a0740 1 -- 172.21.15.40:0/250340974 wait complete. 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.308+0000 7f3c639a0740 1 -- start start 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.309+0000 7f3c639a0740 1 Processor -- start 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.309+0000 7f3c639a0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e93450 0x55bdd8ea6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.309+0000 7f3c639a0740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e97de0 0x55bdd8ea6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.309+0000 7f3c639a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.309+0000 7f3c639a0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1abaf0 con 0x55bdd8e93450 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.309+0000 7f3c639a0740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1acf60 con 0x55bdd8e97de0 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.309+0000 7f3c639a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1ac1c0 con 0x55bdd8e987d0 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.309+0000 7f3c639a0740 10 client.0 _ll_register_callbacks cb 0x7f3c44004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.309+0000 7f3c5a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e93450 0x55bdd8ea6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.309+0000 7f3c5a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e93450 0x55bdd8ea6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55746/0 (socket says 172.21.15.40:55746) 2024-04-27T00:05:33.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.309+0000 7f3c5a800640 1 -- 172.21.15.40:0/3306828103 learned_addr learned my addr 172.21.15.40:0/3306828103 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.309+0000 7f3c51400640 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e97de0 0x55bdd8ea6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.310+0000 7f3c5a800640 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.310+0000 7f3c51400640 1 -- 172.21.15.40:0/3306828103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e987d0 msgr2=0x55bdd8ea7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.310+0000 7f3c51400640 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.310+0000 7f3c51400640 1 -- 172.21.15.40:0/3306828103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e93450 msgr2=0x55bdd8ea6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.310+0000 7f3c51400640 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e93450 0x55bdd8ea6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.310+0000 7f3c51400640 1 -- 172.21.15.40:0/3306828103 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdd8db6cb0 con 0x55bdd8e97de0 2024-04-27T00:05:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.310+0000 7f3c5a800640 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e93450 0x55bdd8ea6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:05:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.310+0000 7f3c51400640 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e97de0 0x55bdd8ea6d40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400ba70 tx=0x7f3c4401b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=208f527940a266ac server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.310+0000 7f3c3f400640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:05:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.311+0000 7f3c3f400640 1 -- 172.21.15.40:0/3306828103 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c44037070 con 0x55bdd8e97de0 2024-04-27T00:05:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.311+0000 7f3c3f400640 1 -- 172.21.15.40:0/3306828103 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f3c4401bad0 con 0x55bdd8e97de0 2024-04-27T00:05:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.311+0000 7f3c639a0740 1 -- 172.21.15.40:0/3306828103 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c4400b400 con 0x55bdd8e97de0 2024-04-27T00:05:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.311+0000 7f3c3f400640 1 -- 172.21.15.40:0/3306828103 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3c44040050 con 0x55bdd8e97de0 2024-04-27T00:05:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.311+0000 7f3c639a0740 10 client.25682 fetch_fsmap learned FSMap version 24 2024-04-27T00:05:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.311+0000 7f3c639a0740 1 -- 172.21.15.40:0/3306828103 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bdd8ea9430 con 0x55bdd8e97de0 2024-04-27T00:05:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.311+0000 7f3c3f400640 1 -- 172.21.15.40:0/3306828103 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c4400b400 con 0x55bdd8e97de0 2024-04-27T00:05:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.312+0000 7f3c3f400640 1 -- 172.21.15.40:0/3306828103 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3c4403a020 con 0x55bdd8e97de0 2024-04-27T00:05:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.312+0000 7f3c639a0740 10 client.25682 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:05:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.312+0000 7f3c639a0740 10 client.25682 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:05:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.312+0000 7f3c639a0740 20 client.25682 populate_metadata read hostname 'smithi040' 2024-04-27T00:05:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.312+0000 7f3c639a0740 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55bdd8e9d940 0x55bdd8f163b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.312+0000 7f3c639a0740 4 client.25682 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:05:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.312+0000 7f3c639a0740 1 -- 172.21.15.40:0/3306828103 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bdd8e03e00 con 0x55bdd8e9d940 2024-04-27T00:05:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.312+0000 7f3c5a800640 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55bdd8e9d940 0x55bdd8f163b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.313+0000 7f3c5a800640 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55bdd8e9d940 0x55bdd8f163b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab15bf1e951896ff server_cookie=eb5abc38768f1562 in_seq=0 out_seq=0 2024-04-27T00:05:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.313+0000 7f3c3f400640 10 client.25682 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:33.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c3f400640 1 -- 172.21.15.40:0/3306828103 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bdd8e03e00 con 0x55bdd8e9d940 2024-04-27T00:05:33.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c3f400640 10 client.25682 handle_command_reply: tid=0 2024-04-27T00:05:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c639a0740 1 client.25682 shutdown 2024-04-27T00:05:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c639a0740 1 -- 172.21.15.40:0/3306828103 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e97de0 msgr2=0x55bdd8ea6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c639a0740 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e97de0 0x55bdd8ea6d40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400ba70 tx=0x7f3c4401b6d0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c639a0740 1 -- 172.21.15.40:0/3306828103 shutdown_connections 2024-04-27T00:05:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c639a0740 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55bdd8e9d940 0x55bdd8f163b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c639a0740 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdd8e987d0 0x55bdd8ea7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c639a0740 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55bdd8e97de0 0x55bdd8ea6d40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c639a0740 1 --2- 172.21.15.40:0/3306828103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdd8e93450 0x55bdd8ea6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c3f400640 0 client.25682 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c639a0740 1 -- 172.21.15.40:0/3306828103 >> 172.21.15.40:0/3306828103 conn(0x55bdd8e9d410 msgr2=0x55bdd8ea2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c639a0740 1 -- 172.21.15.40:0/3306828103 shutdown_connections 2024-04-27T00:05:33.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.328+0000 7f3c639a0740 1 -- 172.21.15.40:0/3306828103 wait complete. 2024-04-27T00:05:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.329+0000 7f3c639a0740 20 client.25682 trim_cache size 0 max 16384 2024-04-27T00:05:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:05:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:05:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:05:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:05:33.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.329+0000 7f3c5b200640 1 -- 172.21.15.40:0/552954989 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3c400a3c30 msgr2=0x7f3c400a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:33.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.329+0000 7f3c5b200640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3c400a3c30 0x7f3c400a60f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55bdd8e98bd0 tx=0x7f3c4c0d4950 comp rx=0 tx=0).stop 2024-04-27T00:05:33.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.329+0000 7f3c5b200640 1 -- 172.21.15.40:0/552954989 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c5c1049f0 msgr2=0x7f3c5c19aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:33.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.329+0000 7f3c5b200640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c5c1049f0 0x7f3c5c19aba0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00ea80 tx=0x7f3c4c00ef40 comp rx=0 tx=0).stop 2024-04-27T00:05:33.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.329+0000 7f3c5b200640 1 -- 172.21.15.40:0/552954989 shutdown_connections 2024-04-27T00:05:33.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.329+0000 7f3c5b200640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3c400a3c30 0x7f3c400a60f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.329+0000 7f3c5b200640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c5c1065a0 0x7f3c5c1a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.329+0000 7f3c5b200640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c5c105be0 0x7f3c5c19b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.329+0000 7f3c5b200640 1 --2- 172.21.15.40:0/552954989 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c5c1049f0 0x7f3c5c19aba0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:33.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.330+0000 7f3c5b200640 1 -- 172.21.15.40:0/552954989 >> 172.21.15.40:0/552954989 conn(0x7f3c5c0fffa0 msgr2=0x7f3c5c108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:33.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.330+0000 7f3c5b200640 1 -- 172.21.15.40:0/552954989 shutdown_connections 2024-04-27T00:05:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:33.330+0000 7f3c5b200640 1 -- 172.21.15.40:0/552954989 wait complete. 2024-04-27T00:05:34.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:33 smithi080 ceph-mon[32362]: pgmap v528: 129 pgs: 129 active+clean; 2.2 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:34.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:33 smithi103 ceph-mon[32952]: pgmap v528: 129 pgs: 129 active+clean; 2.2 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:34.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:33 smithi040 ceph-mon[28095]: pgmap v528: 129 pgs: 129 active+clean; 2.2 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:36.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:35 smithi080 ceph-mon[32362]: pgmap v529: 129 pgs: 129 active+clean; 2.2 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:05:36.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:35 smithi103 ceph-mon[32952]: pgmap v529: 129 pgs: 129 active+clean; 2.2 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:05:36.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:35 smithi040 ceph-mon[28095]: pgmap v529: 129 pgs: 129 active+clean; 2.2 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:05:38.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:37 smithi080 ceph-mon[32362]: pgmap v530: 129 pgs: 129 active+clean; 2.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:38.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:37 smithi103 ceph-mon[32952]: pgmap v530: 129 pgs: 129 active+clean; 2.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:38.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:37 smithi040 ceph-mon[28095]: pgmap v530: 129 pgs: 129 active+clean; 2.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:40.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:39 smithi080 ceph-mon[32362]: pgmap v531: 129 pgs: 129 active+clean; 2.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:40.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:39 smithi103 ceph-mon[32952]: pgmap v531: 129 pgs: 129 active+clean; 2.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:40.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:39 smithi040 ceph-mon[28095]: pgmap v531: 129 pgs: 129 active+clean; 2.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:42.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:41 smithi080 ceph-mon[32362]: pgmap v532: 129 pgs: 129 active+clean; 2.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:42.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:41 smithi103 ceph-mon[32952]: pgmap v532: 129 pgs: 129 active+clean; 2.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:42.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:41 smithi040 ceph-mon[28095]: pgmap v532: 129 pgs: 129 active+clean; 2.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:44.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:43 smithi080 ceph-mon[32362]: pgmap v533: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:44.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:43 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:05:44.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:43 smithi103 ceph-mon[32952]: pgmap v533: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:44.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:43 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:05:44.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:43 smithi040 ceph-mon[28095]: pgmap v533: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:44.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:43 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:05:46.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:46 smithi080 ceph-mon[32362]: pgmap v534: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:46.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:05:46.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:05:46.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:05:46.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:46 smithi103 ceph-mon[32952]: pgmap v534: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:46.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:05:46.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:05:46.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:05:46.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:46 smithi040 ceph-mon[28095]: pgmap v534: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:46.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:05:46.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:05:46.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:05:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.477+0000 7f2953200640 1 -- 172.21.15.40:0/1775011035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2954106e30 msgr2=0x7f2954109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.477+0000 7f2953200640 1 --2- 172.21.15.40:0/1775011035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2954106e30 0x7f2954109220 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f294c009be0 tx=0x7f294c02a6c0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.477+0000 7f2953200640 1 -- 172.21.15.40:0/1775011035 shutdown_connections 2024-04-27T00:05:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.477+0000 7f2953200640 1 --2- 172.21.15.40:0/1775011035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2954106e30 0x7f2954109220 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.477+0000 7f2953200640 1 --2- 172.21.15.40:0/1775011035 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2954102130 0x7f29541025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.477+0000 7f2953200640 1 --2- 172.21.15.40:0/1775011035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2954069300 0x7f2954101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.477+0000 7f2953200640 1 -- 172.21.15.40:0/1775011035 >> 172.21.15.40:0/1775011035 conn(0x7f29540fbf90 msgr2=0x7f29540fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:46.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.477+0000 7f2953200640 1 -- 172.21.15.40:0/1775011035 shutdown_connections 2024-04-27T00:05:46.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.477+0000 7f2953200640 1 -- 172.21.15.40:0/1775011035 wait complete. 2024-04-27T00:05:46.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.478+0000 7f2953200640 1 Processor -- start 2024-04-27T00:05:46.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.479+0000 7f2953200640 1 -- start start 2024-04-27T00:05:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.479+0000 7f2953200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2954069300 0x7f29541989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.479+0000 7f2953200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2954102130 0x7f2954198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.479+0000 7f2953200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2954106e30 0x7f295419ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.479+0000 7f2953200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f295410e300 con 0x7f2954069300 2024-04-27T00:05:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.479+0000 7f2953200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f295410e180 con 0x7f2954102130 2024-04-27T00:05:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.479+0000 7f2953200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f295410e480 con 0x7f2954106e30 2024-04-27T00:05:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.479+0000 7f2951e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2954069300 0x7f29541989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.479+0000 7f2951400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2954102130 0x7f2954198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.479+0000 7f2951e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2954069300 0x7f29541989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34392/0 (socket says 172.21.15.40:34392) 2024-04-27T00:05:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.479+0000 7f2951e00640 1 -- 172.21.15.40:0/1432016335 learned_addr learned my addr 172.21.15.40:0/1432016335 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.479+0000 7f2952800640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2954106e30 0x7f295419ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.480+0000 7f2951400640 1 -- 172.21.15.40:0/1432016335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2954106e30 msgr2=0x7f295419ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.480+0000 7f2951400640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2954106e30 0x7f295419ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.480+0000 7f2951400640 1 -- 172.21.15.40:0/1432016335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2954069300 msgr2=0x7f29541989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.480+0000 7f2951400640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2954069300 0x7f29541989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.480+0000 7f2951400640 1 -- 172.21.15.40:0/1432016335 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29541a0600 con 0x7f2954102130 2024-04-27T00:05:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.480+0000 7f2951400640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2954102130 0x7f2954198f20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2940005fd0 tx=0x7f294000ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.480+0000 7f294aa00640 1 -- 172.21.15.40:0/1432016335 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294000ccc0 con 0x7f2954102130 2024-04-27T00:05:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.481+0000 7f294aa00640 1 -- 172.21.15.40:0/1432016335 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f294000ee70 con 0x7f2954102130 2024-04-27T00:05:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.481+0000 7f2953200640 1 -- 172.21.15.40:0/1432016335 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29541a08f0 con 0x7f2954102130 2024-04-27T00:05:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.481+0000 7f294aa00640 1 -- 172.21.15.40:0/1432016335 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2940003e10 con 0x7f2954102130 2024-04-27T00:05:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.481+0000 7f2953200640 1 -- 172.21.15.40:0/1432016335 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29541a0e40 con 0x7f2954102130 2024-04-27T00:05:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.484+0000 7f294aa00640 1 -- 172.21.15.40:0/1432016335 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2940003fb0 con 0x7f2954102130 2024-04-27T00:05:46.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.484+0000 7f295bde7740 1 Processor -- start 2024-04-27T00:05:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.484+0000 7f295bde7740 1 -- start start 2024-04-27T00:05:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.485+0000 7f295bde7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d03930610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.485+0000 7f295bde7740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d039356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.485+0000 7f295bde7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03935d50 0x558d039381e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.485+0000 7f295bde7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558d03890380 con 0x558d03935d50 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.485+0000 7f295bde7740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x558d0382cc30 con 0x558d03930be0 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.485+0000 7f295bde7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558d0388c340 con 0x558d039301f0 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.485+0000 7f2952800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d03930610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.485+0000 7f2952800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d03930610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38820/0 (socket says 172.21.15.40:38820) 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.485+0000 7f2952800640 1 -- 172.21.15.40:0/1615097221 learned_addr learned my addr 172.21.15.40:0/1615097221 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.485+0000 7f2951e00640 1 --2- 172.21.15.40:0/1615097221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d039356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.485+0000 7f2951400640 1 --2- 172.21.15.40:0/1615097221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03935d50 0x558d039381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f2952800640 1 -- 172.21.15.40:0/1615097221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 msgr2=0x558d039356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f2952800640 1 --2- 172.21.15.40:0/1615097221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d039356e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f2952800640 1 -- 172.21.15.40:0/1615097221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03935d50 msgr2=0x558d039381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f2952800640 1 --2- 172.21.15.40:0/1615097221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03935d50 0x558d039381e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f2952800640 1 -- 172.21.15.40:0/1615097221 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d0384e940 con 0x558d039301f0 2024-04-27T00:05:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f2951e00640 1 --2- 172.21.15.40:0/1615097221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d039356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:05:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f294aa00640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29380a3c30 0x7f29380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f2952800640 1 --2- 172.21.15.40:0/1615097221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d03930610 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f294c02acb0 tx=0x7f294c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64617a18e5a983ca server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f294aa00640 1 -- 172.21.15.40:0/1432016335 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f29400d0b50 con 0x7f2954102130 2024-04-27T00:05:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f2951e00640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29380a3c30 0x7f29380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f293f400640 1 -- 172.21.15.40:0/1615097221 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c041030 con 0x558d039301f0 2024-04-27T00:05:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f293f400640 1 -- 172.21.15.40:0/1615097221 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f294c030570 con 0x558d039301f0 2024-04-27T00:05:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.486+0000 7f293f400640 1 -- 172.21.15.40:0/1615097221 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c036040 con 0x558d039301f0 2024-04-27T00:05:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.487+0000 7f295bde7740 1 -- 172.21.15.40:0/1615097221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 msgr2=0x558d03930610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.487+0000 7f2951e00640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29380a3c30 0x7f29380a60f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x558d03930fe0 tx=0x7f2944005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.487+0000 7f295bde7740 1 --2- 172.21.15.40:0/1615097221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d03930610 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f294c02acb0 tx=0x7f294c005fd0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.487+0000 7f295bde7740 1 -- 172.21.15.40:0/1615097221 shutdown_connections 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.487+0000 7f295bde7740 1 --2- 172.21.15.40:0/1615097221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03935d50 0x558d039381e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.487+0000 7f295bde7740 1 --2- 172.21.15.40:0/1615097221 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d039356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.487+0000 7f295bde7740 1 --2- 172.21.15.40:0/1615097221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d03930610 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.487+0000 7f295bde7740 1 -- 172.21.15.40:0/1615097221 >> 172.21.15.40:0/1615097221 conn(0x558d03927450 msgr2=0x558d03927850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.487+0000 7f295bde7740 1 -- 172.21.15.40:0/1615097221 shutdown_connections 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.487+0000 7f295bde7740 1 -- 172.21.15.40:0/1615097221 wait complete. 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.488+0000 7f295bde7740 1 -- start start 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.488+0000 7f295bde7740 1 Processor -- start 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.488+0000 7f295bde7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d0393a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f295bde7740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d0393a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f295bde7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03935d50 0x558d039419b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f295bde7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558d03938ca0 con 0x558d03935d50 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f295bde7740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x558d03817c60 con 0x558d03930be0 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f295bde7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558d03938e20 con 0x558d039301f0 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f295bde7740 10 client.0 _ll_register_callbacks cb 0x558d03843e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f2952800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03935d50 0x558d039419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f2951e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d0393a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f2951400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d0393a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f2951e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d0393a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38824/0 (socket says 172.21.15.40:38824) 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f2952800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03935d50 0x558d039419b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34400/0 (socket says 172.21.15.40:34400) 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f2951e00640 1 -- 172.21.15.40:0/3256392360 learned_addr learned my addr 172.21.15.40:0/3256392360 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f2951400640 1 -- 172.21.15.40:0/3256392360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 msgr2=0x558d0393a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f2951400640 1 --2- 172.21.15.40:0/3256392360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d0393a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f2951400640 1 -- 172.21.15.40:0/3256392360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03935d50 msgr2=0x558d039419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f2951400640 1 --2- 172.21.15.40:0/3256392360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03935d50 0x558d039419b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f2951400640 1 -- 172.21.15.40:0/3256392360 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d03826510 con 0x558d03930be0 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.489+0000 7f2951e00640 1 --2- 172.21.15.40:0/3256392360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d0393a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.490+0000 7f2951400640 1 --2- 172.21.15.40:0/3256392360 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d0393a970 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f29400ce100 tx=0x7f2940098c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e03e69dbc239a41a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.490+0000 7f2933e00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.490+0000 7f2933e00640 1 -- 172.21.15.40:0/3256392360 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2940085e60 con 0x558d03930be0 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.490+0000 7f2933e00640 1 -- 172.21.15.40:0/3256392360 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f29400042a0 con 0x558d03930be0 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.490+0000 7f295bde7740 1 -- 172.21.15.40:0/3256392360 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f294c030570 con 0x558d03930be0 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.490+0000 7f2933e00640 1 -- 172.21.15.40:0/3256392360 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f294009cfd0 con 0x558d03930be0 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.490+0000 7f295bde7740 10 client.25694 fetch_fsmap learned FSMap version 24 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.490+0000 7f295bde7740 1 -- 172.21.15.40:0/3256392360 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558d035a2210 con 0x558d03930be0 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.490+0000 7f2933e00640 1 -- 172.21.15.40:0/3256392360 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f294c030570 con 0x558d03930be0 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.491+0000 7f2933e00640 1 -- 172.21.15.40:0/3256392360 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f29400870b0 con 0x558d03930be0 2024-04-27T00:05:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.491+0000 7f295bde7740 10 client.25694 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:05:46.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.491+0000 7f295bde7740 10 client.25694 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:05:46.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.491+0000 7f295bde7740 20 client.25694 populate_metadata read hostname 'smithi040' 2024-04-27T00:05:46.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.491+0000 7f295bde7740 1 --2- 172.21.15.40:0/3256392360 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558d03945ea0 0x558d039482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.491+0000 7f295bde7740 4 client.25694 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:05:46.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.491+0000 7f295bde7740 1 -- 172.21.15.40:0/3256392360 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d0389df90 con 0x558d03945ea0 2024-04-27T00:05:46.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.491+0000 7f2951e00640 1 --2- 172.21.15.40:0/3256392360 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558d03945ea0 0x558d039482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.492+0000 7f2951e00640 1 --2- 172.21.15.40:0/3256392360 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558d03945ea0 0x558d039482d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fde5598bc1225d0a server_cookie=dde5879d98c281c6 in_seq=0 out_seq=0 2024-04-27T00:05:46.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.492+0000 7f2933e00640 10 client.25694 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:46.497 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f2933e00640 1 -- 172.21.15.40:0/3256392360 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d0389df90 con 0x558d03945ea0 2024-04-27T00:05:46.497 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f2933e00640 10 client.25694 handle_command_reply: tid=0 2024-04-27T00:05:46.497 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f295bde7740 1 client.25694 shutdown 2024-04-27T00:05:46.497 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f295bde7740 1 -- 172.21.15.40:0/3256392360 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 msgr2=0x558d0393a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:46.497 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f295bde7740 1 --2- 172.21.15.40:0/3256392360 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d0393a970 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f29400ce100 tx=0x7f2940098c60 comp rx=0 tx=0).stop 2024-04-27T00:05:46.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f295bde7740 1 -- 172.21.15.40:0/3256392360 shutdown_connections 2024-04-27T00:05:46.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f295bde7740 1 --2- 172.21.15.40:0/3256392360 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558d03945ea0 0x558d039482d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f295bde7740 1 --2- 172.21.15.40:0/3256392360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03935d50 0x558d039419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f295bde7740 1 --2- 172.21.15.40:0/3256392360 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d0393a970 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f2933e00640 0 client.25694 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:46.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f295bde7740 1 --2- 172.21.15.40:0/3256392360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d0393a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f295bde7740 1 -- 172.21.15.40:0/3256392360 >> 172.21.15.40:0/3256392360 conn(0x558d03927450 msgr2=0x558d03926650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:46.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f295bde7740 1 -- 172.21.15.40:0/3256392360 shutdown_connections 2024-04-27T00:05:46.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f295bde7740 1 -- 172.21.15.40:0/3256392360 wait complete. 2024-04-27T00:05:46.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.493+0000 7f295bde7740 20 client.25694 trim_cache size 0 max 16384 2024-04-27T00:05:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.510+0000 7f295bde7740 1 Processor -- start 2024-04-27T00:05:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.510+0000 7f295bde7740 1 -- start start 2024-04-27T00:05:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.510+0000 7f295bde7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03927450 0x558d03925dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.510+0000 7f295bde7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d03941970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.510+0000 7f295bde7740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d03926300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.510+0000 7f295bde7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f29541ab9d0 con 0x558d03927450 2024-04-27T00:05:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.510+0000 7f295bde7740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f29541a92a0 con 0x558d03930be0 2024-04-27T00:05:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.510+0000 7f295bde7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f29541a70f0 con 0x558d039301f0 2024-04-27T00:05:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2951e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03927450 0x558d03925dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2951400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d03941970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2951400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d03941970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38840/0 (socket says 172.21.15.40:38840) 2024-04-27T00:05:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2951400640 1 -- 172.21.15.40:0/1236489870 learned_addr learned my addr 172.21.15.40:0/1236489870 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2951e00640 1 --2- 172.21.15.40:0/1236489870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03927450 0x558d03925dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34414/0 (socket says 172.21.15.40:34414) 2024-04-27T00:05:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2952800640 1 --2- 172.21.15.40:0/1236489870 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d03926300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2951400640 1 -- 172.21.15.40:0/1236489870 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 msgr2=0x558d03926300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2951400640 1 --2- 172.21.15.40:0/1236489870 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d03926300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2951400640 1 -- 172.21.15.40:0/1236489870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03927450 msgr2=0x558d03925dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2951400640 1 --2- 172.21.15.40:0/1236489870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03927450 0x558d03925dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2951400640 1 -- 172.21.15.40:0/1236489870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d0366e620 con 0x558d039301f0 2024-04-27T00:05:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2952800640 1 --2- 172.21.15.40:0/1236489870 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d03926300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:05:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.511+0000 7f2951400640 1 --2- 172.21.15.40:0/1236489870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d03941970 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f294009c5d0 tx=0x7f2940003800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=692aba9b43d8f5ec server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.512+0000 7f293ea00640 1 -- 172.21.15.40:0/1236489870 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2940085e60 con 0x558d039301f0 2024-04-27T00:05:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.512+0000 7f293ea00640 1 -- 172.21.15.40:0/1236489870 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29400a0c80 con 0x558d039301f0 2024-04-27T00:05:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.512+0000 7f293ea00640 1 -- 172.21.15.40:0/1236489870 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29400ce350 con 0x558d039301f0 2024-04-27T00:05:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.512+0000 7f295bde7740 1 -- 172.21.15.40:0/1236489870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 msgr2=0x558d03941970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.512+0000 7f295bde7740 1 --2- 172.21.15.40:0/1236489870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d03941970 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f294009c5d0 tx=0x7f2940003800 comp rx=0 tx=0).stop 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.512+0000 7f295bde7740 1 -- 172.21.15.40:0/1236489870 shutdown_connections 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.512+0000 7f295bde7740 1 --2- 172.21.15.40:0/1236489870 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d03926300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.512+0000 7f295bde7740 1 --2- 172.21.15.40:0/1236489870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d039301f0 0x558d03941970 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.512+0000 7f295bde7740 1 --2- 172.21.15.40:0/1236489870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d03927450 0x558d03925dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.512+0000 7f295bde7740 1 -- 172.21.15.40:0/1236489870 >> 172.21.15.40:0/1236489870 conn(0x558d03935820 msgr2=0x558d03927830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.512+0000 7f295bde7740 1 -- 172.21.15.40:0/1236489870 shutdown_connections 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.512+0000 7f295bde7740 1 -- 172.21.15.40:0/1236489870 wait complete. 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.513+0000 7f295bde7740 1 -- start start 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.513+0000 7f295bde7740 1 Processor -- start 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.513+0000 7f295bde7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d03927450 0x558d0393f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.513+0000 7f295bde7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d039301f0 0x558d0393faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.513+0000 7f295bde7740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d03940b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.513+0000 7f295bde7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f29541a7760 con 0x558d039301f0 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.513+0000 7f295bde7740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f29541a8bd0 con 0x558d03930be0 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.513+0000 7f295bde7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f29541a7e30 con 0x558d03927450 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.513+0000 7f295bde7740 10 client.0 _ll_register_callbacks cb 0x7f294000cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2951400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d039301f0 0x558d0393faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2951400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d039301f0 0x558d0393faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34428/0 (socket says 172.21.15.40:34428) 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2951400640 1 -- 172.21.15.40:0/645503970 learned_addr learned my addr 172.21.15.40:0/645503970 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2951e00640 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d03927450 0x558d0393f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2952800640 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d03940b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2951400640 1 -- 172.21.15.40:0/645503970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d03927450 msgr2=0x558d0393f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2951400640 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d03927450 0x558d0393f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2951400640 1 -- 172.21.15.40:0/645503970 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 msgr2=0x558d03940b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2951400640 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d03940b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2951400640 1 -- 172.21.15.40:0/645503970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d0384acb0 con 0x558d039301f0 2024-04-27T00:05:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2951e00640 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d03927450 0x558d0393f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:05:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2951400640 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d039301f0 0x558d0393faf0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f29400d0f10 tx=0x7f294000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383d15cd83d132e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.514+0000 7f2933e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:05:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.515+0000 7f2933e00640 1 -- 172.21.15.40:0/645503970 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2940085e60 con 0x558d039301f0 2024-04-27T00:05:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.515+0000 7f2933e00640 1 -- 172.21.15.40:0/645503970 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f29400042a0 con 0x558d039301f0 2024-04-27T00:05:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.515+0000 7f295bde7740 1 -- 172.21.15.40:0/645503970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558d03897e00 con 0x558d039301f0 2024-04-27T00:05:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.515+0000 7f2933e00640 1 -- 172.21.15.40:0/645503970 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f294009bd20 con 0x558d039301f0 2024-04-27T00:05:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.515+0000 7f295bde7740 10 client.15582 fetch_fsmap learned FSMap version 24 2024-04-27T00:05:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.515+0000 7f295bde7740 1 -- 172.21.15.40:0/645503970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558d0384acb0 con 0x558d039301f0 2024-04-27T00:05:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.515+0000 7f2933e00640 1 -- 172.21.15.40:0/645503970 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558d03897e00 con 0x558d039301f0 2024-04-27T00:05:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.515+0000 7f2933e00640 1 -- 172.21.15.40:0/645503970 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f294009bec0 con 0x558d039301f0 2024-04-27T00:05:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.515+0000 7f295bde7740 10 client.15582 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:05:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.515+0000 7f295bde7740 10 client.15582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:05:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.515+0000 7f295bde7740 20 client.15582 populate_metadata read hostname 'smithi040' 2024-04-27T00:05:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.515+0000 7f295bde7740 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558d03935d50 0x558d039aee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.516+0000 7f295bde7740 4 client.15582 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:05:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.516+0000 7f295bde7740 1 -- 172.21.15.40:0/645503970 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558d038134b0 con 0x558d03935d50 2024-04-27T00:05:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.516+0000 7f2951e00640 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558d03935d50 0x558d039aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.516+0000 7f2951e00640 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558d03935d50 0x558d039aee00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9abb30c304124ba0 server_cookie=4be10cb0988d50f3 in_seq=0 out_seq=0 2024-04-27T00:05:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.516+0000 7f2933e00640 10 client.15582 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:46.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.531+0000 7f2933e00640 1 -- 172.21.15.40:0/645503970 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558d038134b0 con 0x558d03935d50 2024-04-27T00:05:46.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.531+0000 7f2933e00640 10 client.15582 handle_command_reply: tid=0 2024-04-27T00:05:46.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.531+0000 7f295bde7740 1 client.15582 shutdown 2024-04-27T00:05:46.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.531+0000 7f295bde7740 1 -- 172.21.15.40:0/645503970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d039301f0 msgr2=0x558d0393faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:46.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.531+0000 7f295bde7740 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d039301f0 0x558d0393faf0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f29400d0f10 tx=0x7f294000ce60 comp rx=0 tx=0).stop 2024-04-27T00:05:46.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.532+0000 7f295bde7740 1 -- 172.21.15.40:0/645503970 shutdown_connections 2024-04-27T00:05:46.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.532+0000 7f295bde7740 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x558d03935d50 0x558d039aee00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.532+0000 7f295bde7740 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x558d03930be0 0x558d03940b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.532+0000 7f295bde7740 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558d039301f0 0x558d0393faf0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.532+0000 7f2933e00640 0 client.15582 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:46.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.532+0000 7f295bde7740 1 --2- 172.21.15.40:0/645503970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558d03927450 0x558d0393f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.532+0000 7f295bde7740 1 -- 172.21.15.40:0/645503970 >> 172.21.15.40:0/645503970 conn(0x558d03935820 msgr2=0x558d0392a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:46.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.532+0000 7f295bde7740 1 -- 172.21.15.40:0/645503970 shutdown_connections 2024-04-27T00:05:46.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.532+0000 7f295bde7740 1 -- 172.21.15.40:0/645503970 wait complete. 2024-04-27T00:05:46.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.532+0000 7f295bde7740 20 client.15582 trim_cache size 0 max 16384 2024-04-27T00:05:46.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:05:46.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:05:46.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:05:46.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:05:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.533+0000 7f2953200640 1 -- 172.21.15.40:0/1432016335 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29380a3c30 msgr2=0x7f29380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.533+0000 7f2953200640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29380a3c30 0x7f29380a60f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x558d03930fe0 tx=0x7f2944005f90 comp rx=0 tx=0).stop 2024-04-27T00:05:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.533+0000 7f2953200640 1 -- 172.21.15.40:0/1432016335 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2954102130 msgr2=0x7f2954198f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.533+0000 7f2953200640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2954102130 0x7f2954198f20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2940005fd0 tx=0x7f294000ec20 comp rx=0 tx=0).stop 2024-04-27T00:05:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.533+0000 7f2953200640 1 -- 172.21.15.40:0/1432016335 shutdown_connections 2024-04-27T00:05:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.533+0000 7f2953200640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29380a3c30 0x7f29380a60f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.533+0000 7f2953200640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2954106e30 0x7f295419ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.533+0000 7f2953200640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2954102130 0x7f2954198f20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.533+0000 7f2953200640 1 --2- 172.21.15.40:0/1432016335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2954069300 0x7f29541989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.533+0000 7f2953200640 1 -- 172.21.15.40:0/1432016335 >> 172.21.15.40:0/1432016335 conn(0x7f29540fbf90 msgr2=0x7f2954102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.533+0000 7f2953200640 1 -- 172.21.15.40:0/1432016335 shutdown_connections 2024-04-27T00:05:46.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:46.533+0000 7f2953200640 1 -- 172.21.15.40:0/1432016335 wait complete. 2024-04-27T00:05:48.269 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:48 smithi040 ceph-mon[28095]: pgmap v535: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:05:48.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:48 smithi080 ceph-mon[32362]: pgmap v535: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:05:48.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:48 smithi103 ceph-mon[32952]: pgmap v535: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:05:50.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:50 smithi080 ceph-mon[32362]: pgmap v536: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:05:50.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:50 smithi103 ceph-mon[32952]: pgmap v536: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:05:50.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:50 smithi040 ceph-mon[28095]: pgmap v536: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:05:52.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:52 smithi080 ceph-mon[32362]: pgmap v537: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:05:52.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:52 smithi103 ceph-mon[32952]: pgmap v537: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:05:52.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:52 smithi040 ceph-mon[28095]: pgmap v537: 129 pgs: 129 active+clean; 2.4 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:05:54.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:54 smithi080 ceph-mon[32362]: pgmap v538: 129 pgs: 129 active+clean; 2.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:54.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:54 smithi103 ceph-mon[32952]: pgmap v538: 129 pgs: 129 active+clean; 2.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:54.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:54 smithi040 ceph-mon[28095]: pgmap v538: 129 pgs: 129 active+clean; 2.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:05:56.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:56 smithi080 ceph-mon[32362]: pgmap v539: 129 pgs: 129 active+clean; 2.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:56.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:56 smithi103 ceph-mon[32952]: pgmap v539: 129 pgs: 129 active+clean; 2.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:56.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:56 smithi040 ceph-mon[28095]: pgmap v539: 129 pgs: 129 active+clean; 2.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:05:58.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:05:58 smithi080 ceph-mon[32362]: pgmap v540: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:05:58.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:05:58 smithi103 ceph-mon[32952]: pgmap v540: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:05:58.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:05:58 smithi040 ceph-mon[28095]: pgmap v540: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:05:59.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.681+0000 7f1d65400640 1 -- 172.21.15.40:0/3758037171 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d60075390 msgr2=0x7f1d60075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:59.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.681+0000 7f1d65400640 1 --2- 172.21.15.40:0/3758037171 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d60075390 0x7f1d60075810 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009900 tx=0x7f1d4802a3f0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.681+0000 7f1d65400640 1 -- 172.21.15.40:0/3758037171 shutdown_connections 2024-04-27T00:05:59.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.681+0000 7f1d65400640 1 --2- 172.21.15.40:0/3758037171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d6010a680 0x7f1d6010cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.681+0000 7f1d65400640 1 --2- 172.21.15.40:0/3758037171 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d60075390 0x7f1d60075810 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.681+0000 7f1d65400640 1 --2- 172.21.15.40:0/3758037171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d60076970 0x7f1d60074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.681+0000 7f1d65400640 1 -- 172.21.15.40:0/3758037171 >> 172.21.15.40:0/3758037171 conn(0x7f1d600fdd30 msgr2=0x7f1d60100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:59.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.681+0000 7f1d65400640 1 -- 172.21.15.40:0/3758037171 shutdown_connections 2024-04-27T00:05:59.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.681+0000 7f1d65400640 1 -- 172.21.15.40:0/3758037171 wait complete. 2024-04-27T00:05:59.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.683+0000 7f1d65400640 1 Processor -- start 2024-04-27T00:05:59.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.683+0000 7f1d65400640 1 -- start start 2024-04-27T00:05:59.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.683+0000 7f1d65400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d60075390 0x7f1d6019ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.683+0000 7f1d65400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d60076970 0x7f1d6019b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.683+0000 7f1d65400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d6010a680 0x7f1d601a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.683+0000 7f1d65400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1d60110520 con 0x7f1d60075390 2024-04-27T00:05:59.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.683+0000 7f1d65400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1d601103a0 con 0x7f1d6010a680 2024-04-27T00:05:59.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.684+0000 7f1d65400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1d601106a0 con 0x7f1d60076970 2024-04-27T00:05:59.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.684+0000 7f1d5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d60075390 0x7f1d6019ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.684+0000 7f1d5f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d60076970 0x7f1d6019b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.684+0000 7f1d5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d60075390 0x7f1d6019ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34060/0 (socket says 172.21.15.40:34060) 2024-04-27T00:05:59.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.684+0000 7f1d5fe00640 1 -- 172.21.15.40:0/2416195967 learned_addr learned my addr 172.21.15.40:0/2416195967 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:59.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.684+0000 7f1d64a00640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d6010a680 0x7f1d601a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.684+0000 7f1d5f400640 1 -- 172.21.15.40:0/2416195967 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d6010a680 msgr2=0x7f1d601a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.684+0000 7f1d5f400640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d6010a680 0x7f1d601a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.684+0000 7f1d5f400640 1 -- 172.21.15.40:0/2416195967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d60075390 msgr2=0x7f1d6019ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:59.686 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.684+0000 7f1d5f400640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d60075390 0x7f1d6019ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.684+0000 7f1d5f400640 1 -- 172.21.15.40:0/2416195967 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d601a2810 con 0x7f1d60076970 2024-04-27T00:05:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.685+0000 7f1d5f400640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d60076970 0x7f1d6019b180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1d4802a9e0 tx=0x7f1d48003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.685+0000 7f1d5cc00640 1 -- 172.21.15.40:0/2416195967 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4803f070 con 0x7f1d60076970 2024-04-27T00:05:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.685+0000 7f1d5cc00640 1 -- 172.21.15.40:0/2416195967 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d48031c50 con 0x7f1d60076970 2024-04-27T00:05:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.685+0000 7f1d5cc00640 1 -- 172.21.15.40:0/2416195967 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d48035030 con 0x7f1d60076970 2024-04-27T00:05:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.685+0000 7f1d65400640 1 -- 172.21.15.40:0/2416195967 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d601a2aa0 con 0x7f1d60076970 2024-04-27T00:05:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.685+0000 7f1d65400640 1 -- 172.21.15.40:0/2416195967 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d601a2f10 con 0x7f1d60076970 2024-04-27T00:05:59.689 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.688+0000 7f1d697e2740 1 Processor -- start 2024-04-27T00:05:59.689 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d697e2740 1 -- start start 2024-04-27T00:05:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d697e2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31d9fede0 0x55d31d9ff200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d697e2740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da042d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d697e2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31da04940 0x55d31da06dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d697e2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d31d963380 con 0x55d31da04940 2024-04-27T00:05:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d697e2740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55d31d8ffc30 con 0x55d31d9ff7d0 2024-04-27T00:05:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d697e2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d31d95f340 con 0x55d31d9fede0 2024-04-27T00:05:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d5cc00640 1 -- 172.21.15.40:0/2416195967 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1d48042070 con 0x7f1d60076970 2024-04-27T00:05:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d64a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31d9fede0 0x55d31d9ff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d5fe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d64a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31d9fede0 0x55d31d9ff200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:45564/0 (socket says 172.21.15.40:45564) 2024-04-27T00:05:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d64a00640 1 -- 172.21.15.40:0/885125661 learned_addr learned my addr 172.21.15.40:0/885125661 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.689+0000 7f1d5f400640 1 --2- 172.21.15.40:0/885125661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31da04940 0x55d31da06dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.690+0000 7f1d64a00640 1 -- 172.21.15.40:0/885125661 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 msgr2=0x55d31da042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.690+0000 7f1d64a00640 1 --2- 172.21.15.40:0/885125661 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.690+0000 7f1d64a00640 1 -- 172.21.15.40:0/885125661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31da04940 msgr2=0x55d31da06dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.690+0000 7f1d64a00640 1 --2- 172.21.15.40:0/885125661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31da04940 0x55d31da06dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.690+0000 7f1d64a00640 1 -- 172.21.15.40:0/885125661 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d31d921940 con 0x55d31d9fede0 2024-04-27T00:05:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.690+0000 7f1d5fe00640 1 --2- 172.21.15.40:0/885125661 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:05:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.690+0000 7f1d5f400640 1 --2- 172.21.15.40:0/885125661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31da04940 0x55d31da06dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:05:59.692 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.690+0000 7f1d64a00640 1 --2- 172.21.15.40:0/885125661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31d9fede0 0x55d31d9ff200 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009fd0 tx=0x7f1d580118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=242ceb0d717eba server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.690+0000 7f1d55200640 1 -- 172.21.15.40:0/885125661 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d58004d20 con 0x55d31d9fede0 2024-04-27T00:05:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d55200640 1 -- 172.21.15.40:0/885125661 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d58004050 con 0x55d31d9fede0 2024-04-27T00:05:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d55200640 1 -- 172.21.15.40:0/885125661 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d58005280 con 0x55d31d9fede0 2024-04-27T00:05:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d5cc00640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1d440a3cb0 0x7f1d440a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d5cc00640 1 -- 172.21.15.40:0/2416195967 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f1d480f0fb0 con 0x7f1d60076970 2024-04-27T00:05:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d5fe00640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1d440a3cb0 0x7f1d440a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d697e2740 1 -- 172.21.15.40:0/885125661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31d9fede0 msgr2=0x55d31d9ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d697e2740 1 --2- 172.21.15.40:0/885125661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31d9fede0 0x55d31d9ff200 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009fd0 tx=0x7f1d580118d0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d697e2740 1 -- 172.21.15.40:0/885125661 shutdown_connections 2024-04-27T00:05:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d697e2740 1 --2- 172.21.15.40:0/885125661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31da04940 0x55d31da06dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d697e2740 1 --2- 172.21.15.40:0/885125661 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d697e2740 1 --2- 172.21.15.40:0/885125661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31d9fede0 0x55d31d9ff200 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d697e2740 1 -- 172.21.15.40:0/885125661 >> 172.21.15.40:0/885125661 conn(0x55d31d9fa450 msgr2=0x55d31d9fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.691+0000 7f1d697e2740 1 -- 172.21.15.40:0/885125661 shutdown_connections 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.692+0000 7f1d697e2740 1 -- 172.21.15.40:0/885125661 wait complete. 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.692+0000 7f1d5fe00640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1d440a3cb0 0x7f1d440a6170 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x55d31d9fb3c0 tx=0x7f1d5000b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.692+0000 7f1d697e2740 1 -- start start 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.692+0000 7f1d697e2740 1 Processor -- start 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.693+0000 7f1d697e2740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9fede0 0x55d31da08ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.693+0000 7f1d697e2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31d9ff7d0 0x55d31da09530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.693+0000 7f1d697e2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31da04940 0x55d31da10570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.693+0000 7f1d697e2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d31da078f0 con 0x55d31d9ff7d0 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.693+0000 7f1d697e2740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55d31d8eac60 con 0x55d31d9fede0 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.693+0000 7f1d697e2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d31da07a70 con 0x55d31da04940 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.693+0000 7f1d697e2740 10 client.0 _ll_register_callbacks cb 0x55d31d916e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:05:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.693+0000 7f1d5f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31d9ff7d0 0x55d31da09530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.693+0000 7f1d5fe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9fede0 0x55d31da08ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.693+0000 7f1d5f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31d9ff7d0 0x55d31da09530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34080/0 (socket says 172.21.15.40:34080) 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.693+0000 7f1d5f400640 1 -- 172.21.15.40:0/4142317561 learned_addr learned my addr 172.21.15.40:0/4142317561 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.693+0000 7f1d64a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31da04940 0x55d31da10570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.694+0000 7f1d5fe00640 1 -- 172.21.15.40:0/4142317561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31da04940 msgr2=0x55d31da10570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.694+0000 7f1d5fe00640 1 --2- 172.21.15.40:0/4142317561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31da04940 0x55d31da10570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.694+0000 7f1d5fe00640 1 -- 172.21.15.40:0/4142317561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31d9ff7d0 msgr2=0x55d31da09530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.694+0000 7f1d5fe00640 1 --2- 172.21.15.40:0/4142317561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31d9ff7d0 0x55d31da09530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.694+0000 7f1d5fe00640 1 -- 172.21.15.40:0/4142317561 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d31d8f9510 con 0x55d31d9fede0 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.694+0000 7f1d5fe00640 1 --2- 172.21.15.40:0/4142317561 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9fede0 0x55d31da08ff0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1d5000bb30 tx=0x7f1d50004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a4a0bdafd7ae81c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.694+0000 7f1d4d600640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.694+0000 7f1d4d600640 1 -- 172.21.15.40:0/4142317561 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5001fe10 con 0x55d31d9fede0 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.694+0000 7f1d4d600640 1 -- 172.21.15.40:0/4142317561 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f1d50017030 con 0x55d31d9fede0 2024-04-27T00:05:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.694+0000 7f1d697e2740 1 -- 172.21.15.40:0/4142317561 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d58004050 con 0x55d31d9fede0 2024-04-27T00:05:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.695+0000 7f1d4d600640 1 -- 172.21.15.40:0/4142317561 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1d50005020 con 0x55d31d9fede0 2024-04-27T00:05:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.695+0000 7f1d697e2740 10 client.25712 fetch_fsmap learned FSMap version 24 2024-04-27T00:05:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.695+0000 7f1d697e2740 1 -- 172.21.15.40:0/4142317561 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d31d675210 con 0x55d31d9fede0 2024-04-27T00:05:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.695+0000 7f1d4d600640 1 -- 172.21.15.40:0/4142317561 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d58004050 con 0x55d31d9fede0 2024-04-27T00:05:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.695+0000 7f1d4d600640 1 -- 172.21.15.40:0/4142317561 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1d50027410 con 0x55d31d9fede0 2024-04-27T00:05:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.695+0000 7f1d697e2740 10 client.25712 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:05:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.695+0000 7f1d697e2740 10 client.25712 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:05:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.695+0000 7f1d697e2740 20 client.25712 populate_metadata read hostname 'smithi040' 2024-04-27T00:05:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.696+0000 7f1d697e2740 1 --2- 172.21.15.40:0/4142317561 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d31da14ad0 0x55d31da16ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.696+0000 7f1d697e2740 4 client.25712 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:05:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.696+0000 7f1d697e2740 1 -- 172.21.15.40:0/4142317561 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d31d970f90 con 0x55d31da14ad0 2024-04-27T00:05:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.696+0000 7f1d5f400640 1 --2- 172.21.15.40:0/4142317561 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d31da14ad0 0x55d31da16ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.696+0000 7f1d5f400640 1 --2- 172.21.15.40:0/4142317561 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d31da14ad0 0x55d31da16ee0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1541daf8d84ca70 server_cookie=68f393d07b0e1fab in_seq=0 out_seq=0 2024-04-27T00:05:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.696+0000 7f1d4d600640 10 client.25712 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.697+0000 7f1d4d600640 1 -- 172.21.15.40:0/4142317561 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d31d970f90 con 0x55d31da14ad0 2024-04-27T00:05:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.697+0000 7f1d4d600640 10 client.25712 handle_command_reply: tid=0 2024-04-27T00:05:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.697+0000 7f1d697e2740 1 client.25712 shutdown 2024-04-27T00:05:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.697+0000 7f1d697e2740 1 -- 172.21.15.40:0/4142317561 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9fede0 msgr2=0x55d31da08ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.697+0000 7f1d697e2740 1 --2- 172.21.15.40:0/4142317561 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9fede0 0x55d31da08ff0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1d5000bb30 tx=0x7f1d50004730 comp rx=0 tx=0).stop 2024-04-27T00:05:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.697+0000 7f1d697e2740 1 -- 172.21.15.40:0/4142317561 shutdown_connections 2024-04-27T00:05:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.697+0000 7f1d697e2740 1 --2- 172.21.15.40:0/4142317561 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d31da14ad0 0x55d31da16ee0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.697+0000 7f1d697e2740 1 --2- 172.21.15.40:0/4142317561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31da04940 0x55d31da10570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.697+0000 7f1d697e2740 1 --2- 172.21.15.40:0/4142317561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31d9ff7d0 0x55d31da09530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.697+0000 7f1d4d600640 0 client.25712 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.697+0000 7f1d697e2740 1 --2- 172.21.15.40:0/4142317561 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9fede0 0x55d31da08ff0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.697+0000 7f1d697e2740 1 -- 172.21.15.40:0/4142317561 >> 172.21.15.40:0/4142317561 conn(0x55d31d9fa450 msgr2=0x55d31d9f9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.698+0000 7f1d697e2740 1 -- 172.21.15.40:0/4142317561 shutdown_connections 2024-04-27T00:05:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.698+0000 7f1d697e2740 1 -- 172.21.15.40:0/4142317561 wait complete. 2024-04-27T00:05:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.698+0000 7f1d697e2740 20 client.25712 trim_cache size 0 max 16384 2024-04-27T00:05:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.714+0000 7f1d697e2740 1 Processor -- start 2024-04-27T00:05:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.715+0000 7f1d697e2740 1 -- start start 2024-04-27T00:05:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.715+0000 7f1d697e2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31d9fede0 0x55d31da08c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.715+0000 7f1d697e2740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da09140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.715+0000 7f1d697e2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31da04940 0x55d31d9fd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.715+0000 7f1d697e2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1d601afd60 con 0x55d31da04940 2024-04-27T00:05:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.715+0000 7f1d697e2740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1d601ad630 con 0x55d31d9ff7d0 2024-04-27T00:05:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.715+0000 7f1d697e2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1d601ab480 con 0x55d31d9fede0 2024-04-27T00:05:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.715+0000 7f1d64a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31da04940 0x55d31d9fd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.715+0000 7f1d5f400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da09140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.715+0000 7f1d64a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31da04940 0x55d31d9fd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34096/0 (socket says 172.21.15.40:34096) 2024-04-27T00:05:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.715+0000 7f1d64a00640 1 -- 172.21.15.40:0/528893749 learned_addr learned my addr 172.21.15.40:0/528893749 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.715+0000 7f1d5fe00640 1 --2- 172.21.15.40:0/528893749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31d9fede0 0x55d31da08c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.716+0000 7f1d5f400640 1 -- 172.21.15.40:0/528893749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31d9fede0 msgr2=0x55d31da08c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.716+0000 7f1d5f400640 1 --2- 172.21.15.40:0/528893749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31d9fede0 0x55d31da08c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.716+0000 7f1d5f400640 1 -- 172.21.15.40:0/528893749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31da04940 msgr2=0x55d31d9fd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.716+0000 7f1d5f400640 1 --2- 172.21.15.40:0/528893749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31da04940 0x55d31d9fd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.716+0000 7f1d5f400640 1 -- 172.21.15.40:0/528893749 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d31d741620 con 0x55d31d9ff7d0 2024-04-27T00:05:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.716+0000 7f1d5f400640 1 --2- 172.21.15.40:0/528893749 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da09140 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1d480b92a0 tx=0x7f1d48005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa12665c93b5a7e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.716+0000 7f1d4fe00640 1 -- 172.21.15.40:0/528893749 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d48049040 con 0x55d31d9ff7d0 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.716+0000 7f1d4fe00640 1 -- 172.21.15.40:0/528893749 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d48031e00 con 0x55d31d9ff7d0 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.716+0000 7f1d4fe00640 1 -- 172.21.15.40:0/528893749 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d480bb1d0 con 0x55d31d9ff7d0 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.717+0000 7f1d697e2740 1 -- 172.21.15.40:0/528893749 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 msgr2=0x55d31da09140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.717+0000 7f1d697e2740 1 --2- 172.21.15.40:0/528893749 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da09140 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1d480b92a0 tx=0x7f1d48005fd0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.717+0000 7f1d697e2740 1 -- 172.21.15.40:0/528893749 shutdown_connections 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.717+0000 7f1d697e2740 1 --2- 172.21.15.40:0/528893749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31da04940 0x55d31d9fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.717+0000 7f1d697e2740 1 --2- 172.21.15.40:0/528893749 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da09140 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.717+0000 7f1d697e2740 1 --2- 172.21.15.40:0/528893749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31d9fede0 0x55d31da08c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.717+0000 7f1d697e2740 1 -- 172.21.15.40:0/528893749 >> 172.21.15.40:0/528893749 conn(0x55d31d9fa450 msgr2=0x55d31da00950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.717+0000 7f1d697e2740 1 -- 172.21.15.40:0/528893749 shutdown_connections 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.717+0000 7f1d697e2740 1 -- 172.21.15.40:0/528893749 wait complete. 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.717+0000 7f1d697e2740 1 -- start start 2024-04-27T00:05:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.718+0000 7f1d697e2740 1 Processor -- start 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.718+0000 7f1d697e2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31d9fede0 0x55d31da0d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.718+0000 7f1d697e2740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da0d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.718+0000 7f1d697e2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31da04940 0x55d31da0e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.718+0000 7f1d697e2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1d601abaf0 con 0x55d31d9fede0 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.718+0000 7f1d697e2740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1d601acf60 con 0x55d31d9ff7d0 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.718+0000 7f1d697e2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1d601ac1c0 con 0x55d31da04940 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.718+0000 7f1d697e2740 10 client.0 _ll_register_callbacks cb 0x7f1d5000b6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d5f400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da0d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d64a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31da04940 0x55d31da0e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d5f400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da0d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:43782/0 (socket says 172.21.15.40:43782) 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d5f400640 1 -- 172.21.15.40:0/3825358001 learned_addr learned my addr 172.21.15.40:0/3825358001 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d5fe00640 1 --2- 172.21.15.40:0/3825358001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31d9fede0 0x55d31da0d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d5f400640 1 -- 172.21.15.40:0/3825358001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31da04940 msgr2=0x55d31da0e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d5f400640 1 --2- 172.21.15.40:0/3825358001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31da04940 0x55d31da0e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d5f400640 1 -- 172.21.15.40:0/3825358001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31d9fede0 msgr2=0x55d31da0d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:05:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d5f400640 1 --2- 172.21.15.40:0/3825358001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31d9fede0 0x55d31da0d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d5f400640 1 -- 172.21.15.40:0/3825358001 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d31d91dcb0 con 0x55d31d9ff7d0 2024-04-27T00:05:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d5f400640 1 --2- 172.21.15.40:0/3825358001 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da0d800 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1d480ee530 tx=0x7f1d480f46f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c16b557955c6f6dd server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:05:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d4d600640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:05:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.719+0000 7f1d4d600640 1 -- 172.21.15.40:0/3825358001 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d48049040 con 0x55d31d9ff7d0 2024-04-27T00:05:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d697e2740 1 -- 172.21.15.40:0/3825358001 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d31d96ae00 con 0x55d31d9ff7d0 2024-04-27T00:05:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d4d600640 1 -- 172.21.15.40:0/3825358001 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f1d480b94b0 con 0x55d31d9ff7d0 2024-04-27T00:05:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d4d600640 1 -- 172.21.15.40:0/3825358001 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1d480bce40 con 0x55d31d9ff7d0 2024-04-27T00:05:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d697e2740 10 client.25718 fetch_fsmap learned FSMap version 24 2024-04-27T00:05:59.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d697e2740 1 -- 172.21.15.40:0/3825358001 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d31da7b060 con 0x55d31d9ff7d0 2024-04-27T00:05:59.724 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d4d600640 1 -- 172.21.15.40:0/3825358001 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d31d96ae00 con 0x55d31d9ff7d0 2024-04-27T00:05:59.724 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d4d600640 1 -- 172.21.15.40:0/3825358001 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1d480eebd0 con 0x55d31d9ff7d0 2024-04-27T00:05:59.724 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d697e2740 10 client.25718 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:05:59.724 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d697e2740 10 client.25718 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:05:59.724 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d697e2740 20 client.25718 populate_metadata read hostname 'smithi040' 2024-04-27T00:05:59.724 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d697e2740 1 --2- 172.21.15.40:0/3825358001 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d31da7b8b0 0x55d31da7dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:05:59.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d697e2740 4 client.25718 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:05:59.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.720+0000 7f1d697e2740 1 -- 172.21.15.40:0/3825358001 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d31d8e64b0 con 0x55d31da7b8b0 2024-04-27T00:05:59.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.721+0000 7f1d5fe00640 1 --2- 172.21.15.40:0/3825358001 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d31da7b8b0 0x55d31da7dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:05:59.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.721+0000 7f1d5fe00640 1 --2- 172.21.15.40:0/3825358001 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d31da7b8b0 0x55d31da7dcd0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65e24aa83fa20f8 server_cookie=78ec5aace6ba814a in_seq=0 out_seq=0 2024-04-27T00:05:59.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.721+0000 7f1d4d600640 10 client.25718 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.739+0000 7f1d4d600640 1 -- 172.21.15.40:0/3825358001 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d31d8e64b0 con 0x55d31da7b8b0 2024-04-27T00:05:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.739+0000 7f1d4d600640 10 client.25718 handle_command_reply: tid=0 2024-04-27T00:05:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.739+0000 7f1d697e2740 1 client.25718 shutdown 2024-04-27T00:05:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.739+0000 7f1d697e2740 1 -- 172.21.15.40:0/3825358001 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 msgr2=0x55d31da0d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:59.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.739+0000 7f1d697e2740 1 --2- 172.21.15.40:0/3825358001 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da0d800 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1d480ee530 tx=0x7f1d480f46f0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.739+0000 7f1d697e2740 1 -- 172.21.15.40:0/3825358001 shutdown_connections 2024-04-27T00:05:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.739+0000 7f1d697e2740 1 --2- 172.21.15.40:0/3825358001 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d31da7b8b0 0x55d31da7dcd0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.739+0000 7f1d697e2740 1 --2- 172.21.15.40:0/3825358001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d31da04940 0x55d31da0e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.739+0000 7f1d697e2740 1 --2- 172.21.15.40:0/3825358001 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d31d9ff7d0 0x55d31da0d800 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.739+0000 7f1d697e2740 1 --2- 172.21.15.40:0/3825358001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d31d9fede0 0x55d31da0d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.739+0000 7f1d4d600640 0 client.25718 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:05:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.739+0000 7f1d697e2740 1 -- 172.21.15.40:0/3825358001 >> 172.21.15.40:0/3825358001 conn(0x55d31d9fa450 msgr2=0x55d31da08c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:59.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.740+0000 7f1d697e2740 1 -- 172.21.15.40:0/3825358001 shutdown_connections 2024-04-27T00:05:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.740+0000 7f1d697e2740 1 -- 172.21.15.40:0/3825358001 wait complete. 2024-04-27T00:05:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.740+0000 7f1d697e2740 20 client.25718 trim_cache size 0 max 16384 2024-04-27T00:05:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:05:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:05:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:05:59.742 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:05:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.741+0000 7f1d65400640 1 -- 172.21.15.40:0/2416195967 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1d440a3cb0 msgr2=0x7f1d440a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.741+0000 7f1d65400640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1d440a3cb0 0x7f1d440a6170 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x55d31d9fb3c0 tx=0x7f1d5000b480 comp rx=0 tx=0).stop 2024-04-27T00:05:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.741+0000 7f1d65400640 1 -- 172.21.15.40:0/2416195967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d60076970 msgr2=0x7f1d6019b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:05:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.741+0000 7f1d65400640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d60076970 0x7f1d6019b180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1d4802a9e0 tx=0x7f1d48003c10 comp rx=0 tx=0).stop 2024-04-27T00:05:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.741+0000 7f1d65400640 1 -- 172.21.15.40:0/2416195967 shutdown_connections 2024-04-27T00:05:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.741+0000 7f1d65400640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1d6010a680 0x7f1d601a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.741+0000 7f1d65400640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1d440a3cb0 0x7f1d440a6170 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.741+0000 7f1d65400640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d60076970 0x7f1d6019b180 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.741+0000 7f1d65400640 1 --2- 172.21.15.40:0/2416195967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d60075390 0x7f1d6019ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:05:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.741+0000 7f1d65400640 1 -- 172.21.15.40:0/2416195967 >> 172.21.15.40:0/2416195967 conn(0x7f1d600fdd30 msgr2=0x7f1d60106580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:05:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.741+0000 7f1d65400640 1 -- 172.21.15.40:0/2416195967 shutdown_connections 2024-04-27T00:05:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:05:59.741+0000 7f1d65400640 1 -- 172.21.15.40:0/2416195967 wait complete. 2024-04-27T00:06:00.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:00 smithi080 ceph-mon[32362]: pgmap v541: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:00.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:00 smithi103 ceph-mon[32952]: pgmap v541: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:00.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:00 smithi040 ceph-mon[28095]: pgmap v541: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:02.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:02 smithi080 ceph-mon[32362]: pgmap v542: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:02.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:02 smithi103 ceph-mon[32952]: pgmap v542: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:02.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:02 smithi040 ceph-mon[28095]: pgmap v542: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:04.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:04 smithi080 ceph-mon[32362]: pgmap v543: 129 pgs: 129 active+clean; 2.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-27T00:06:04.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:04 smithi103 ceph-mon[32952]: pgmap v543: 129 pgs: 129 active+clean; 2.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-27T00:06:04.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:04 smithi040 ceph-mon[28095]: pgmap v543: 129 pgs: 129 active+clean; 2.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-27T00:06:06.350 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:06 smithi103 ceph-mon[32952]: pgmap v544: 129 pgs: 129 active+clean; 2.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:06 smithi040 ceph-mon[28095]: pgmap v544: 129 pgs: 129 active+clean; 2.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:06.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:06 smithi080 ceph-mon[32362]: pgmap v544: 129 pgs: 129 active+clean; 2.6 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:08.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:08 smithi103 ceph-mon[32952]: pgmap v545: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:08.351 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:08 smithi080 ceph-mon[32362]: pgmap v545: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:08.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:08 smithi040 ceph-mon[28095]: pgmap v545: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:10.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:10 smithi103 ceph-mon[32952]: pgmap v546: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:10.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:10 smithi040 ceph-mon[28095]: pgmap v546: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:10.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:10 smithi080 ceph-mon[32362]: pgmap v546: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:12.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:12 smithi103 ceph-mon[32952]: pgmap v547: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:12.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:12 smithi040 ceph-mon[28095]: pgmap v547: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:12.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:12 smithi080 ceph-mon[32362]: pgmap v547: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.895+0000 7f7af6e00640 1 -- 172.21.15.40:0/1848706512 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7af01045e0 msgr2=0x7f7af0104a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.895+0000 7f7af6e00640 1 --2- 172.21.15.40:0/1848706512 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7af01045e0 0x7f7af0104a60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009900 tx=0x7f7adc02a3f0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.895+0000 7f7af6e00640 1 -- 172.21.15.40:0/1848706512 shutdown_connections 2024-04-27T00:06:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.895+0000 7f7af6e00640 1 --2- 172.21.15.40:0/1848706512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af0104fa0 0x7f7af010ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.895+0000 7f7af6e00640 1 --2- 172.21.15.40:0/1848706512 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7af01045e0 0x7f7af0104a60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.895+0000 7f7af6e00640 1 --2- 172.21.15.40:0/1848706512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7af01060b0 0x7f7af01064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.895+0000 7f7af6e00640 1 -- 172.21.15.40:0/1848706512 >> 172.21.15.40:0/1848706512 conn(0x7f7af00fff40 msgr2=0x7f7af0102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.896+0000 7f7af6e00640 1 -- 172.21.15.40:0/1848706512 shutdown_connections 2024-04-27T00:06:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.896+0000 7f7af6e00640 1 -- 172.21.15.40:0/1848706512 wait complete. 2024-04-27T00:06:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.897+0000 7f7af6e00640 1 Processor -- start 2024-04-27T00:06:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.897+0000 7f7af6e00640 1 -- start start 2024-04-27T00:06:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.897+0000 7f7af6e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7af01045e0 0x7f7af019ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af6e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af0104fa0 0x7f7af019b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af6e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7af01060b0 0x7f7af01a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af6e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7af0110650 con 0x7f7af0104fa0 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af6e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01104d0 con 0x7f7af01060b0 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af6e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01107d0 con 0x7f7af01045e0 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7af01045e0 0x7f7af019ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af6400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7af01060b0 0x7f7af01a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7af01045e0 0x7f7af019ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60004/0 (socket says 172.21.15.40:60004) 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af0104fa0 0x7f7af019b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af5a00640 1 -- 172.21.15.40:0/2432251618 learned_addr learned my addr 172.21.15.40:0/2432251618 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af6400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7af01060b0 0x7f7af01a2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:45926/0 (socket says 172.21.15.40:45926) 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af6400640 1 -- 172.21.15.40:0/2432251618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7af01045e0 msgr2=0x7f7af019ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af6400640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7af01045e0 0x7f7af019ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af6400640 1 -- 172.21.15.40:0/2432251618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af0104fa0 msgr2=0x7f7af019b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.898+0000 7f7af6400640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af0104fa0 0x7f7af019b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.899+0000 7f7af6400640 1 -- 172.21.15.40:0/2432251618 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af01a2920 con 0x7f7af01060b0 2024-04-27T00:06:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.899+0000 7f7af5a00640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7af01045e0 0x7f7af019ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:06:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.899+0000 7f7af6400640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7af01060b0 0x7f7af01a2280 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800d680 tx=0x7f7ae800db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.899+0000 7f7aee000640 1 -- 172.21.15.40:0/2432251618 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8017070 con 0x7f7af01060b0 2024-04-27T00:06:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.899+0000 7f7aee000640 1 -- 172.21.15.40:0/2432251618 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ae8009820 con 0x7f7af01060b0 2024-04-27T00:06:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.899+0000 7f7aee000640 1 -- 172.21.15.40:0/2432251618 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8004ce0 con 0x7f7af01060b0 2024-04-27T00:06:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.899+0000 7f7af6e00640 1 -- 172.21.15.40:0/2432251618 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af01a2c10 con 0x7f7af01060b0 2024-04-27T00:06:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.899+0000 7f7af6e00640 1 -- 172.21.15.40:0/2432251618 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af01a3110 con 0x7f7af01060b0 2024-04-27T00:06:12.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.903+0000 7f7afb1bd740 1 Processor -- start 2024-04-27T00:06:12.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.904+0000 7f7aee000640 1 -- 172.21.15.40:0/2432251618 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7ae80099d0 con 0x7f7af01060b0 2024-04-27T00:06:12.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.904+0000 7f7afb1bd740 1 -- start start 2024-04-27T00:06:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.904+0000 7f7afb1bd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fcffa5f0 0x5631fcffaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.904+0000 7f7afb1bd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffafe0 0x5631fcfffae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.904+0000 7f7afb1bd740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fd000150 0x5631fd0025e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.904+0000 7f7afb1bd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5631fcf56380 con 0x5631fcffa5f0 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.904+0000 7f7afb1bd740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5631fcef2c30 con 0x5631fd000150 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.904+0000 7f7afb1bd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5631fcf52340 con 0x5631fcffafe0 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.905+0000 7f7af5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffafe0 0x5631fcfffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.905+0000 7f7af5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fcffa5f0 0x5631fcffaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.905+0000 7f7af5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffafe0 0x5631fcfffae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60008/0 (socket says 172.21.15.40:60008) 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.905+0000 7f7af5a00640 1 -- 172.21.15.40:0/2807004316 learned_addr learned my addr 172.21.15.40:0/2807004316 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.905+0000 7f7af5a00640 1 -- 172.21.15.40:0/2807004316 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fd000150 msgr2=0x5631fd0025e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.905+0000 7f7af5a00640 1 --2- 172.21.15.40:0/2807004316 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fd000150 0x5631fd0025e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.905+0000 7f7af5a00640 1 -- 172.21.15.40:0/2807004316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fcffa5f0 msgr2=0x5631fcffaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.905+0000 7f7af5a00640 1 --2- 172.21.15.40:0/2807004316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fcffa5f0 0x5631fcffaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.905+0000 7f7af5a00640 1 -- 172.21.15.40:0/2807004316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631fcf14940 con 0x5631fcffafe0 2024-04-27T00:06:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.905+0000 7f7af5000640 1 --2- 172.21.15.40:0/2807004316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fcffa5f0 0x5631fcffaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:06:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.905+0000 7f7af5a00640 1 --2- 172.21.15.40:0/2807004316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffafe0 0x5631fcfffae0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40118f0 tx=0x7f7ae4011db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f63fe96fe2fa0f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.906+0000 7f7aee000640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7ad80a3c30 0x7f7ad80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.906+0000 7f7ae2a00640 1 -- 172.21.15.40:0/2807004316 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4011310 con 0x5631fcffafe0 2024-04-27T00:06:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.906+0000 7f7ae2a00640 1 -- 172.21.15.40:0/2807004316 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ae4003e20 con 0x5631fcffafe0 2024-04-27T00:06:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.906+0000 7f7ae2a00640 1 -- 172.21.15.40:0/2807004316 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4004420 con 0x5631fcffafe0 2024-04-27T00:06:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.906+0000 7f7aee000640 1 -- 172.21.15.40:0/2432251618 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f7ae80d0240 con 0x7f7af01060b0 2024-04-27T00:06:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.906+0000 7f7af5000640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7ad80a3c30 0x7f7ad80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.907+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2807004316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffafe0 msgr2=0x5631fcfffae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.907+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/2807004316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffafe0 0x5631fcfffae0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40118f0 tx=0x7f7ae4011db0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.907+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2807004316 shutdown_connections 2024-04-27T00:06:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.907+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/2807004316 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fd000150 0x5631fd0025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.907+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/2807004316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffafe0 0x5631fcfffae0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.907+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/2807004316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fcffa5f0 0x5631fcffaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.907+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2807004316 >> 172.21.15.40:0/2807004316 conn(0x5631fcfed450 msgr2=0x5631fcfed850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.907+0000 7f7af5000640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7ad80a3c30 0x7f7ad80a60f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7adc031500 tx=0x7f7adc005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.907+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2807004316 shutdown_connections 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.907+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2807004316 wait complete. 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.907+0000 7f7afb1bd740 1 -- start start 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7afb1bd740 1 Processor -- start 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7afb1bd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fcffa5f0 0x5631fd0046a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7afb1bd740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffafe0 0x5631fd004be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7afb1bd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fd000150 0x5631fd00bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7afb1bd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5631fd003100 con 0x5631fcffa5f0 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7afb1bd740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5631fceddc60 con 0x5631fcffafe0 2024-04-27T00:06:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7afb1bd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5631fd003280 con 0x5631fd000150 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7afb1bd740 10 client.0 _ll_register_callbacks cb 0x5631fcf09e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7af5000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffafe0 0x5631fd004be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7af5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fcffa5f0 0x5631fd0046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7af5000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffafe0 0x5631fd004be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:45942/0 (socket says 172.21.15.40:45942) 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7af5000640 1 -- 172.21.15.40:0/2705719952 learned_addr learned my addr 172.21.15.40:0/2705719952 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.908+0000 7f7af6400640 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fd000150 0x5631fd00bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.909+0000 7f7af5a00640 1 -- 172.21.15.40:0/2705719952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fd000150 msgr2=0x5631fd00bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.909+0000 7f7af5a00640 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fd000150 0x5631fd00bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.909+0000 7f7af5a00640 1 -- 172.21.15.40:0/2705719952 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffafe0 msgr2=0x5631fd004be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.909+0000 7f7af5a00640 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffafe0 0x5631fd004be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.909+0000 7f7af5a00640 1 -- 172.21.15.40:0/2705719952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631fceec510 con 0x5631fcffa5f0 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.909+0000 7f7af5000640 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffafe0 0x5631fd004be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.909+0000 7f7af5a00640 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fcffa5f0 0x5631fd0046a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x5631fcfee3c0 tx=0x7f7ae400fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25d9b74359eb3cd server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.909+0000 7f7ad7400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.909+0000 7f7ad7400640 1 -- 172.21.15.40:0/2705719952 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4029020 con 0x5631fcffa5f0 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2705719952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ae4003e20 con 0x5631fcffa5f0 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7ad7400640 1 -- 172.21.15.40:0/2705719952 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f7ae4004a40 con 0x5631fcffa5f0 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7ad7400640 1 -- 172.21.15.40:0/2705719952 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7ae4034050 con 0x5631fcffa5f0 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7afb1bd740 10 client.15612 fetch_fsmap learned FSMap version 24 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2705719952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5631fcc68210 con 0x5631fcffa5f0 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7ad7400640 1 -- 172.21.15.40:0/2705719952 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ae4003e20 con 0x5631fcffa5f0 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7ad7400640 1 -- 172.21.15.40:0/2705719952 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7ae4005040 con 0x5631fcffa5f0 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7afb1bd740 10 client.15612 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7afb1bd740 10 client.15612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7afb1bd740 20 client.15612 populate_metadata read hostname 'smithi040' 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5631fd00ffc0 0x5631fd0123b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7afb1bd740 4 client.15612 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:06:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.910+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2705719952 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631fcf63f90 con 0x5631fd00ffc0 2024-04-27T00:06:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.911+0000 7f7af5000640 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5631fd00ffc0 0x5631fd0123b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.912+0000 7f7af5000640 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5631fd00ffc0 0x5631fd0123b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cac5366787dee31 server_cookie=16dd89a37978ccde in_seq=0 out_seq=0 2024-04-27T00:06:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.912+0000 7f7ad7400640 10 client.15612 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.912+0000 7f7ad7400640 1 -- 172.21.15.40:0/2705719952 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631fcf63f90 con 0x5631fd00ffc0 2024-04-27T00:06:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.912+0000 7f7ad7400640 10 client.15612 handle_command_reply: tid=0 2024-04-27T00:06:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.912+0000 7f7afb1bd740 1 client.15612 shutdown 2024-04-27T00:06:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.913+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2705719952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fcffa5f0 msgr2=0x5631fd0046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.913+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fcffa5f0 0x5631fd0046a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x5631fcfee3c0 tx=0x7f7ae400fb60 comp rx=0 tx=0).stop 2024-04-27T00:06:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.913+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2705719952 shutdown_connections 2024-04-27T00:06:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.913+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5631fd00ffc0 0x5631fd0123b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.913+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fd000150 0x5631fd00bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.913+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffafe0 0x5631fd004be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.913+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/2705719952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fcffa5f0 0x5631fd0046a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.913+0000 7f7ad7400640 0 client.15612 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.913+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2705719952 >> 172.21.15.40:0/2705719952 conn(0x5631fcfed450 msgr2=0x5631fcfec650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.913+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2705719952 shutdown_connections 2024-04-27T00:06:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.913+0000 7f7afb1bd740 1 -- 172.21.15.40:0/2705719952 wait complete. 2024-04-27T00:06:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.913+0000 7f7afb1bd740 20 client.15612 trim_cache size 0 max 16384 2024-04-27T00:06:12.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.929+0000 7f7afb1bd740 1 Processor -- start 2024-04-27T00:06:12.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.930+0000 7f7afb1bd740 1 -- start start 2024-04-27T00:06:12.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.930+0000 7f7afb1bd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffa5f0 0x5631fd0049c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.930+0000 7f7afb1bd740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffafe0 0x5631fd00bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.930+0000 7f7afb1bd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fd000150 0x5631fcfec4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.930+0000 7f7afb1bd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01ab9c0 con 0x5631fd000150 2024-04-27T00:06:12.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.930+0000 7f7afb1bd740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01a9290 con 0x5631fcffafe0 2024-04-27T00:06:12.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.930+0000 7f7afb1bd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01a70e0 con 0x5631fcffa5f0 2024-04-27T00:06:12.935 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.930+0000 7f7af5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffa5f0 0x5631fd0049c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.930+0000 7f7af5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffa5f0 0x5631fd0049c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60026/0 (socket says 172.21.15.40:60026) 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.930+0000 7f7af5a00640 1 -- 172.21.15.40:0/1645852964 learned_addr learned my addr 172.21.15.40:0/1645852964 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.930+0000 7f7af5000640 1 --2- 172.21.15.40:0/1645852964 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffafe0 0x5631fd00bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.930+0000 7f7af6400640 1 --2- 172.21.15.40:0/1645852964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fd000150 0x5631fcfec4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.931+0000 7f7af5a00640 1 -- 172.21.15.40:0/1645852964 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffafe0 msgr2=0x5631fd00bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.931+0000 7f7af5a00640 1 --2- 172.21.15.40:0/1645852964 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffafe0 0x5631fd00bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.931+0000 7f7af5a00640 1 -- 172.21.15.40:0/1645852964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fd000150 msgr2=0x5631fcfec4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.931+0000 7f7af5a00640 1 --2- 172.21.15.40:0/1645852964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fd000150 0x5631fcfec4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.931+0000 7f7af5a00640 1 -- 172.21.15.40:0/1645852964 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631fcd34620 con 0x5631fcffa5f0 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.931+0000 7f7af5a00640 1 --2- 172.21.15.40:0/1645852964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffa5f0 0x5631fd0049c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4011810 tx=0x7f7ae4005bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e6c3e7b2f44a9a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.931+0000 7f7ae2000640 1 -- 172.21.15.40:0/1645852964 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae403c020 con 0x5631fcffa5f0 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.931+0000 7f7ae2000640 1 -- 172.21.15.40:0/1645852964 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ae4011320 con 0x5631fcffa5f0 2024-04-27T00:06:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.931+0000 7f7ae2000640 1 -- 172.21.15.40:0/1645852964 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4029070 con 0x5631fcffa5f0 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.932+0000 7f7afb1bd740 1 -- 172.21.15.40:0/1645852964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffa5f0 msgr2=0x5631fd0049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.932+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/1645852964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffa5f0 0x5631fd0049c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4011810 tx=0x7f7ae4005bf0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.932+0000 7f7afb1bd740 1 -- 172.21.15.40:0/1645852964 shutdown_connections 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.932+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/1645852964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fd000150 0x5631fcfec4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.932+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/1645852964 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffafe0 0x5631fd00bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.932+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/1645852964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffa5f0 0x5631fd0049c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.932+0000 7f7afb1bd740 1 -- 172.21.15.40:0/1645852964 >> 172.21.15.40:0/1645852964 conn(0x5631fcfed450 msgr2=0x5631fcfebdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.932+0000 7f7afb1bd740 1 -- 172.21.15.40:0/1645852964 shutdown_connections 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.932+0000 7f7afb1bd740 1 -- 172.21.15.40:0/1645852964 wait complete. 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.933+0000 7f7afb1bd740 1 -- start start 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.933+0000 7f7afb1bd740 1 Processor -- start 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.933+0000 7f7afb1bd740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffa5f0 0x5631fd009750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.933+0000 7f7afb1bd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffafe0 0x5631fd009c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7afb1bd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fd000150 0x5631fd00ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7afb1bd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01a7750 con 0x5631fd000150 2024-04-27T00:06:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7afb1bd740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01a8bc0 con 0x5631fcffa5f0 2024-04-27T00:06:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7afb1bd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7af01a7e20 con 0x5631fcffafe0 2024-04-27T00:06:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7afb1bd740 10 client.0 _ll_register_callbacks cb 0x7f7ae400f9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:06:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7af6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fd000150 0x5631fd00ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7af6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fd000150 0x5631fd00ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46342/0 (socket says 172.21.15.40:46342) 2024-04-27T00:06:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7af6400640 1 -- 172.21.15.40:0/3945382478 learned_addr learned my addr 172.21.15.40:0/3945382478 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7af5a00640 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffa5f0 0x5631fd009750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7af5000640 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffafe0 0x5631fd009c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7af6400640 1 -- 172.21.15.40:0/3945382478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffafe0 msgr2=0x5631fd009c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7af6400640 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffafe0 0x5631fd009c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7af6400640 1 -- 172.21.15.40:0/3945382478 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffa5f0 msgr2=0x5631fd009750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7af6400640 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffa5f0 0x5631fd009750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7af6400640 1 -- 172.21.15.40:0/3945382478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631fcf10cb0 con 0x5631fd000150 2024-04-27T00:06:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.934+0000 7f7af5a00640 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffa5f0 0x5631fd009750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:06:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.935+0000 7f7af6400640 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fd000150 0x5631fd00ac30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80989b0 tx=0x7f7ae8004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d86f2d5374d10c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.935+0000 7f7ad7400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:06:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.935+0000 7f7ad7400640 1 -- 172.21.15.40:0/3945382478 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8020040 con 0x5631fd000150 2024-04-27T00:06:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.935+0000 7f7ad7400640 1 -- 172.21.15.40:0/3945382478 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f7ae8098650 con 0x5631fd000150 2024-04-27T00:06:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.935+0000 7f7afb1bd740 1 -- 172.21.15.40:0/3945382478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ae4011320 con 0x5631fd000150 2024-04-27T00:06:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.935+0000 7f7ad7400640 1 -- 172.21.15.40:0/3945382478 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7ae8096b50 con 0x5631fd000150 2024-04-27T00:06:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.935+0000 7f7afb1bd740 10 client.15618 fetch_fsmap learned FSMap version 24 2024-04-27T00:06:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.935+0000 7f7afb1bd740 1 -- 172.21.15.40:0/3945382478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5631fd076760 con 0x5631fd000150 2024-04-27T00:06:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.935+0000 7f7ad7400640 1 -- 172.21.15.40:0/3945382478 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ae807ae50 con 0x5631fd000150 2024-04-27T00:06:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.936+0000 7f7ad7400640 1 -- 172.21.15.40:0/3945382478 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7ae809e030 con 0x5631fd000150 2024-04-27T00:06:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.936+0000 7f7afb1bd740 10 client.15618 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:06:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.936+0000 7f7afb1bd740 10 client.15618 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:06:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.936+0000 7f7afb1bd740 20 client.15618 populate_metadata read hostname 'smithi040' 2024-04-27T00:06:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.936+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5631fd077100 0x5631fd079520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.936+0000 7f7afb1bd740 4 client.15618 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:06:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.936+0000 7f7afb1bd740 1 -- 172.21.15.40:0/3945382478 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5631fcf5de00 con 0x5631fd077100 2024-04-27T00:06:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.936+0000 7f7af5a00640 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5631fd077100 0x5631fd079520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.937+0000 7f7af5a00640 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5631fd077100 0x5631fd079520 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77656d2aea305225 server_cookie=2fc6d2d90d5eb208 in_seq=0 out_seq=0 2024-04-27T00:06:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.937+0000 7f7ad7400640 10 client.15618 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7ad7400640 1 -- 172.21.15.40:0/3945382478 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5631fcf5de00 con 0x5631fd077100 2024-04-27T00:06:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7ad7400640 10 client.15618 handle_command_reply: tid=0 2024-04-27T00:06:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7afb1bd740 1 client.15618 shutdown 2024-04-27T00:06:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7afb1bd740 1 -- 172.21.15.40:0/3945382478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fd000150 msgr2=0x5631fd00ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fd000150 0x5631fd00ac30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80989b0 tx=0x7f7ae8004f50 comp rx=0 tx=0).stop 2024-04-27T00:06:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7afb1bd740 1 -- 172.21.15.40:0/3945382478 shutdown_connections 2024-04-27T00:06:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5631fd077100 0x5631fd079520 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631fd000150 0x5631fd00ac30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631fcffafe0 0x5631fd009c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7afb1bd740 1 --2- 172.21.15.40:0/3945382478 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5631fcffa5f0 0x5631fd009750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7ad7400640 0 client.15618 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7afb1bd740 1 -- 172.21.15.40:0/3945382478 >> 172.21.15.40:0/3945382478 conn(0x5631fcfed450 msgr2=0x5631fcfec650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7afb1bd740 1 -- 172.21.15.40:0/3945382478 shutdown_connections 2024-04-27T00:06:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.951+0000 7f7afb1bd740 1 -- 172.21.15.40:0/3945382478 wait complete. 2024-04-27T00:06:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.952+0000 7f7afb1bd740 20 client.15618 trim_cache size 0 max 16384 2024-04-27T00:06:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:06:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:06:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:06:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:06:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.952+0000 7f7af6e00640 1 -- 172.21.15.40:0/2432251618 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7ad80a3c30 msgr2=0x7f7ad80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.952+0000 7f7af6e00640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7ad80a3c30 0x7f7ad80a60f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7adc031500 tx=0x7f7adc005d50 comp rx=0 tx=0).stop 2024-04-27T00:06:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.952+0000 7f7af6e00640 1 -- 172.21.15.40:0/2432251618 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7af01060b0 msgr2=0x7f7af01a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.952+0000 7f7af6e00640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7af01060b0 0x7f7af01a2280 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800d680 tx=0x7f7ae800db40 comp rx=0 tx=0).stop 2024-04-27T00:06:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.953+0000 7f7af6e00640 1 -- 172.21.15.40:0/2432251618 shutdown_connections 2024-04-27T00:06:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.953+0000 7f7af6e00640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7af01060b0 0x7f7af01a2280 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.953+0000 7f7af6e00640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af0104fa0 0x7f7af019b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.953+0000 7f7af6e00640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f7ad80a3c30 0x7f7ad80a60f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.953+0000 7f7af6e00640 1 --2- 172.21.15.40:0/2432251618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7af01045e0 0x7f7af019ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.953+0000 7f7af6e00640 1 -- 172.21.15.40:0/2432251618 >> 172.21.15.40:0/2432251618 conn(0x7f7af00fff40 msgr2=0x7f7af0108f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.953+0000 7f7af6e00640 1 -- 172.21.15.40:0/2432251618 shutdown_connections 2024-04-27T00:06:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:12.953+0000 7f7af6e00640 1 -- 172.21.15.40:0/2432251618 wait complete. 2024-04-27T00:06:14.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:14 smithi103 ceph-mon[32952]: pgmap v548: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:06:14.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:14 smithi040 ceph-mon[28095]: pgmap v548: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:06:14.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:14 smithi080 ceph-mon[32362]: pgmap v548: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:06:16.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:16 smithi103 ceph-mon[32952]: pgmap v549: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:06:16.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:16 smithi040 ceph-mon[28095]: pgmap v549: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:06:16.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:16 smithi080 ceph-mon[32362]: pgmap v549: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:06:18.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:18 smithi103 ceph-mon[32952]: pgmap v550: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:18.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:18 smithi040 ceph-mon[28095]: pgmap v550: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:18.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:18 smithi080 ceph-mon[32362]: pgmap v550: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:20.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:20 smithi103 ceph-mon[32952]: pgmap v551: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:20.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:20 smithi040 ceph-mon[28095]: pgmap v551: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:20.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:20 smithi080 ceph-mon[32362]: pgmap v551: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:22.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:22 smithi103 ceph-mon[32952]: pgmap v552: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:22.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:22 smithi040 ceph-mon[28095]: pgmap v552: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:22.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:22 smithi080 ceph-mon[32362]: pgmap v552: 129 pgs: 129 active+clean; 2.8 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:24.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:24 smithi040 ceph-mon[28095]: pgmap v553: 129 pgs: 129 active+clean; 2.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:24.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:24 smithi080 ceph-mon[32362]: pgmap v553: 129 pgs: 129 active+clean; 2.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:24.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:24 smithi103 ceph-mon[32952]: pgmap v553: 129 pgs: 129 active+clean; 2.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:26.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.102+0000 7fb7df200640 1 -- 172.21.15.40:0/782438265 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e0109fd0 msgr2=0x7fb7e010a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:26.103 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.102+0000 7fb7df200640 1 --2- 172.21.15.40:0/782438265 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e0109fd0 0x7fb7e010a450 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009900 tx=0x7fb7c802a3f0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.102+0000 7fb7df200640 1 -- 172.21.15.40:0/782438265 shutdown_connections 2024-04-27T00:06:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.102+0000 7fb7df200640 1 --2- 172.21.15.40:0/782438265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e010a990 0x7fb7e0111220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.102+0000 7fb7df200640 1 --2- 172.21.15.40:0/782438265 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e0109fd0 0x7fb7e010a450 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.102+0000 7fb7df200640 1 --2- 172.21.15.40:0/782438265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e0108de0 0x7fb7e0109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.102+0000 7fb7df200640 1 -- 172.21.15.40:0/782438265 >> 172.21.15.40:0/782438265 conn(0x7fb7e0077dc0 msgr2=0x7fb7e007a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.103+0000 7fb7df200640 1 -- 172.21.15.40:0/782438265 shutdown_connections 2024-04-27T00:06:26.105 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.103+0000 7fb7df200640 1 -- 172.21.15.40:0/782438265 wait complete. 2024-04-27T00:06:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.104+0000 7fb7df200640 1 Processor -- start 2024-04-27T00:06:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.104+0000 7fb7df200640 1 -- start start 2024-04-27T00:06:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.104+0000 7fb7df200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e0108de0 0x7fb7e019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.104+0000 7fb7df200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e0109fd0 0x7fb7e019f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.104+0000 7fb7df200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e010a990 0x7fb7e01a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.104+0000 7fb7df200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e0114940 con 0x7fb7e010a990 2024-04-27T00:06:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.104+0000 7fb7df200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e01147c0 con 0x7fb7e0109fd0 2024-04-27T00:06:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.104+0000 7fb7df200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e0114ac0 con 0x7fb7e0108de0 2024-04-27T00:06:26.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.105+0000 7fb7d5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e0109fd0 0x7fb7e019f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.105+0000 7fb7d5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e0109fd0 0x7fb7e019f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49944/0 (socket says 172.21.15.40:49944) 2024-04-27T00:06:26.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.105+0000 7fb7d5400640 1 -- 172.21.15.40:0/1647108307 learned_addr learned my addr 172.21.15.40:0/1647108307 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:26.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.105+0000 7fb7dde00640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e0108de0 0x7fb7e019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.105+0000 7fb7de800640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e010a990 0x7fb7e01a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.105+0000 7fb7d5400640 1 -- 172.21.15.40:0/1647108307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e0108de0 msgr2=0x7fb7e019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:26.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.105+0000 7fb7d5400640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e0108de0 0x7fb7e019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.105+0000 7fb7d5400640 1 -- 172.21.15.40:0/1647108307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e010a990 msgr2=0x7fb7e01a6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:26.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.105+0000 7fb7d5400640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e010a990 0x7fb7e01a6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.105+0000 7fb7d5400640 1 -- 172.21.15.40:0/1647108307 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e01a6c00 con 0x7fb7e0109fd0 2024-04-27T00:06:26.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.105+0000 7fb7dde00640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e0108de0 0x7fb7e019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:06:26.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.106+0000 7fb7d5400640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e0109fd0 0x7fb7e019f520 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005ab0 tx=0x7fb7c8003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:26.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.107+0000 7fb7d7400640 1 -- 172.21.15.40:0/1647108307 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c803f070 con 0x7fb7e0109fd0 2024-04-27T00:06:26.114 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.107+0000 7fb7d7400640 1 -- 172.21.15.40:0/1647108307 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7c8031c50 con 0x7fb7e0109fd0 2024-04-27T00:06:26.114 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.107+0000 7fb7d7400640 1 -- 172.21.15.40:0/1647108307 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8035030 con 0x7fb7e0109fd0 2024-04-27T00:06:26.114 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.107+0000 7fb7df200640 1 -- 172.21.15.40:0/1647108307 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e01a6e90 con 0x7fb7e0109fd0 2024-04-27T00:06:26.114 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.107+0000 7fb7df200640 1 -- 172.21.15.40:0/1647108307 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e01a7300 con 0x7fb7e0109fd0 2024-04-27T00:06:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.111+0000 7fb7d7400640 1 -- 172.21.15.40:0/1647108307 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb7c8042070 con 0x7fb7e0109fd0 2024-04-27T00:06:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.111+0000 7fb7e79b1740 1 Processor -- start 2024-04-27T00:06:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.111+0000 7fb7e79b1740 1 -- start start 2024-04-27T00:06:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7e79b1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 0x5562ec4ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7e79b1740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4d32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7e79b1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4d3940 0x5562ec4d5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7e79b1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5562ec432380 con 0x5562ec4d3940 2024-04-27T00:06:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7e79b1740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5562ec3cec30 con 0x5562ec4ce7d0 2024-04-27T00:06:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7e79b1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5562ec42e340 con 0x5562ec4cdde0 2024-04-27T00:06:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7de800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 0x5562ec4ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7de800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 0x5562ec4ce200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47636/0 (socket says 172.21.15.40:47636) 2024-04-27T00:06:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7de800640 1 -- 172.21.15.40:0/1922743063 learned_addr learned my addr 172.21.15.40:0/1922743063 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7d5400640 1 --2- 172.21.15.40:0/1922743063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4d3940 0x5562ec4d5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7dde00640 1 --2- 172.21.15.40:0/1922743063 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4d32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7de800640 1 -- 172.21.15.40:0/1922743063 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 msgr2=0x5562ec4d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7de800640 1 --2- 172.21.15.40:0/1922743063 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4d32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7de800640 1 -- 172.21.15.40:0/1922743063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4d3940 msgr2=0x5562ec4d5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7de800640 1 --2- 172.21.15.40:0/1922743063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4d3940 0x5562ec4d5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7de800640 1 -- 172.21.15.40:0/1922743063 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562ec3f0940 con 0x5562ec4cdde0 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7dde00640 1 --2- 172.21.15.40:0/1922743063 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4d32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.112+0000 7fb7d7400640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7c40a3cb0 0x7fb7c40a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.113+0000 7fb7d7400640 1 -- 172.21.15.40:0/1647108307 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fb7c80f0ff0 con 0x7fb7e0109fd0 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.113+0000 7fb7dde00640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7c40a3cb0 0x7fb7c40a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.113+0000 7fb7de800640 1 --2- 172.21.15.40:0/1922743063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 0x5562ec4ce200 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009ca0 tx=0x7fb7d8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d31a314b6ccf4b5a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.113+0000 7fb7cf000640 1 -- 172.21.15.40:0/1922743063 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d801e070 con 0x5562ec4cdde0 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.113+0000 7fb7cf000640 1 -- 172.21.15.40:0/1922743063 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7d8009e50 con 0x5562ec4cdde0 2024-04-27T00:06:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.113+0000 7fb7cf000640 1 -- 172.21.15.40:0/1922743063 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d800ea50 con 0x5562ec4cdde0 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.114+0000 7fb7e79b1740 1 -- 172.21.15.40:0/1922743063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 msgr2=0x5562ec4ce200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.114+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/1922743063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 0x5562ec4ce200 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009ca0 tx=0x7fb7d8011810 comp rx=0 tx=0).stop 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.114+0000 7fb7dde00640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7c40a3cb0 0x7fb7c40a6170 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x5562ec4cebd0 tx=0x7fb7d0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.114+0000 7fb7e79b1740 1 -- 172.21.15.40:0/1922743063 shutdown_connections 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.114+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/1922743063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4d3940 0x5562ec4d5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.114+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/1922743063 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4d32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.114+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/1922743063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 0x5562ec4ce200 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.114+0000 7fb7e79b1740 1 -- 172.21.15.40:0/1922743063 >> 172.21.15.40:0/1922743063 conn(0x5562ec4c9450 msgr2=0x5562ec4c9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.114+0000 7fb7e79b1740 1 -- 172.21.15.40:0/1922743063 shutdown_connections 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.114+0000 7fb7e79b1740 1 -- 172.21.15.40:0/1922743063 wait complete. 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.115+0000 7fb7e79b1740 1 -- start start 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.115+0000 7fb7e79b1740 1 Processor -- start 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.115+0000 7fb7e79b1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4cdde0 0x5562ec4d80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.128 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.115+0000 7fb7e79b1740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4d8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.115+0000 7fb7e79b1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4d3940 0x5562ec4df5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.115+0000 7fb7e79b1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5562ec4d68f0 con 0x5562ec4cdde0 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.115+0000 7fb7e79b1740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5562ec3b9c60 con 0x5562ec4ce7d0 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.115+0000 7fb7e79b1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5562ec4d6a70 con 0x5562ec4d3940 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7e79b1740 10 client.0 _ll_register_callbacks cb 0x5562ec3e5e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7dde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4cdde0 0x5562ec4d80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7d5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4d8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7dde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4cdde0 0x5562ec4d80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56282/0 (socket says 172.21.15.40:56282) 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7dde00640 1 -- 172.21.15.40:0/2898166850 learned_addr learned my addr 172.21.15.40:0/2898166850 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7de800640 1 --2- 172.21.15.40:0/2898166850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4d3940 0x5562ec4df5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7d5400640 1 -- 172.21.15.40:0/2898166850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4d3940 msgr2=0x5562ec4df5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7d5400640 1 --2- 172.21.15.40:0/2898166850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4d3940 0x5562ec4df5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7d5400640 1 -- 172.21.15.40:0/2898166850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4cdde0 msgr2=0x5562ec4d80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:26.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7d5400640 1 --2- 172.21.15.40:0/2898166850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4cdde0 0x5562ec4d80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7d5400640 1 -- 172.21.15.40:0/2898166850 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562ec3c8510 con 0x5562ec4ce7d0 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7d5400640 1 --2- 172.21.15.40:0/2898166850 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4d8620 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80ee5c0 tx=0x7fb7c8003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37a61cc196bc10d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.116+0000 7fb7c3400640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.117+0000 7fb7c3400640 1 -- 172.21.15.40:0/2898166850 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8049050 con 0x5562ec4ce7d0 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.117+0000 7fb7c3400640 1 -- 172.21.15.40:0/2898166850 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fb7c80f3110 con 0x5562ec4ce7d0 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.117+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2898166850 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7d8009e50 con 0x5562ec4ce7d0 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.117+0000 7fb7c3400640 1 -- 172.21.15.40:0/2898166850 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb7c80bc7a0 con 0x5562ec4ce7d0 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.117+0000 7fb7e79b1740 10 client.25745 fetch_fsmap learned FSMap version 24 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.117+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2898166850 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5562ec144210 con 0x5562ec4ce7d0 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.117+0000 7fb7c3400640 1 -- 172.21.15.40:0/2898166850 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7d8009e50 con 0x5562ec4ce7d0 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.118+0000 7fb7c3400640 1 -- 172.21.15.40:0/2898166850 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb7c803b460 con 0x5562ec4ce7d0 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.118+0000 7fb7e79b1740 10 client.25745 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.118+0000 7fb7e79b1740 10 client.25745 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.118+0000 7fb7e79b1740 20 client.25745 populate_metadata read hostname 'smithi040' 2024-04-27T00:06:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.118+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/2898166850 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562ec4e3b30 0x5562ec4e5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.118+0000 7fb7e79b1740 4 client.25745 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.118+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2898166850 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562ec43ff90 con 0x5562ec4e3b30 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.118+0000 7fb7dde00640 1 --2- 172.21.15.40:0/2898166850 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562ec4e3b30 0x5562ec4e5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.119+0000 7fb7dde00640 1 --2- 172.21.15.40:0/2898166850 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562ec4e3b30 0x5562ec4e5f40 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cceb52bba7758ed server_cookie=f322dfac208d7740 in_seq=0 out_seq=0 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.119+0000 7fb7c3400640 10 client.25745 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.120+0000 7fb7c3400640 1 -- 172.21.15.40:0/2898166850 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5562ec43ff90 con 0x5562ec4e3b30 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.120+0000 7fb7c3400640 10 client.25745 handle_command_reply: tid=0 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7e79b1740 1 client.25745 shutdown 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2898166850 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 msgr2=0x5562ec4d8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/2898166850 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4d8620 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80ee5c0 tx=0x7fb7c8003f90 comp rx=0 tx=0).stop 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2898166850 shutdown_connections 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/2898166850 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562ec4e3b30 0x5562ec4e5f40 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/2898166850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4d3940 0x5562ec4df5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/2898166850 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4d8620 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/2898166850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4cdde0 0x5562ec4d80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7c3400640 0 client.25745 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:26.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2898166850 >> 172.21.15.40:0/2898166850 conn(0x5562ec4c9450 msgr2=0x5562ec4c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:26.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2898166850 shutdown_connections 2024-04-27T00:06:26.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2898166850 wait complete. 2024-04-27T00:06:26.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.121+0000 7fb7e79b1740 20 client.25745 trim_cache size 0 max 16384 2024-04-27T00:06:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.139+0000 7fb7e79b1740 1 Processor -- start 2024-04-27T00:06:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.139+0000 7fb7e79b1740 1 -- start start 2024-04-27T00:06:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.139+0000 7fb7e79b1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4c9450 0x5562ec4cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.139+0000 7fb7e79b1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 0x5562ec4cce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.139+0000 7fb7e79b1740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4df5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.139+0000 7fb7e79b1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e01b41f0 con 0x5562ec4c9450 2024-04-27T00:06:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.139+0000 7fb7e79b1740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e01b1ac0 con 0x5562ec4ce7d0 2024-04-27T00:06:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.139+0000 7fb7e79b1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e01af910 con 0x5562ec4cdde0 2024-04-27T00:06:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.140+0000 7fb7de800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4df5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.140+0000 7fb7de800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4df5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49968/0 (socket says 172.21.15.40:49968) 2024-04-27T00:06:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.140+0000 7fb7de800640 1 -- 172.21.15.40:0/2227853652 learned_addr learned my addr 172.21.15.40:0/2227853652 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.140+0000 7fb7d5400640 1 --2- 172.21.15.40:0/2227853652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 0x5562ec4cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.140+0000 7fb7dde00640 1 --2- 172.21.15.40:0/2227853652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4c9450 0x5562ec4cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.140+0000 7fb7de800640 1 -- 172.21.15.40:0/2227853652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 msgr2=0x5562ec4cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.140+0000 7fb7de800640 1 --2- 172.21.15.40:0/2227853652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 0x5562ec4cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.140+0000 7fb7de800640 1 -- 172.21.15.40:0/2227853652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4c9450 msgr2=0x5562ec4cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.140+0000 7fb7de800640 1 --2- 172.21.15.40:0/2227853652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4c9450 0x5562ec4cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.140+0000 7fb7de800640 1 -- 172.21.15.40:0/2227853652 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562ec210620 con 0x5562ec4ce7d0 2024-04-27T00:06:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.140+0000 7fb7d5400640 1 --2- 172.21.15.40:0/2227853652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 0x5562ec4cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:06:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.141+0000 7fb7de800640 1 --2- 172.21.15.40:0/2227853652 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4df5b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009800 tx=0x7fb7d8009a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78b8da5cfd2e79f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.141+0000 7fb7ce600640 1 -- 172.21.15.40:0/2227853652 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8027030 con 0x5562ec4ce7d0 2024-04-27T00:06:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.141+0000 7fb7ce600640 1 -- 172.21.15.40:0/2227853652 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7d8005630 con 0x5562ec4ce7d0 2024-04-27T00:06:26.142 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.141+0000 7fb7ce600640 1 -- 172.21.15.40:0/2227853652 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d801e040 con 0x5562ec4ce7d0 2024-04-27T00:06:26.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.142+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2227853652 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 msgr2=0x5562ec4df5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:26.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.142+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/2227853652 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4df5b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009800 tx=0x7fb7d8009a10 comp rx=0 tx=0).stop 2024-04-27T00:06:26.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.142+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2227853652 shutdown_connections 2024-04-27T00:06:26.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.142+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/2227853652 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4ce7d0 0x5562ec4df5b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.142+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/2227853652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4cdde0 0x5562ec4cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.145 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.142+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/2227853652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4c9450 0x5562ec4cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.142+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2227853652 >> 172.21.15.40:0/2227853652 conn(0x5562ec4d3410 msgr2=0x5562ec4cc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.142+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2227853652 shutdown_connections 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.142+0000 7fb7e79b1740 1 -- 172.21.15.40:0/2227853652 wait complete. 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.142+0000 7fb7e79b1740 1 -- start start 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.143+0000 7fb7e79b1740 1 Processor -- start 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.143+0000 7fb7e79b1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4c9450 0x5562ec4dd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.143+0000 7fb7e79b1740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4cdde0 0x5562ec4dd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.143+0000 7fb7e79b1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4ce7d0 0x5562ec4de540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.143+0000 7fb7e79b1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e01aff80 con 0x5562ec4c9450 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.143+0000 7fb7e79b1740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e01b13f0 con 0x5562ec4cdde0 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.143+0000 7fb7e79b1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e01b0650 con 0x5562ec4ce7d0 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.143+0000 7fb7e79b1740 10 client.0 _ll_register_callbacks cb 0x7fb7c8009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:06:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.143+0000 7fb7dde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4c9450 0x5562ec4dd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.143+0000 7fb7de800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4ce7d0 0x5562ec4de540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.143+0000 7fb7de800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4ce7d0 0x5562ec4de540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47656/0 (socket says 172.21.15.40:47656) 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.143+0000 7fb7de800640 1 -- 172.21.15.40:0/4271794871 learned_addr learned my addr 172.21.15.40:0/4271794871 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.144+0000 7fb7d5400640 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4cdde0 0x5562ec4dd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.144+0000 7fb7de800640 1 -- 172.21.15.40:0/4271794871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4cdde0 msgr2=0x5562ec4dd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.144+0000 7fb7de800640 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4cdde0 0x5562ec4dd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.144+0000 7fb7de800640 1 -- 172.21.15.40:0/4271794871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4c9450 msgr2=0x5562ec4dd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.144+0000 7fb7de800640 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4c9450 0x5562ec4dd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.144+0000 7fb7de800640 1 -- 172.21.15.40:0/4271794871 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562ec3eccb0 con 0x5562ec4ce7d0 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.144+0000 7fb7d5400640 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4cdde0 0x5562ec4dd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.144+0000 7fb7de800640 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4ce7d0 0x5562ec4de540 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8014040 tx=0x7fb7d8005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a50bd45dc319f203 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.144+0000 7fb7c3400640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.144+0000 7fb7c3400640 1 -- 172.21.15.40:0/4271794871 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d802f070 con 0x5562ec4ce7d0 2024-04-27T00:06:26.147 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.144+0000 7fb7e79b1740 1 -- 172.21.15.40:0/4271794871 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5562ec439e00 con 0x5562ec4ce7d0 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.144+0000 7fb7c3400640 1 -- 172.21.15.40:0/4271794871 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fb7d8004880 con 0x5562ec4ce7d0 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.145+0000 7fb7c3400640 1 -- 172.21.15.40:0/4271794871 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb7d803b050 con 0x5562ec4ce7d0 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.145+0000 7fb7e79b1740 10 client.25780 fetch_fsmap learned FSMap version 24 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.145+0000 7fb7e79b1740 1 -- 172.21.15.40:0/4271794871 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5562ec54a240 con 0x5562ec4ce7d0 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.145+0000 7fb7c3400640 1 -- 172.21.15.40:0/4271794871 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5562ec439e00 con 0x5562ec4ce7d0 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.145+0000 7fb7c3400640 1 -- 172.21.15.40:0/4271794871 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb7d8023cc0 con 0x5562ec4ce7d0 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.145+0000 7fb7e79b1740 10 client.25780 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.145+0000 7fb7e79b1740 10 client.25780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.145+0000 7fb7e79b1740 20 client.25780 populate_metadata read hostname 'smithi040' 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.145+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562ec4d3940 0x5562ec54c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.145+0000 7fb7e79b1740 4 client.25780 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.145+0000 7fb7e79b1740 1 -- 172.21.15.40:0/4271794871 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5562ec3b54b0 con 0x5562ec4d3940 2024-04-27T00:06:26.148 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.146+0000 7fb7dde00640 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562ec4d3940 0x5562ec54c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.146+0000 7fb7dde00640 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562ec4d3940 0x5562ec54c980 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25f0775d0497c219 server_cookie=36746b69509ec427 in_seq=0 out_seq=0 2024-04-27T00:06:26.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.146+0000 7fb7c3400640 10 client.25780 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7c3400640 1 -- 172.21.15.40:0/4271794871 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5562ec3b54b0 con 0x5562ec4d3940 2024-04-27T00:06:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7c3400640 10 client.25780 handle_command_reply: tid=0 2024-04-27T00:06:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7e79b1740 1 client.25780 shutdown 2024-04-27T00:06:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7e79b1740 1 -- 172.21.15.40:0/4271794871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4ce7d0 msgr2=0x5562ec4de540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4ce7d0 0x5562ec4de540 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8014040 tx=0x7fb7d8005970 comp rx=0 tx=0).stop 2024-04-27T00:06:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7e79b1740 1 -- 172.21.15.40:0/4271794871 shutdown_connections 2024-04-27T00:06:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562ec4d3940 0x5562ec54c980 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ec4ce7d0 0x5562ec4de540 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ec4cdde0 0x5562ec4dd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7e79b1740 1 --2- 172.21.15.40:0/4271794871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ec4c9450 0x5562ec4dd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7e79b1740 1 -- 172.21.15.40:0/4271794871 >> 172.21.15.40:0/4271794871 conn(0x5562ec4d3410 msgr2=0x5562ec4c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7c3400640 0 client.25780 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7e79b1740 1 -- 172.21.15.40:0/4271794871 shutdown_connections 2024-04-27T00:06:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7e79b1740 1 -- 172.21.15.40:0/4271794871 wait complete. 2024-04-27T00:06:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.162+0000 7fb7e79b1740 20 client.25780 trim_cache size 0 max 16384 2024-04-27T00:06:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:06:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:06:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:06:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:06:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.163+0000 7fb7df200640 1 -- 172.21.15.40:0/1647108307 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7c40a3cb0 msgr2=0x7fb7c40a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.163+0000 7fb7df200640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7c40a3cb0 0x7fb7c40a6170 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x5562ec4cebd0 tx=0x7fb7d0009500 comp rx=0 tx=0).stop 2024-04-27T00:06:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.163+0000 7fb7df200640 1 -- 172.21.15.40:0/1647108307 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e0109fd0 msgr2=0x7fb7e019f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.163+0000 7fb7df200640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e0109fd0 0x7fb7e019f520 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005ab0 tx=0x7fb7c8003b90 comp rx=0 tx=0).stop 2024-04-27T00:06:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.163+0000 7fb7df200640 1 -- 172.21.15.40:0/1647108307 shutdown_connections 2024-04-27T00:06:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.163+0000 7fb7df200640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7e010a990 0x7fb7e01a6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.163+0000 7fb7df200640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb7e0109fd0 0x7fb7e019f520 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.163+0000 7fb7df200640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb7c40a3cb0 0x7fb7c40a6170 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.163+0000 7fb7df200640 1 --2- 172.21.15.40:0/1647108307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7e0108de0 0x7fb7e019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.163+0000 7fb7df200640 1 -- 172.21.15.40:0/1647108307 >> 172.21.15.40:0/1647108307 conn(0x7fb7e0077dc0 msgr2=0x7fb7e010d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.164+0000 7fb7df200640 1 -- 172.21.15.40:0/1647108307 shutdown_connections 2024-04-27T00:06:26.165 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:26.164+0000 7fb7df200640 1 -- 172.21.15.40:0/1647108307 wait complete. 2024-04-27T00:06:26.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:26 smithi040 ceph-mon[28095]: pgmap v554: 129 pgs: 129 active+clean; 2.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:26.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:26 smithi080 ceph-mon[32362]: pgmap v554: 129 pgs: 129 active+clean; 2.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:26.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:26 smithi103 ceph-mon[32952]: pgmap v554: 129 pgs: 129 active+clean; 2.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:28.399 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:28 smithi103 ceph-mon[32952]: pgmap v555: 129 pgs: 129 active+clean; 2.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:06:28.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:28 smithi040 ceph-mon[28095]: pgmap v555: 129 pgs: 129 active+clean; 2.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:06:28.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:28 smithi080 ceph-mon[32362]: pgmap v555: 129 pgs: 129 active+clean; 2.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:06:29.991 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.991 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.991 INFO:tasks.workunit.client.0.smithi040.stdout:127/600: open fd = 3 2024-04-27T00:06:29.991 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.991 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:128/600: open fd = 3 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:129/600: open fd = 3 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:130/600: open fd = 3 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.992 INFO:tasks.workunit.client.0.smithi040.stdout:131/600: open fd = 3 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:132/600: open fd = 3 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:133/600: open fd = 3 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:134/600: open fd = 3 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.993 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:135/600: open fd = 3 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:136/600: open fd = 3 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:137/600: open fd = 3 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:138/600: open fd = 3 2024-04-27T00:06:29.994 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:139/600: open fd = 3 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:140/600: open fd = 3 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:141/600: open fd = 3 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.995 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:142/600: open fd = 3 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:143/600: open fd = 3 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:144/600: open fd = 3 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:145/600: open fd = 3 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.996 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:146/600: open fd = 3 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:147/600: open fd = 3 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:148/600: open fd = 3 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.997 INFO:tasks.workunit.client.0.smithi040.stdout:149/600: open fd = 3 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:150/600: open fd = 3 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:151/600: open fd = 3 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:152/600: open fd = 3 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.998 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:153/600: open fd = 3 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:154/600: open fd = 3 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:155/600: open fd = 3 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:156/600: open fd = 3 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:29.999 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:157/600: open fd = 3 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:158/600: open fd = 3 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:159/600: open fd = 3 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:160/600: open fd = 3 2024-04-27T00:06:30.000 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:161/600: open fd = 3 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:162/600: open fd = 3 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:163/600: open fd = 3 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.001 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:164/600: open fd = 3 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:165/600: open fd = 3 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:166/600: open fd = 3 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:167/600: open fd = 3 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.002 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:168/600: open fd = 3 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:169/600: open fd = 3 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:170/600: open fd = 3 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:171/600: open fd = 3 2024-04-27T00:06:30.003 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:172/600: open fd = 3 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:173/600: open fd = 3 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:174/600: open fd = 3 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.004 INFO:tasks.workunit.client.0.smithi040.stdout:175/600: open fd = 3 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:176/600: open fd = 3 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:177/600: open fd = 3 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:178/600: open fd = 3 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.005 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:179/600: open fd = 3 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:180/600: open fd = 3 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:181/600: open fd = 3 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:182/600: open fd = 3 2024-04-27T00:06:30.006 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:183/600: open fd = 3 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:184/600: open fd = 3 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:185/600: open fd = 3 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.007 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.008 INFO:tasks.workunit.client.0.smithi040.stdout:186/600: open fd = 3 2024-04-27T00:06:30.008 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.008 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.008 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.008 INFO:tasks.workunit.client.0.smithi040.stdout:187/600: open fd = 3 2024-04-27T00:06:30.008 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.008 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.008 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:06:30.008 INFO:tasks.workunit.client.0.smithi040.stdout:188/600: open fd = 3 2024-04-27T00:06:30.008 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.008 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:06:30.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:30 smithi040 ceph-mon[28095]: pgmap v556: 129 pgs: 129 active+clean; 2.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:06:30.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:30 smithi080 ceph-mon[32362]: pgmap v556: 129 pgs: 129 active+clean; 2.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:06:30.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:30 smithi103 ceph-mon[32952]: pgmap v556: 129 pgs: 129 active+clean; 2.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:06:32.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:32 smithi040 ceph-mon[28095]: pgmap v557: 129 pgs: 129 active+clean; 2.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-27T00:06:32.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:32 smithi080 ceph-mon[32362]: pgmap v557: 129 pgs: 129 active+clean; 2.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-27T00:06:32.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:32 smithi103 ceph-mon[32952]: pgmap v557: 129 pgs: 129 active+clean; 2.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-27T00:06:34.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:34 smithi040 ceph-mon[28095]: pgmap v558: 129 pgs: 129 active+clean; 3.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:34.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:34 smithi080 ceph-mon[32362]: pgmap v558: 129 pgs: 129 active+clean; 3.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:34.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:34 smithi103 ceph-mon[32952]: pgmap v558: 129 pgs: 129 active+clean; 3.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:36.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:36 smithi040 ceph-mon[28095]: pgmap v559: 129 pgs: 129 active+clean; 3.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:36.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:36 smithi080 ceph-mon[32362]: pgmap v559: 129 pgs: 129 active+clean; 3.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:36.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:36 smithi103 ceph-mon[32952]: pgmap v559: 129 pgs: 129 active+clean; 3.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:38.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:38 smithi040 ceph-mon[28095]: pgmap v560: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:38.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:38 smithi080 ceph-mon[32362]: pgmap v560: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:38.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:38 smithi103 ceph-mon[32952]: pgmap v560: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.314+0000 7fee19800640 1 -- 172.21.15.40:0/3608889001 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee141016d0 msgr2=0x7fee14101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.314+0000 7fee19800640 1 --2- 172.21.15.40:0/3608889001 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee141016d0 0x7fee14101af0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fee04009900 tx=0x7fee0402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.315+0000 7fee19800640 1 -- 172.21.15.40:0/3608889001 shutdown_connections 2024-04-27T00:06:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.315+0000 7fee19800640 1 --2- 172.21.15.40:0/3608889001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee14103280 0x7fee14109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.315+0000 7fee19800640 1 --2- 172.21.15.40:0/3608889001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee141028c0 0x7fee14102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.315+0000 7fee19800640 1 --2- 172.21.15.40:0/3608889001 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee141016d0 0x7fee14101af0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.315+0000 7fee19800640 1 -- 172.21.15.40:0/3608889001 >> 172.21.15.40:0/3608889001 conn(0x7fee140fcc60 msgr2=0x7fee140ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.315+0000 7fee19800640 1 -- 172.21.15.40:0/3608889001 shutdown_connections 2024-04-27T00:06:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.315+0000 7fee19800640 1 -- 172.21.15.40:0/3608889001 wait complete. 2024-04-27T00:06:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.316+0000 7fee19800640 1 Processor -- start 2024-04-27T00:06:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.316+0000 7fee19800640 1 -- start start 2024-04-27T00:06:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee19800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee141016d0 0x7fee1419ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee19800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee141028c0 0x7fee1419b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee19800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee14103280 0x7fee141a2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee19800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee1410d200 con 0x7fee141016d0 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee19800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fee1410d080 con 0x7fee141028c0 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee19800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee1410d380 con 0x7fee14103280 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee141016d0 0x7fee1419ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee0be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee141028c0 0x7fee1419b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee141016d0 0x7fee1419ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43512/0 (socket says 172.21.15.40:43512) 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee13e00640 1 -- 172.21.15.40:0/214534958 learned_addr learned my addr 172.21.15.40:0/214534958 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee18e00640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee14103280 0x7fee141a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee0be00640 1 -- 172.21.15.40:0/214534958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee14103280 msgr2=0x7fee141a2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.317+0000 7fee0be00640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee14103280 0x7fee141a2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.318+0000 7fee0be00640 1 -- 172.21.15.40:0/214534958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee141016d0 msgr2=0x7fee1419ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.318+0000 7fee0be00640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee141016d0 0x7fee1419ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.318+0000 7fee0be00640 1 -- 172.21.15.40:0/214534958 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee141a28b0 con 0x7fee141028c0 2024-04-27T00:06:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.318+0000 7fee0be00640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee141028c0 0x7fee1419b1b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00e9f0 tx=0x7fedfc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.318+0000 7fee11600640 1 -- 172.21.15.40:0/214534958 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc00ccc0 con 0x7fee141028c0 2024-04-27T00:06:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.318+0000 7fee19800640 1 -- 172.21.15.40:0/214534958 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee141a2ba0 con 0x7fee141028c0 2024-04-27T00:06:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.318+0000 7fee11600640 1 -- 172.21.15.40:0/214534958 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedfc0108e0 con 0x7fee141028c0 2024-04-27T00:06:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.318+0000 7fee11600640 1 -- 172.21.15.40:0/214534958 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc018880 con 0x7fee141028c0 2024-04-27T00:06:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.318+0000 7fee19800640 1 -- 172.21.15.40:0/214534958 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee141a3140 con 0x7fee141028c0 2024-04-27T00:06:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.321+0000 7fee1dd93740 1 Processor -- start 2024-04-27T00:06:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee1dd93740 1 -- start start 2024-04-27T00:06:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee11600640 1 -- 172.21.15.40:0/214534958 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fedfc010430 con 0x7fee141028c0 2024-04-27T00:06:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee1dd93740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 0x563d05f1f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee1dd93740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f1f7d0 0x563d05f242d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee1dd93740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f24940 0x563d05f26dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee1dd93740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563d05e83380 con 0x563d05f1f7d0 2024-04-27T00:06:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee1dd93740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x563d05e1fc30 con 0x563d05f24940 2024-04-27T00:06:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee1dd93740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563d05e7f340 con 0x563d05f1ede0 2024-04-27T00:06:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee18e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 0x563d05f1f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f1f7d0 0x563d05f242d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee0be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f24940 0x563d05f26dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee18e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 0x563d05f1f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36176/0 (socket says 172.21.15.40:36176) 2024-04-27T00:06:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.322+0000 7fee18e00640 1 -- 172.21.15.40:0/3997345330 learned_addr learned my addr 172.21.15.40:0/3997345330 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.323+0000 7fee18e00640 1 -- 172.21.15.40:0/3997345330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f24940 msgr2=0x563d05f26dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.323+0000 7fee18e00640 1 --2- 172.21.15.40:0/3997345330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f24940 0x563d05f26dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.323+0000 7fee18e00640 1 -- 172.21.15.40:0/3997345330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f1f7d0 msgr2=0x563d05f242d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.323+0000 7fee18e00640 1 --2- 172.21.15.40:0/3997345330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f1f7d0 0x563d05f242d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.323+0000 7fee18e00640 1 -- 172.21.15.40:0/3997345330 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d05e41940 con 0x563d05f1ede0 2024-04-27T00:06:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.323+0000 7fee18e00640 1 --2- 172.21.15.40:0/3997345330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 0x563d05f1f200 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009fd0 tx=0x7fee0c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e56cf25111a2d3e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.323+0000 7fee09600640 1 -- 172.21.15.40:0/3997345330 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0c004d20 con 0x563d05f1ede0 2024-04-27T00:06:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.323+0000 7fee09600640 1 -- 172.21.15.40:0/3997345330 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee0c004050 con 0x563d05f1ede0 2024-04-27T00:06:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.323+0000 7fee09600640 1 -- 172.21.15.40:0/3997345330 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0c005280 con 0x563d05f1ede0 2024-04-27T00:06:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.324+0000 7fee11600640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fedf80a3cb0 0x7fedf80a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.324+0000 7fee1dd93740 1 -- 172.21.15.40:0/3997345330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 msgr2=0x563d05f1f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.324+0000 7fee1dd93740 1 --2- 172.21.15.40:0/3997345330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 0x563d05f1f200 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009fd0 tx=0x7fee0c0118d0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.324+0000 7fee11600640 1 -- 172.21.15.40:0/214534958 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fedfc0d0c10 con 0x7fee141028c0 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.324+0000 7fee1dd93740 1 -- 172.21.15.40:0/3997345330 shutdown_connections 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.324+0000 7fee1dd93740 1 --2- 172.21.15.40:0/3997345330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f24940 0x563d05f26dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.324+0000 7fee1dd93740 1 --2- 172.21.15.40:0/3997345330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f1f7d0 0x563d05f242d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.324+0000 7fee1dd93740 1 --2- 172.21.15.40:0/3997345330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 0x563d05f1f200 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.324+0000 7fee1dd93740 1 -- 172.21.15.40:0/3997345330 >> 172.21.15.40:0/3997345330 conn(0x563d05f1a450 msgr2=0x563d05f1a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.324+0000 7fee13e00640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fedf80a3cb0 0x7fedf80a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.324+0000 7fee1dd93740 1 -- 172.21.15.40:0/3997345330 shutdown_connections 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.324+0000 7fee1dd93740 1 -- 172.21.15.40:0/3997345330 wait complete. 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.325+0000 7fee1dd93740 1 -- start start 2024-04-27T00:06:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.325+0000 7fee13e00640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fedf80a3cb0 0x7fedf80a6170 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x563d05f1fbd0 tx=0x7fee040314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.325+0000 7fee1dd93740 1 Processor -- start 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.325+0000 7fee1dd93740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f1ede0 0x563d05f28fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.325+0000 7fee1dd93740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f1f7d0 0x563d05f29510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.325+0000 7fee1dd93740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f24940 0x563d05f30550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee1dd93740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563d05f278f0 con 0x563d05f1f7d0 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee1dd93740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x563d05e0ac60 con 0x563d05f1ede0 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee1dd93740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563d05f27a70 con 0x563d05f24940 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee1dd93740 10 client.0 _ll_register_callbacks cb 0x563d05e36e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee0be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f1f7d0 0x563d05f29510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee0be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f1f7d0 0x563d05f29510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43534/0 (socket says 172.21.15.40:43534) 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee0be00640 1 -- 172.21.15.40:0/3720827789 learned_addr learned my addr 172.21.15.40:0/3720827789 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee18e00640 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f24940 0x563d05f30550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee13e00640 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f1ede0 0x563d05f28fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee0be00640 1 -- 172.21.15.40:0/3720827789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f24940 msgr2=0x563d05f30550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee0be00640 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f24940 0x563d05f30550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee0be00640 1 -- 172.21.15.40:0/3720827789 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f1ede0 msgr2=0x563d05f28fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee0be00640 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f1ede0 0x563d05f28fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.326+0000 7fee0be00640 1 -- 172.21.15.40:0/3720827789 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d05e19510 con 0x563d05f1f7d0 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.327+0000 7fee0be00640 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f1f7d0 0x563d05f29510 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fedfc098ea0 tx=0x7fedfc00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473a66f779970f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.327+0000 7fee02000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.327+0000 7fee02000640 1 -- 172.21.15.40:0/3720827789 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc010720 con 0x563d05f1f7d0 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.327+0000 7fee02000640 1 -- 172.21.15.40:0/3720827789 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fedfc0ce410 con 0x563d05f1f7d0 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.327+0000 7fee1dd93740 1 -- 172.21.15.40:0/3720827789 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee0c004050 con 0x563d05f1f7d0 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.327+0000 7fee02000640 1 -- 172.21.15.40:0/3720827789 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fedfc09ca40 con 0x563d05f1f7d0 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.327+0000 7fee1dd93740 10 client.15648 fetch_fsmap learned FSMap version 24 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.327+0000 7fee1dd93740 1 -- 172.21.15.40:0/3720827789 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563d05b95210 con 0x563d05f1f7d0 2024-04-27T00:06:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.327+0000 7fee02000640 1 -- 172.21.15.40:0/3720827789 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee0c004050 con 0x563d05f1f7d0 2024-04-27T00:06:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.328+0000 7fee02000640 1 -- 172.21.15.40:0/3720827789 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fedfc0d9030 con 0x563d05f1f7d0 2024-04-27T00:06:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.328+0000 7fee1dd93740 10 client.15648 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:06:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.328+0000 7fee1dd93740 10 client.15648 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:06:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.328+0000 7fee1dd93740 20 client.15648 populate_metadata read hostname 'smithi040' 2024-04-27T00:06:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.328+0000 7fee1dd93740 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563d05f34a50 0x563d05f36e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.328+0000 7fee1dd93740 4 client.15648 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:06:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.328+0000 7fee1dd93740 1 -- 172.21.15.40:0/3720827789 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d05e90f90 con 0x563d05f34a50 2024-04-27T00:06:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.328+0000 7fee13e00640 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563d05f34a50 0x563d05f36e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.329+0000 7fee13e00640 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563d05f34a50 0x563d05f36e80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6009d72b4bf4de2f server_cookie=83218fcc944b88aa in_seq=0 out_seq=0 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.329+0000 7fee02000640 10 client.15648 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee02000640 1 -- 172.21.15.40:0/3720827789 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563d05e90f90 con 0x563d05f34a50 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee02000640 10 client.15648 handle_command_reply: tid=0 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee1dd93740 1 client.15648 shutdown 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee1dd93740 1 -- 172.21.15.40:0/3720827789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f1f7d0 msgr2=0x563d05f29510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee1dd93740 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f1f7d0 0x563d05f29510 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fedfc098ea0 tx=0x7fedfc00ce60 comp rx=0 tx=0).stop 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee1dd93740 1 -- 172.21.15.40:0/3720827789 shutdown_connections 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee1dd93740 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563d05f34a50 0x563d05f36e80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee1dd93740 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f24940 0x563d05f30550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee1dd93740 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f1f7d0 0x563d05f29510 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee1dd93740 1 --2- 172.21.15.40:0/3720827789 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f1ede0 0x563d05f28fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee02000640 0 client.15648 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee1dd93740 1 -- 172.21.15.40:0/3720827789 >> 172.21.15.40:0/3720827789 conn(0x563d05f1a450 msgr2=0x563d05f19650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee1dd93740 1 -- 172.21.15.40:0/3720827789 shutdown_connections 2024-04-27T00:06:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee1dd93740 1 -- 172.21.15.40:0/3720827789 wait complete. 2024-04-27T00:06:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.330+0000 7fee1dd93740 20 client.15648 trim_cache size 0 max 16384 2024-04-27T00:06:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee1dd93740 1 Processor -- start 2024-04-27T00:06:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee1dd93740 1 -- start start 2024-04-27T00:06:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee1dd93740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 0x563d05f19dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee1dd93740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f1f7d0 0x563d05f30510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee1dd93740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f24940 0x563d05f3b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee1dd93740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee141afed0 con 0x563d05f24940 2024-04-27T00:06:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee1dd93740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fee141ad7a0 con 0x563d05f1f7d0 2024-04-27T00:06:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee1dd93740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee141ab5f0 con 0x563d05f1ede0 2024-04-27T00:06:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee0be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f1f7d0 0x563d05f30510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee18e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f24940 0x563d05f3b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee0be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f1f7d0 0x563d05f30510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:51192/0 (socket says 172.21.15.40:51192) 2024-04-27T00:06:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee0be00640 1 -- 172.21.15.40:0/2732274045 learned_addr learned my addr 172.21.15.40:0/2732274045 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.347+0000 7fee13e00640 1 --2- 172.21.15.40:0/2732274045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 0x563d05f19dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.348+0000 7fee13e00640 1 -- 172.21.15.40:0/2732274045 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f1f7d0 msgr2=0x563d05f30510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.348+0000 7fee13e00640 1 --2- 172.21.15.40:0/2732274045 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f1f7d0 0x563d05f30510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.348+0000 7fee13e00640 1 -- 172.21.15.40:0/2732274045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f24940 msgr2=0x563d05f3b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.348+0000 7fee13e00640 1 --2- 172.21.15.40:0/2732274045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f24940 0x563d05f3b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.348+0000 7fee13e00640 1 -- 172.21.15.40:0/2732274045 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d05c61620 con 0x563d05f1ede0 2024-04-27T00:06:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.348+0000 7fee13e00640 1 --2- 172.21.15.40:0/2732274045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 0x563d05f19dd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fee04000c00 tx=0x7fee04040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=305263a820d4129 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.348+0000 7fee08c00640 1 -- 172.21.15.40:0/2732274045 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04059070 con 0x563d05f1ede0 2024-04-27T00:06:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.348+0000 7fee08c00640 1 -- 172.21.15.40:0/2732274045 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee040035f0 con 0x563d05f1ede0 2024-04-27T00:06:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.348+0000 7fee08c00640 1 -- 172.21.15.40:0/2732274045 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04035040 con 0x563d05f1ede0 2024-04-27T00:06:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.349+0000 7fee1dd93740 1 -- 172.21.15.40:0/2732274045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 msgr2=0x563d05f19dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.349+0000 7fee1dd93740 1 --2- 172.21.15.40:0/2732274045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 0x563d05f19dd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fee04000c00 tx=0x7fee04040000 comp rx=0 tx=0).stop 2024-04-27T00:06:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.349+0000 7fee1dd93740 1 -- 172.21.15.40:0/2732274045 shutdown_connections 2024-04-27T00:06:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.349+0000 7fee1dd93740 1 --2- 172.21.15.40:0/2732274045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f24940 0x563d05f3b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.349+0000 7fee1dd93740 1 --2- 172.21.15.40:0/2732274045 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f1f7d0 0x563d05f30510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.349+0000 7fee1dd93740 1 --2- 172.21.15.40:0/2732274045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1ede0 0x563d05f19dd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.349+0000 7fee1dd93740 1 -- 172.21.15.40:0/2732274045 >> 172.21.15.40:0/2732274045 conn(0x563d05f1a450 msgr2=0x563d05f1da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.349+0000 7fee1dd93740 1 -- 172.21.15.40:0/2732274045 shutdown_connections 2024-04-27T00:06:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.349+0000 7fee1dd93740 1 -- 172.21.15.40:0/2732274045 wait complete. 2024-04-27T00:06:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.350+0000 7fee1dd93740 1 -- start start 2024-04-27T00:06:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.350+0000 7fee1dd93740 1 Processor -- start 2024-04-27T00:06:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.350+0000 7fee1dd93740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1f7d0 0x563d05f2fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.350+0000 7fee1dd93740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f24940 0x563d05f303c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.350+0000 7fee1dd93740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f2e4e0 0x563d05f2e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.350+0000 7fee1dd93740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee141abc60 con 0x563d05f24940 2024-04-27T00:06:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.350+0000 7fee1dd93740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fee141ad0d0 con 0x563d05f2e4e0 2024-04-27T00:06:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.350+0000 7fee1dd93740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee141ac330 con 0x563d05f1f7d0 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.350+0000 7fee1dd93740 10 client.0 _ll_register_callbacks cb 0x7fedfc0d43b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.350+0000 7fee13e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1f7d0 0x563d05f2fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee13e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1f7d0 0x563d05f2fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36222/0 (socket says 172.21.15.40:36222) 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee13e00640 1 -- 172.21.15.40:0/1114280320 learned_addr learned my addr 172.21.15.40:0/1114280320 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee0be00640 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f24940 0x563d05f303c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee18e00640 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f2e4e0 0x563d05f2e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee13e00640 1 -- 172.21.15.40:0/1114280320 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f2e4e0 msgr2=0x563d05f2e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee13e00640 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f2e4e0 0x563d05f2e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee13e00640 1 -- 172.21.15.40:0/1114280320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f24940 msgr2=0x563d05f303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee13e00640 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f24940 0x563d05f303c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee13e00640 1 -- 172.21.15.40:0/1114280320 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d05e3dcb0 con 0x563d05f1f7d0 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee18e00640 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f2e4e0 0x563d05f2e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee13e00640 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1f7d0 0x563d05f2fe80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fee04005ab0 tx=0x7fee04004550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3447deaae3b8975b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee02000640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.351+0000 7fee02000640 1 -- 172.21.15.40:0/1114280320 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04059070 con 0x563d05f1f7d0 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee02000640 1 -- 172.21.15.40:0/1114280320 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fee04053830 con 0x563d05f1f7d0 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee1dd93740 1 -- 172.21.15.40:0/1114280320 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee040035f0 con 0x563d05f1f7d0 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee02000640 1 -- 172.21.15.40:0/1114280320 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fee04067050 con 0x563d05f1f7d0 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee1dd93740 10 client.25798 fetch_fsmap learned FSMap version 24 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee1dd93740 1 -- 172.21.15.40:0/1114280320 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563d05f9bcf0 con 0x563d05f1f7d0 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee02000640 1 -- 172.21.15.40:0/1114280320 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee040035f0 con 0x563d05f1f7d0 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee02000640 1 -- 172.21.15.40:0/1114280320 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fee04050070 con 0x563d05f1f7d0 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee1dd93740 10 client.25798 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee1dd93740 10 client.25798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee1dd93740 20 client.25798 populate_metadata read hostname 'smithi040' 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee1dd93740 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563d05f9c5e0 0x563d05f9ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee1dd93740 4 client.25798 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:06:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.352+0000 7fee1dd93740 1 -- 172.21.15.40:0/1114280320 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563d05e8ae00 con 0x563d05f9c5e0 2024-04-27T00:06:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.353+0000 7fee0be00640 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563d05f9c5e0 0x563d05f9ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.353+0000 7fee0be00640 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563d05f9c5e0 0x563d05f9ea00 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52948559df2f029 server_cookie=64d7fe38ad881584 in_seq=0 out_seq=0 2024-04-27T00:06:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.353+0000 7fee02000640 10 client.25798 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.368+0000 7fee02000640 1 -- 172.21.15.40:0/1114280320 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563d05e8ae00 con 0x563d05f9c5e0 2024-04-27T00:06:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.368+0000 7fee02000640 10 client.25798 handle_command_reply: tid=0 2024-04-27T00:06:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.368+0000 7fee1dd93740 1 client.25798 shutdown 2024-04-27T00:06:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.369+0000 7fee1dd93740 1 -- 172.21.15.40:0/1114280320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1f7d0 msgr2=0x563d05f2fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.369+0000 7fee1dd93740 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1f7d0 0x563d05f2fe80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fee04005ab0 tx=0x7fee04004550 comp rx=0 tx=0).stop 2024-04-27T00:06:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.369+0000 7fee1dd93740 1 -- 172.21.15.40:0/1114280320 shutdown_connections 2024-04-27T00:06:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.369+0000 7fee1dd93740 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563d05f9c5e0 0x563d05f9ea00 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.369+0000 7fee1dd93740 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563d05f2e4e0 0x563d05f2e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.369+0000 7fee1dd93740 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563d05f24940 0x563d05f303c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.369+0000 7fee1dd93740 1 --2- 172.21.15.40:0/1114280320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d05f1f7d0 0x563d05f2fe80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.369+0000 7fee02000640 0 client.25798 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:39.370 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.369+0000 7fee1dd93740 1 -- 172.21.15.40:0/1114280320 >> 172.21.15.40:0/1114280320 conn(0x563d05f1a450 msgr2=0x563d05f18dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.369+0000 7fee1dd93740 1 -- 172.21.15.40:0/1114280320 shutdown_connections 2024-04-27T00:06:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.369+0000 7fee1dd93740 1 -- 172.21.15.40:0/1114280320 wait complete. 2024-04-27T00:06:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.369+0000 7fee1dd93740 20 client.25798 trim_cache size 0 max 16384 2024-04-27T00:06:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:06:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:06:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:06:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:06:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.370+0000 7fee19800640 1 -- 172.21.15.40:0/214534958 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fedf80a3cb0 msgr2=0x7fedf80a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.370+0000 7fee19800640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fedf80a3cb0 0x7fedf80a6170 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x563d05f1fbd0 tx=0x7fee040314a0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.370+0000 7fee19800640 1 -- 172.21.15.40:0/214534958 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee141028c0 msgr2=0x7fee1419b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.370+0000 7fee19800640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee141028c0 0x7fee1419b1b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00e9f0 tx=0x7fedfc00eeb0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.370+0000 7fee19800640 1 -- 172.21.15.40:0/214534958 shutdown_connections 2024-04-27T00:06:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.370+0000 7fee19800640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fedf80a3cb0 0x7fedf80a6170 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.370+0000 7fee19800640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee14103280 0x7fee141a2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.370+0000 7fee19800640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fee141028c0 0x7fee1419b1b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.370+0000 7fee19800640 1 --2- 172.21.15.40:0/214534958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee141016d0 0x7fee1419ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.370+0000 7fee19800640 1 -- 172.21.15.40:0/214534958 >> 172.21.15.40:0/214534958 conn(0x7fee140fcc60 msgr2=0x7fee1406aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.370+0000 7fee19800640 1 -- 172.21.15.40:0/214534958 shutdown_connections 2024-04-27T00:06:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:39.370+0000 7fee19800640 1 -- 172.21.15.40:0/214534958 wait complete. 2024-04-27T00:06:40.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:40 smithi040 ceph-mon[28095]: pgmap v561: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:06:40.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:40 smithi080 ceph-mon[32362]: pgmap v561: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:06:40.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:40 smithi103 ceph-mon[32952]: pgmap v561: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:06:42.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:42 smithi040 ceph-mon[28095]: pgmap v562: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:06:42.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:42 smithi080 ceph-mon[32362]: pgmap v562: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:06:42.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:42 smithi103 ceph-mon[32952]: pgmap v562: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:06:44.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:44 smithi040 ceph-mon[28095]: pgmap v563: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:06:44.551 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:44 smithi080 ceph-mon[32362]: pgmap v563: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:06:44.683 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:44 smithi103 ceph-mon[32952]: pgmap v563: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:06:46.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:46 smithi040 ceph-mon[28095]: pgmap v564: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:46.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:46 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:06:46.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:46 smithi080 ceph-mon[32362]: pgmap v564: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:46.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:46 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:06:46.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:46 smithi103 ceph-mon[32952]: pgmap v564: 129 pgs: 129 active+clean; 3.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:46.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:46 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:06:47.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:47 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:06:47.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:47 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:06:47.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:47 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:06:47.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:47 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:06:47.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:47 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:06:47.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:47 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:06:47.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:47 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:06:47.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:47 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:06:47.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:47 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:06:48.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:48 smithi040 ceph-mon[28095]: pgmap v565: 129 pgs: 129 active+clean; 3.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:48.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:48 smithi080 ceph-mon[32362]: pgmap v565: 129 pgs: 129 active+clean; 3.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:48.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:48 smithi103 ceph-mon[32952]: pgmap v565: 129 pgs: 129 active+clean; 3.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:06:50.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:50 smithi040 ceph-mon[28095]: pgmap v566: 129 pgs: 129 active+clean; 3.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:50.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:50 smithi080 ceph-mon[32362]: pgmap v566: 129 pgs: 129 active+clean; 3.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:50.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:50 smithi103 ceph-mon[32952]: pgmap v566: 129 pgs: 129 active+clean; 3.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:52.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:52 smithi040 ceph-mon[28095]: pgmap v567: 129 pgs: 129 active+clean; 3.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:52.519 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.518+0000 7f682ca00640 1 -- 172.21.15.40:0/383289256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f682810a6a0 msgr2=0x7f682810cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.518+0000 7f682ca00640 1 --2- 172.21.15.40:0/383289256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f682810a6a0 0x7f682810cb30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6820009ca0 tx=0x7f682002a950 comp rx=0 tx=0).stop 2024-04-27T00:06:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.518+0000 7f682ca00640 1 -- 172.21.15.40:0/383289256 shutdown_connections 2024-04-27T00:06:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.518+0000 7f682ca00640 1 --2- 172.21.15.40:0/383289256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f682810a6a0 0x7f682810cb30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.518+0000 7f682ca00640 1 --2- 172.21.15.40:0/383289256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6828075310 0x7f6828075770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.518+0000 7f682ca00640 1 --2- 172.21.15.40:0/383289256 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f68280768f0 0x7f6828074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.518+0000 7f682ca00640 1 -- 172.21.15.40:0/383289256 >> 172.21.15.40:0/383289256 conn(0x7f68280fdd60 msgr2=0x7f68281001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.518+0000 7f682ca00640 1 -- 172.21.15.40:0/383289256 shutdown_connections 2024-04-27T00:06:52.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.518+0000 7f682ca00640 1 -- 172.21.15.40:0/383289256 wait complete. 2024-04-27T00:06:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.519+0000 7f682ca00640 1 Processor -- start 2024-04-27T00:06:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.519+0000 7f682ca00640 1 -- start start 2024-04-27T00:06:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.520+0000 7f682ca00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6828075310 0x7f682819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.520+0000 7f682ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68280768f0 0x7f682819b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.520+0000 7f682ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682810a6a0 0x7f68281a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.520+0000 7f682ca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6828110580 con 0x7f68280768f0 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.520+0000 7f682ca00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f6828110400 con 0x7f6828075310 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.520+0000 7f682ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6828110700 con 0x7f682810a6a0 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.520+0000 7f6826a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68280768f0 0x7f682819b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.520+0000 7f6826a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68280768f0 0x7f682819b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48066/0 (socket says 172.21.15.40:48066) 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.520+0000 7f6826a00640 1 -- 172.21.15.40:0/3255839621 learned_addr learned my addr 172.21.15.40:0/3255839621 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.520+0000 7f6827400640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6828075310 0x7f682819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.520+0000 7f6827e00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682810a6a0 0x7f68281a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.521+0000 7f6826a00640 1 -- 172.21.15.40:0/3255839621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682810a6a0 msgr2=0x7f68281a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.521+0000 7f6826a00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682810a6a0 0x7f68281a2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.521+0000 7f6826a00640 1 -- 172.21.15.40:0/3255839621 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6828075310 msgr2=0x7f682819abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.521+0000 7f6826a00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6828075310 0x7f682819abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.521+0000 7f6826a00640 1 -- 172.21.15.40:0/3255839621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68281a27d0 con 0x7f68280768f0 2024-04-27T00:06:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.521+0000 7f6827400640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6828075310 0x7f682819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:06:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.521+0000 7f6826a00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68280768f0 0x7f682819b0f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f681000e9f0 tx=0x7f681000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.521+0000 7f681fe00640 1 -- 172.21.15.40:0/3255839621 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681000cd80 con 0x7f68280768f0 2024-04-27T00:06:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.521+0000 7f681fe00640 1 -- 172.21.15.40:0/3255839621 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68100054d0 con 0x7f68280768f0 2024-04-27T00:06:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.521+0000 7f681fe00640 1 -- 172.21.15.40:0/3255839621 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6810010690 con 0x7f68280768f0 2024-04-27T00:06:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.521+0000 7f682ca00640 1 -- 172.21.15.40:0/3255839621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68281a2ac0 con 0x7f68280768f0 2024-04-27T00:06:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.521+0000 7f682ca00640 1 -- 172.21.15.40:0/3255839621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68281a2fc0 con 0x7f68280768f0 2024-04-27T00:06:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.525+0000 7f6830df6740 1 Processor -- start 2024-04-27T00:06:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.525+0000 7f681fe00640 1 -- 172.21.15.40:0/3255839621 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6810004e90 con 0x7f68280768f0 2024-04-27T00:06:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.525+0000 7f6830df6740 1 -- start start 2024-04-27T00:06:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.525+0000 7f6830df6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82ed200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.525+0000 7f6830df6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ed7d0 0x559fe82f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.525+0000 7f6830df6740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82f2940 0x559fe82f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.525+0000 7f6830df6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559fe8251380 con 0x559fe82ecde0 2024-04-27T00:06:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.525+0000 7f6830df6740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x559fe81edc30 con 0x559fe82f2940 2024-04-27T00:06:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.525+0000 7f6830df6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559fe824d340 con 0x559fe82ed7d0 2024-04-27T00:06:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.525+0000 7f6827e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82ed200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.526+0000 7f6827e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82ed200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48082/0 (socket says 172.21.15.40:48082) 2024-04-27T00:06:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.526+0000 7f6826a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82f2940 0x559fe82f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.526+0000 7f6827400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ed7d0 0x559fe82f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.526+0000 7f6827e00640 1 -- 172.21.15.40:0/3851886054 learned_addr learned my addr 172.21.15.40:0/3851886054 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.526+0000 7f6827400640 1 -- 172.21.15.40:0/3851886054 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82f2940 msgr2=0x559fe82f4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.526+0000 7f6827400640 1 --2- 172.21.15.40:0/3851886054 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82f2940 0x559fe82f4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.526+0000 7f6827400640 1 -- 172.21.15.40:0/3851886054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 msgr2=0x559fe82ed200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.526+0000 7f6827400640 1 --2- 172.21.15.40:0/3851886054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82ed200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.526+0000 7f6827400640 1 -- 172.21.15.40:0/3851886054 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fe820f940 con 0x559fe82ed7d0 2024-04-27T00:06:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.526+0000 7f6827400640 1 --2- 172.21.15.40:0/3851886054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ed7d0 0x559fe82f22d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6818005fd0 tx=0x7f681800bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5180da18fedb306d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.527+0000 7f681fe00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f68080a3c30 0x7f68080a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.527+0000 7f6817e00640 1 -- 172.21.15.40:0/3851886054 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6818011700 con 0x559fe82ed7d0 2024-04-27T00:06:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.527+0000 7f6817e00640 1 -- 172.21.15.40:0/3851886054 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6818011d50 con 0x559fe82ed7d0 2024-04-27T00:06:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.527+0000 7f6817e00640 1 -- 172.21.15.40:0/3851886054 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681801a380 con 0x559fe82ed7d0 2024-04-27T00:06:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.527+0000 7f681fe00640 1 -- 172.21.15.40:0/3255839621 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f68100d0d20 con 0x7f68280768f0 2024-04-27T00:06:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.527+0000 7f6827e00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f68080a3c30 0x7f68080a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.528+0000 7f6830df6740 1 -- 172.21.15.40:0/3851886054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ed7d0 msgr2=0x559fe82f22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.528+0000 7f6830df6740 1 --2- 172.21.15.40:0/3851886054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ed7d0 0x559fe82f22d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6818005fd0 tx=0x7f681800bf90 comp rx=0 tx=0).stop 2024-04-27T00:06:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.528+0000 7f6830df6740 1 -- 172.21.15.40:0/3851886054 shutdown_connections 2024-04-27T00:06:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.528+0000 7f6830df6740 1 --2- 172.21.15.40:0/3851886054 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82f2940 0x559fe82f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.528+0000 7f6830df6740 1 --2- 172.21.15.40:0/3851886054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ed7d0 0x559fe82f22d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.528+0000 7f6830df6740 1 --2- 172.21.15.40:0/3851886054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82ed200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.528+0000 7f6830df6740 1 -- 172.21.15.40:0/3851886054 >> 172.21.15.40:0/3851886054 conn(0x559fe82e8450 msgr2=0x559fe82e8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.528+0000 7f6830df6740 1 -- 172.21.15.40:0/3851886054 shutdown_connections 2024-04-27T00:06:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.528+0000 7f6830df6740 1 -- 172.21.15.40:0/3851886054 wait complete. 2024-04-27T00:06:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.528+0000 7f6827e00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f68080a3c30 0x7f68080a60f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6820009a60 tx=0x7f682002f870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.528+0000 7f6830df6740 1 -- start start 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.529+0000 7f6830df6740 1 Processor -- start 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.529+0000 7f6830df6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ecde0 0x559fe82f7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.529+0000 7f6830df6740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82ed7d0 0x559fe82f75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.529+0000 7f6830df6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82f2940 0x559fe82fe600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.529+0000 7f6830df6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559fe82f58f0 con 0x559fe82f2940 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.529+0000 7f6830df6740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x559fe81d8c60 con 0x559fe82ed7d0 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.529+0000 7f6830df6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559fe82f5a70 con 0x559fe82ecde0 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.529+0000 7f6830df6740 10 client.0 _ll_register_callbacks cb 0x559fe8204e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6827400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ecde0 0x559fe82f7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6826a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82ed7d0 0x559fe82f75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6827e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82f2940 0x559fe82fe600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6827400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ecde0 0x559fe82f7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36286/0 (socket says 172.21.15.40:36286) 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6827400640 1 -- 172.21.15.40:0/3868777697 learned_addr learned my addr 172.21.15.40:0/3868777697 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6827400640 1 -- 172.21.15.40:0/3868777697 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82ed7d0 msgr2=0x559fe82f75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6827400640 1 --2- 172.21.15.40:0/3868777697 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82ed7d0 0x559fe82f75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6827400640 1 -- 172.21.15.40:0/3868777697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82f2940 msgr2=0x559fe82fe600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6827400640 1 --2- 172.21.15.40:0/3868777697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82f2940 0x559fe82fe600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6827400640 1 -- 172.21.15.40:0/3868777697 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fe81e7510 con 0x559fe82ecde0 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6826a00640 1 --2- 172.21.15.40:0/3868777697 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82ed7d0 0x559fe82f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6827400640 1 --2- 172.21.15.40:0/3868777697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ecde0 0x559fe82f7080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6818005d00 tx=0x7f681801c6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88fec332f789ddcf server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.530+0000 7f6814c00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.531+0000 7f6814c00640 1 -- 172.21.15.40:0/3868777697 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6818011910 con 0x559fe82ecde0 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.531+0000 7f6814c00640 1 -- 172.21.15.40:0/3868777697 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f6818011ab0 con 0x559fe82ecde0 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.531+0000 7f6830df6740 1 -- 172.21.15.40:0/3868777697 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6818011d50 con 0x559fe82ecde0 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.531+0000 7f6814c00640 1 -- 172.21.15.40:0/3868777697 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6818033050 con 0x559fe82ecde0 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.531+0000 7f6830df6740 10 client.25810 fetch_fsmap learned FSMap version 24 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.531+0000 7f6830df6740 1 -- 172.21.15.40:0/3868777697 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559fe7f63210 con 0x559fe82ecde0 2024-04-27T00:06:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.531+0000 7f6814c00640 1 -- 172.21.15.40:0/3868777697 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6818011d50 con 0x559fe82ecde0 2024-04-27T00:06:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.532+0000 7f6814c00640 1 -- 172.21.15.40:0/3868777697 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f681802e3b0 con 0x559fe82ecde0 2024-04-27T00:06:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.532+0000 7f6830df6740 10 client.25810 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:06:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.532+0000 7f6830df6740 10 client.25810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:06:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.532+0000 7f6830df6740 20 client.25810 populate_metadata read hostname 'smithi040' 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.532+0000 7f6830df6740 1 --2- 172.21.15.40:0/3868777697 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559fe8302810 0x559fe8304c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.532+0000 7f6830df6740 4 client.25810 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.532+0000 7f6830df6740 1 -- 172.21.15.40:0/3868777697 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559fe825ef90 con 0x559fe8302810 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.532+0000 7f6826a00640 1 --2- 172.21.15.40:0/3868777697 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559fe8302810 0x559fe8304c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.533+0000 7f6826a00640 1 --2- 172.21.15.40:0/3868777697 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559fe8302810 0x559fe8304c00 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdd9836acf232192 server_cookie=8245d7a6658c9927 in_seq=0 out_seq=0 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.533+0000 7f6814c00640 10 client.25810 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.533+0000 7f6814c00640 1 -- 172.21.15.40:0/3868777697 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559fe825ef90 con 0x559fe8302810 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.533+0000 7f6814c00640 10 client.25810 handle_command_reply: tid=0 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.533+0000 7f6830df6740 1 client.25810 shutdown 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.534+0000 7f6830df6740 1 -- 172.21.15.40:0/3868777697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ecde0 msgr2=0x559fe82f7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.534+0000 7f6830df6740 1 --2- 172.21.15.40:0/3868777697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ecde0 0x559fe82f7080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6818005d00 tx=0x7f681801c6b0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.534+0000 7f6830df6740 1 -- 172.21.15.40:0/3868777697 shutdown_connections 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.534+0000 7f6830df6740 1 --2- 172.21.15.40:0/3868777697 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559fe8302810 0x559fe8304c00 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.534+0000 7f6830df6740 1 --2- 172.21.15.40:0/3868777697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82f2940 0x559fe82fe600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.534+0000 7f6830df6740 1 --2- 172.21.15.40:0/3868777697 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82ed7d0 0x559fe82f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.534+0000 7f6830df6740 1 --2- 172.21.15.40:0/3868777697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ecde0 0x559fe82f7080 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.534+0000 7f6830df6740 1 -- 172.21.15.40:0/3868777697 >> 172.21.15.40:0/3868777697 conn(0x559fe82e8450 msgr2=0x559fe82e7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.534+0000 7f6814c00640 0 client.25810 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.534+0000 7f6830df6740 1 -- 172.21.15.40:0/3868777697 shutdown_connections 2024-04-27T00:06:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.534+0000 7f6830df6740 1 -- 172.21.15.40:0/3868777697 wait complete. 2024-04-27T00:06:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.534+0000 7f6830df6740 20 client.25810 trim_cache size 0 max 16384 2024-04-27T00:06:52.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:52 smithi080 ceph-mon[32362]: pgmap v567: 129 pgs: 129 active+clean; 3.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.550+0000 7f6830df6740 1 Processor -- start 2024-04-27T00:06:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.551+0000 7f6830df6740 1 -- start start 2024-04-27T00:06:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.551+0000 7f6830df6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82e74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.551+0000 7f6830df6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ed7d0 0x559fe82e7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.551+0000 7f6830df6740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82f2940 0x559fe82fe5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.551+0000 7f6830df6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f68281afe20 con 0x559fe82ecde0 2024-04-27T00:06:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.551+0000 7f6830df6740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f68281ad6f0 con 0x559fe82f2940 2024-04-27T00:06:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.551+0000 7f6830df6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f68281ab540 con 0x559fe82ed7d0 2024-04-27T00:06:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.551+0000 7f6827400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82e74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.551+0000 7f6827400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82e74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48108/0 (socket says 172.21.15.40:48108) 2024-04-27T00:06:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.551+0000 7f6827400640 1 -- 172.21.15.40:0/1522591039 learned_addr learned my addr 172.21.15.40:0/1522591039 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.551+0000 7f6827e00640 1 --2- 172.21.15.40:0/1522591039 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82f2940 0x559fe82fe5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.551+0000 7f6826a00640 1 --2- 172.21.15.40:0/1522591039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ed7d0 0x559fe82e7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.552+0000 7f6827400640 1 -- 172.21.15.40:0/1522591039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ed7d0 msgr2=0x559fe82e7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.552+0000 7f6827400640 1 --2- 172.21.15.40:0/1522591039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ed7d0 0x559fe82e7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.552+0000 7f6827400640 1 -- 172.21.15.40:0/1522591039 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82f2940 msgr2=0x559fe82fe5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.552+0000 7f6827400640 1 --2- 172.21.15.40:0/1522591039 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82f2940 0x559fe82fe5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.552+0000 7f6827400640 1 -- 172.21.15.40:0/1522591039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fe802f620 con 0x559fe82ecde0 2024-04-27T00:06:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.552+0000 7f6827400640 1 --2- 172.21.15.40:0/1522591039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82e74d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f681801f040 tx=0x7f681800b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608afc5e0752b872 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.552+0000 7f6817400640 1 -- 172.21.15.40:0/1522591039 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681800bc50 con 0x559fe82ecde0 2024-04-27T00:06:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.552+0000 7f6817400640 1 -- 172.21.15.40:0/1522591039 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f681802a070 con 0x559fe82ecde0 2024-04-27T00:06:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.552+0000 7f6817400640 1 -- 172.21.15.40:0/1522591039 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6818011710 con 0x559fe82ecde0 2024-04-27T00:06:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.553+0000 7f6830df6740 1 -- 172.21.15.40:0/1522591039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 msgr2=0x559fe82e74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.553+0000 7f6830df6740 1 --2- 172.21.15.40:0/1522591039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82e74d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f681801f040 tx=0x7f681800b540 comp rx=0 tx=0).stop 2024-04-27T00:06:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.553+0000 7f6830df6740 1 -- 172.21.15.40:0/1522591039 shutdown_connections 2024-04-27T00:06:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.553+0000 7f6830df6740 1 --2- 172.21.15.40:0/1522591039 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82f2940 0x559fe82fe5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.553+0000 7f6830df6740 1 --2- 172.21.15.40:0/1522591039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82ed7d0 0x559fe82e7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.553+0000 7f6830df6740 1 --2- 172.21.15.40:0/1522591039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82e74d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.553+0000 7f6830df6740 1 -- 172.21.15.40:0/1522591039 >> 172.21.15.40:0/1522591039 conn(0x559fe82e8450 msgr2=0x559fe82e6dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.553+0000 7f6830df6740 1 -- 172.21.15.40:0/1522591039 shutdown_connections 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.553+0000 7f6830df6740 1 -- 172.21.15.40:0/1522591039 wait complete. 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.553+0000 7f6830df6740 1 -- start start 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6830df6740 1 Processor -- start 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6830df6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82fb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6830df6740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82ed7d0 0x559fe82fbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6830df6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82f2940 0x559fe8366c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6830df6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f68281abbb0 con 0x559fe82ecde0 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6830df6740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f68281ad020 con 0x559fe82ed7d0 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6830df6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f68281ac280 con 0x559fe82f2940 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6830df6740 10 client.0 _ll_register_callbacks cb 0x7f6820009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6827400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82fb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6826a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82ed7d0 0x559fe82fbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6827400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82fb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48110/0 (socket says 172.21.15.40:48110) 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6827400640 1 -- 172.21.15.40:0/656331452 learned_addr learned my addr 172.21.15.40:0/656331452 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.554+0000 7f6827e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82f2940 0x559fe8366c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.555+0000 7f6827400640 1 -- 172.21.15.40:0/656331452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82f2940 msgr2=0x559fe8366c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.555+0000 7f6827400640 1 --2- 172.21.15.40:0/656331452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82f2940 0x559fe8366c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.555+0000 7f6827400640 1 -- 172.21.15.40:0/656331452 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82ed7d0 msgr2=0x559fe82fbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.555+0000 7f6827400640 1 --2- 172.21.15.40:0/656331452 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82ed7d0 0x559fe82fbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.555+0000 7f6827400640 1 -- 172.21.15.40:0/656331452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fe820bcb0 con 0x559fe82ecde0 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.555+0000 7f6827400640 1 --2- 172.21.15.40:0/656331452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82fb590 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f681802e7b0 tx=0x7f681802ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e5eb1fb1c7ae57 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.555+0000 7f6814c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.555+0000 7f6814c00640 1 -- 172.21.15.40:0/656331452 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681802e5b0 con 0x559fe82ecde0 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.555+0000 7f6814c00640 1 -- 172.21.15.40:0/656331452 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f6818004020 con 0x559fe82ecde0 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.555+0000 7f6830df6740 1 -- 172.21.15.40:0/656331452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f681802a070 con 0x559fe82ecde0 2024-04-27T00:06:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.556+0000 7f6814c00640 1 -- 172.21.15.40:0/656331452 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f681804f050 con 0x559fe82ecde0 2024-04-27T00:06:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.556+0000 7f6830df6740 10 client.15672 fetch_fsmap learned FSMap version 24 2024-04-27T00:06:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.556+0000 7f6830df6740 1 -- 172.21.15.40:0/656331452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559fe8368300 con 0x559fe82ecde0 2024-04-27T00:06:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.556+0000 7f6814c00640 1 -- 172.21.15.40:0/656331452 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f681802a070 con 0x559fe82ecde0 2024-04-27T00:06:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.556+0000 7f6814c00640 1 -- 172.21.15.40:0/656331452 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6818047020 con 0x559fe82ecde0 2024-04-27T00:06:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.556+0000 7f6830df6740 10 client.15672 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:06:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.556+0000 7f6830df6740 10 client.15672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:06:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.556+0000 7f6830df6740 20 client.15672 populate_metadata read hostname 'smithi040' 2024-04-27T00:06:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.556+0000 7f6830df6740 1 --2- 172.21.15.40:0/656331452 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559fe8368be0 0x559fe836b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:06:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.556+0000 7f6830df6740 4 client.15672 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:06:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.556+0000 7f6830df6740 1 -- 172.21.15.40:0/656331452 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559fe8258e00 con 0x559fe8368be0 2024-04-27T00:06:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.556+0000 7f6826a00640 1 --2- 172.21.15.40:0/656331452 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559fe8368be0 0x559fe836b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:06:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.557+0000 7f6826a00640 1 --2- 172.21.15.40:0/656331452 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559fe8368be0 0x559fe836b000 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be34318bb073c6c3 server_cookie=f6b3822aab83709f in_seq=0 out_seq=0 2024-04-27T00:06:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.557+0000 7f6814c00640 10 client.15672 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.572+0000 7f6814c00640 1 -- 172.21.15.40:0/656331452 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559fe8258e00 con 0x559fe8368be0 2024-04-27T00:06:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.572+0000 7f6814c00640 10 client.15672 handle_command_reply: tid=0 2024-04-27T00:06:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.572+0000 7f6830df6740 1 client.15672 shutdown 2024-04-27T00:06:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.573+0000 7f6830df6740 1 -- 172.21.15.40:0/656331452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 msgr2=0x559fe82fb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.573+0000 7f6830df6740 1 --2- 172.21.15.40:0/656331452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82fb590 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f681802e7b0 tx=0x7f681802ea50 comp rx=0 tx=0).stop 2024-04-27T00:06:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.573+0000 7f6830df6740 1 -- 172.21.15.40:0/656331452 shutdown_connections 2024-04-27T00:06:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.573+0000 7f6830df6740 1 --2- 172.21.15.40:0/656331452 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559fe8368be0 0x559fe836b000 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.573+0000 7f6830df6740 1 --2- 172.21.15.40:0/656331452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559fe82f2940 0x559fe8366c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.573+0000 7f6830df6740 1 --2- 172.21.15.40:0/656331452 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559fe82ed7d0 0x559fe82fbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.573+0000 7f6830df6740 1 --2- 172.21.15.40:0/656331452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559fe82ecde0 0x559fe82fb590 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.573+0000 7f6814c00640 0 client.15672 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:06:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.573+0000 7f6830df6740 1 -- 172.21.15.40:0/656331452 >> 172.21.15.40:0/656331452 conn(0x559fe82e8450 msgr2=0x559fe82e7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.573+0000 7f6830df6740 1 -- 172.21.15.40:0/656331452 shutdown_connections 2024-04-27T00:06:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.573+0000 7f6830df6740 1 -- 172.21.15.40:0/656331452 wait complete. 2024-04-27T00:06:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.573+0000 7f6830df6740 20 client.15672 trim_cache size 0 max 16384 2024-04-27T00:06:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:06:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:06:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:06:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:06:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.574+0000 7f682ca00640 1 -- 172.21.15.40:0/3255839621 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f68080a3c30 msgr2=0x7f68080a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.574+0000 7f682ca00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f68080a3c30 0x7f68080a60f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6820009a60 tx=0x7f682002f870 comp rx=0 tx=0).stop 2024-04-27T00:06:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.574+0000 7f682ca00640 1 -- 172.21.15.40:0/3255839621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68280768f0 msgr2=0x7f682819b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:06:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.574+0000 7f682ca00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68280768f0 0x7f682819b0f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f681000e9f0 tx=0x7f681000edf0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.574+0000 7f682ca00640 1 -- 172.21.15.40:0/3255839621 shutdown_connections 2024-04-27T00:06:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.574+0000 7f682ca00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f68080a3c30 0x7f68080a60f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.574+0000 7f682ca00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f682810a6a0 0x7f68281a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.574+0000 7f682ca00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68280768f0 0x7f682819b0f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.574+0000 7f682ca00640 1 --2- 172.21.15.40:0/3255839621 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6828075310 0x7f682819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:06:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.574+0000 7f682ca00640 1 -- 172.21.15.40:0/3255839621 >> 172.21.15.40:0/3255839621 conn(0x7f68280fdd60 msgr2=0x7f6828106630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:06:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.574+0000 7f682ca00640 1 -- 172.21.15.40:0/3255839621 shutdown_connections 2024-04-27T00:06:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:06:52.574+0000 7f682ca00640 1 -- 172.21.15.40:0/3255839621 wait complete. 2024-04-27T00:06:52.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:52 smithi103 ceph-mon[32952]: pgmap v567: 129 pgs: 129 active+clean; 3.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:06:54.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:54 smithi040 ceph-mon[28095]: pgmap v568: 129 pgs: 129 active+clean; 3.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-04-27T00:06:54.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:54 smithi080 ceph-mon[32362]: pgmap v568: 129 pgs: 129 active+clean; 3.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-04-27T00:06:54.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:54 smithi103 ceph-mon[32952]: pgmap v568: 129 pgs: 129 active+clean; 3.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 15 op/s 2024-04-27T00:06:56.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:56 smithi040 ceph-mon[28095]: pgmap v569: 129 pgs: 129 active+clean; 3.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:06:56.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:56 smithi080 ceph-mon[32362]: pgmap v569: 129 pgs: 129 active+clean; 3.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:06:56.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:56 smithi103 ceph-mon[32952]: pgmap v569: 129 pgs: 129 active+clean; 3.2 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:06:58.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:06:58 smithi040 ceph-mon[28095]: pgmap v570: 129 pgs: 129 active+clean; 3.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:06:58.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:06:58 smithi080 ceph-mon[32362]: pgmap v570: 129 pgs: 129 active+clean; 3.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:06:58.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:06:58 smithi103 ceph-mon[32952]: pgmap v570: 129 pgs: 129 active+clean; 3.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:07:00.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:00 smithi040 ceph-mon[28095]: pgmap v571: 129 pgs: 129 active+clean; 3.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:07:00.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:00 smithi080 ceph-mon[32362]: pgmap v571: 129 pgs: 129 active+clean; 3.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:07:00.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:00 smithi103 ceph-mon[32952]: pgmap v571: 129 pgs: 129 active+clean; 3.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:07:02.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:02 smithi040 ceph-mon[28095]: pgmap v572: 129 pgs: 129 active+clean; 3.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:07:02.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:02 smithi080 ceph-mon[32362]: pgmap v572: 129 pgs: 129 active+clean; 3.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:07:02.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:02 smithi103 ceph-mon[32952]: pgmap v572: 129 pgs: 129 active+clean; 3.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:07:04.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:04 smithi040 ceph-mon[28095]: pgmap v573: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:07:04.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:04 smithi080 ceph-mon[32362]: pgmap v573: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:07:04.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:04 smithi103 ceph-mon[32952]: pgmap v573: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:07:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.722+0000 7f29c1a00640 1 -- 172.21.15.40:0/965789055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29bc1049f0 msgr2=0x7f29bc104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.722+0000 7f29c1a00640 1 --2- 172.21.15.40:0/965789055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29bc1049f0 0x7f29bc104e10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f29ac009900 tx=0x7f29ac02a3f0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.722+0000 7f29c1a00640 1 -- 172.21.15.40:0/965789055 shutdown_connections 2024-04-27T00:07:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.722+0000 7f29c1a00640 1 --2- 172.21.15.40:0/965789055 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f29bc1065a0 0x7f29bc10ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.722+0000 7f29c1a00640 1 --2- 172.21.15.40:0/965789055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29bc105be0 0x7f29bc106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.722+0000 7f29c1a00640 1 --2- 172.21.15.40:0/965789055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29bc1049f0 0x7f29bc104e10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.722+0000 7f29c1a00640 1 -- 172.21.15.40:0/965789055 >> 172.21.15.40:0/965789055 conn(0x7f29bc0fffa0 msgr2=0x7f29bc1023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.722+0000 7f29c1a00640 1 -- 172.21.15.40:0/965789055 shutdown_connections 2024-04-27T00:07:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.722+0000 7f29c1a00640 1 -- 172.21.15.40:0/965789055 wait complete. 2024-04-27T00:07:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.723+0000 7f29c1a00640 1 Processor -- start 2024-04-27T00:07:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.723+0000 7f29c1a00640 1 -- start start 2024-04-27T00:07:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.724+0000 7f29c1a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f29bc1049f0 0x7f29bc19ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.724+0000 7f29c1a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29bc105be0 0x7f29bc19b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.724+0000 7f29c1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29bc1065a0 0x7f29bc1a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.724+0000 7f29c1a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc110540 con 0x7f29bc1065a0 2024-04-27T00:07:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.724+0000 7f29c1a00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1103c0 con 0x7f29bc1049f0 2024-04-27T00:07:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.724+0000 7f29c1a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1106c0 con 0x7f29bc105be0 2024-04-27T00:07:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.724+0000 7f29bbe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f29bc1049f0 0x7f29bc19ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.724+0000 7f29bbe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f29bc1049f0 0x7f29bc19ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:56604/0 (socket says 172.21.15.40:56604) 2024-04-27T00:07:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.724+0000 7f29b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29bc105be0 0x7f29bc19b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.724+0000 7f29bbe00640 1 -- 172.21.15.40:0/1592642236 learned_addr learned my addr 172.21.15.40:0/1592642236 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.724+0000 7f29c1000640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29bc1065a0 0x7f29bc1a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.730 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.725+0000 7f29b3e00640 1 -- 172.21.15.40:0/1592642236 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f29bc1049f0 msgr2=0x7f29bc19ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:05.730 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.725+0000 7f29b3e00640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f29bc1049f0 0x7f29bc19ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.730 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.725+0000 7f29b3e00640 1 -- 172.21.15.40:0/1592642236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29bc1065a0 msgr2=0x7f29bc1a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:05.730 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.725+0000 7f29b3e00640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29bc1065a0 0x7f29bc1a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.730 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.725+0000 7f29b3e00640 1 -- 172.21.15.40:0/1592642236 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29bc1a2840 con 0x7f29bc105be0 2024-04-27T00:07:05.730 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.725+0000 7f29bbe00640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f29bc1049f0 0x7f29bc19ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:07:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.725+0000 7f29b3e00640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29bc105be0 0x7f29bc19b170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f29a400e9f0 tx=0x7f29a400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.725+0000 7f29b9600640 1 -- 172.21.15.40:0/1592642236 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a400ccc0 con 0x7f29bc105be0 2024-04-27T00:07:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.726+0000 7f29b9600640 1 -- 172.21.15.40:0/1592642236 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29a40108e0 con 0x7f29bc105be0 2024-04-27T00:07:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.726+0000 7f29c1a00640 1 -- 172.21.15.40:0/1592642236 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29bc1a2b30 con 0x7f29bc105be0 2024-04-27T00:07:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.726+0000 7f29b9600640 1 -- 172.21.15.40:0/1592642236 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a4018840 con 0x7f29bc105be0 2024-04-27T00:07:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.726+0000 7f29c1a00640 1 -- 172.21.15.40:0/1592642236 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29bc1a3080 con 0x7f29bc105be0 2024-04-27T00:07:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.729+0000 7f29c5da3740 1 Processor -- start 2024-04-27T00:07:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.729+0000 7f29b9600640 1 -- 172.21.15.40:0/1592642236 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f29a4010430 con 0x7f29bc105be0 2024-04-27T00:07:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.729+0000 7f29c5da3740 1 -- start start 2024-04-27T00:07:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29c5da3740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb1de0 0x556987eb2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29c5da3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb27d0 0x556987eb72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29c5da3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987eb9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29c5da3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556987e16380 con 0x556987eb7940 2024-04-27T00:07:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29c5da3740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x556987db2c30 con 0x556987eb1de0 2024-04-27T00:07:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29c5da3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556987e12340 con 0x556987eb27d0 2024-04-27T00:07:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987eb9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29bbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb27d0 0x556987eb72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29bbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb27d0 0x556987eb72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38834/0 (socket says 172.21.15.40:38834) 2024-04-27T00:07:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29bbe00640 1 -- 172.21.15.40:0/855455503 learned_addr learned my addr 172.21.15.40:0/855455503 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29c1000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb1de0 0x556987eb2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29c1000640 1 -- 172.21.15.40:0/855455503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb27d0 msgr2=0x556987eb72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29c1000640 1 --2- 172.21.15.40:0/855455503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb27d0 0x556987eb72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29c1000640 1 -- 172.21.15.40:0/855455503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 msgr2=0x556987eb9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.730+0000 7f29c1000640 1 --2- 172.21.15.40:0/855455503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987eb9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.731+0000 7f29c1000640 1 -- 172.21.15.40:0/855455503 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556987dd4940 con 0x556987eb1de0 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.731+0000 7f29c1000640 1 --2- 172.21.15.40:0/855455503 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb1de0 0x556987eb2200 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009f10 tx=0x7f29b4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=483338ff14fa475e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.731+0000 7f29b9600640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29a00a3c30 0x7f29a00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.731+0000 7f29b1600640 1 -- 172.21.15.40:0/855455503 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b4003c30 con 0x556987eb1de0 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.731+0000 7f29b1600640 1 -- 172.21.15.40:0/855455503 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29b4003dd0 con 0x556987eb1de0 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.731+0000 7f29b9600640 1 -- 172.21.15.40:0/1592642236 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f29a40d0c60 con 0x7f29bc105be0 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.731+0000 7f29b1600640 1 -- 172.21.15.40:0/855455503 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b400f9a0 con 0x556987eb1de0 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.731+0000 7f29bbe00640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29a00a3c30 0x7f29a00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.732+0000 7f29c5da3740 1 -- 172.21.15.40:0/855455503 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb1de0 msgr2=0x556987eb2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.732+0000 7f29c5da3740 1 --2- 172.21.15.40:0/855455503 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb1de0 0x556987eb2200 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009f10 tx=0x7f29b4011810 comp rx=0 tx=0).stop 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.732+0000 7f29c5da3740 1 -- 172.21.15.40:0/855455503 shutdown_connections 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.732+0000 7f29c5da3740 1 --2- 172.21.15.40:0/855455503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987eb9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.732+0000 7f29bbe00640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29a00a3c30 0x7f29a00a60f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0096c0 tx=0x7f29ac005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.732+0000 7f29c5da3740 1 --2- 172.21.15.40:0/855455503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb27d0 0x556987eb72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.732+0000 7f29c5da3740 1 --2- 172.21.15.40:0/855455503 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb1de0 0x556987eb2200 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.732+0000 7f29c5da3740 1 -- 172.21.15.40:0/855455503 >> 172.21.15.40:0/855455503 conn(0x556987ead450 msgr2=0x556987ead850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.732+0000 7f29c5da3740 1 -- 172.21.15.40:0/855455503 shutdown_connections 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.732+0000 7f29c5da3740 1 -- 172.21.15.40:0/855455503 wait complete. 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.733+0000 7f29c5da3740 1 -- start start 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.733+0000 7f29c5da3740 1 Processor -- start 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.733+0000 7f29c5da3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 0x556987ebc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.733+0000 7f29c5da3740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987ebc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.733+0000 7f29c5da3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.733+0000 7f29c5da3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556987eba8f0 con 0x556987eb7940 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.733+0000 7f29c5da3740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x556987d9dc60 con 0x556987eb27d0 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.733+0000 7f29c5da3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556987ebaa70 con 0x556987eb1de0 2024-04-27T00:07:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.733+0000 7f29c5da3740 10 client.0 _ll_register_callbacks cb 0x556987dc9e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29b3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987ebc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29c1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29b3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987ebc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:56626/0 (socket says 172.21.15.40:56626) 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29b3e00640 1 -- 172.21.15.40:0/932431570 learned_addr learned my addr 172.21.15.40:0/932431570 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29bbe00640 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 0x556987ebc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29b3e00640 1 -- 172.21.15.40:0/932431570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 msgr2=0x556987ebc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29b3e00640 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 0x556987ebc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29b3e00640 1 -- 172.21.15.40:0/932431570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 msgr2=0x556987ec35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29b3e00640 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29b3e00640 1 -- 172.21.15.40:0/932431570 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556987dac510 con 0x556987eb27d0 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29c1000640 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29b3e00640 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987ebc5d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f29a4099320 tx=0x7f29a409a9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db17b3e7f5ed2a49 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.734+0000 7f29aa000640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:07:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.735+0000 7f29aa000640 1 -- 172.21.15.40:0/932431570 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a409c340 con 0x556987eb27d0 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.735+0000 7f29aa000640 1 -- 172.21.15.40:0/932431570 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f29a40bbba0 con 0x556987eb27d0 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.735+0000 7f29c5da3740 1 -- 172.21.15.40:0/932431570 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29b4003dd0 con 0x556987eb27d0 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.735+0000 7f29aa000640 1 -- 172.21.15.40:0/932431570 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f29a409ab20 con 0x556987eb27d0 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.735+0000 7f29c5da3740 10 client.25799 fetch_fsmap learned FSMap version 24 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.735+0000 7f29c5da3740 1 -- 172.21.15.40:0/932431570 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556987b28210 con 0x556987eb27d0 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.735+0000 7f29aa000640 1 -- 172.21.15.40:0/932431570 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29b4003dd0 con 0x556987eb27d0 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.735+0000 7f29aa000640 1 -- 172.21.15.40:0/932431570 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f29a40870b0 con 0x556987eb27d0 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.735+0000 7f29c5da3740 10 client.25799 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.735+0000 7f29c5da3740 10 client.25799 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.735+0000 7f29c5da3740 20 client.25799 populate_metadata read hostname 'smithi040' 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.735+0000 7f29c5da3740 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556987ec7ae0 0x556987ec9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.736+0000 7f29c5da3740 4 client.25799 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.736+0000 7f29c5da3740 1 -- 172.21.15.40:0/932431570 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556987e23f90 con 0x556987ec7ae0 2024-04-27T00:07:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.736+0000 7f29bbe00640 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556987ec7ae0 0x556987ec9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.736+0000 7f29bbe00640 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556987ec7ae0 0x556987ec9ef0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ab64b1ec123537c server_cookie=f4b614826924825f in_seq=0 out_seq=0 2024-04-27T00:07:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.736+0000 7f29aa000640 10 client.25799 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.737+0000 7f29aa000640 1 -- 172.21.15.40:0/932431570 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556987e23f90 con 0x556987ec7ae0 2024-04-27T00:07:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.737+0000 7f29aa000640 10 client.25799 handle_command_reply: tid=0 2024-04-27T00:07:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.737+0000 7f29c5da3740 1 client.25799 shutdown 2024-04-27T00:07:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.737+0000 7f29c5da3740 1 -- 172.21.15.40:0/932431570 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 msgr2=0x556987ebc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.737+0000 7f29c5da3740 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987ebc5d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f29a4099320 tx=0x7f29a409a9f0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.738+0000 7f29c5da3740 1 -- 172.21.15.40:0/932431570 shutdown_connections 2024-04-27T00:07:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.738+0000 7f29c5da3740 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556987ec7ae0 0x556987ec9ef0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.743 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.738+0000 7f29c5da3740 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.738+0000 7f29c5da3740 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987ebc5d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.738+0000 7f29c5da3740 1 --2- 172.21.15.40:0/932431570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 0x556987ebc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.738+0000 7f29c5da3740 1 -- 172.21.15.40:0/932431570 >> 172.21.15.40:0/932431570 conn(0x556987ead450 msgr2=0x556987eac650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.738+0000 7f29aa000640 0 client.25799 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.738+0000 7f29c5da3740 1 -- 172.21.15.40:0/932431570 shutdown_connections 2024-04-27T00:07:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.738+0000 7f29c5da3740 1 -- 172.21.15.40:0/932431570 wait complete. 2024-04-27T00:07:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.738+0000 7f29c5da3740 20 client.25799 trim_cache size 0 max 16384 2024-04-27T00:07:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.754+0000 7f29c5da3740 1 Processor -- start 2024-04-27T00:07:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29c5da3740 1 -- start start 2024-04-27T00:07:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29c5da3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 0x556987eb08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29c5da3740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987eb0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29c5da3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29c5da3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1afcc0 con 0x556987eb7940 2024-04-27T00:07:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29c5da3740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1ad590 con 0x556987eb27d0 2024-04-27T00:07:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29c5da3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1ab3e0 con 0x556987eb1de0 2024-04-27T00:07:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29b3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987eb0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29bbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 0x556987eb08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29b3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987eb0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:56634/0 (socket says 172.21.15.40:56634) 2024-04-27T00:07:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29b3e00640 1 -- 172.21.15.40:0/1209059357 learned_addr learned my addr 172.21.15.40:0/1209059357 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29c1000640 1 --2- 172.21.15.40:0/1209059357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29b3e00640 1 -- 172.21.15.40:0/1209059357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 msgr2=0x556987eb08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29b3e00640 1 --2- 172.21.15.40:0/1209059357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 0x556987eb08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29b3e00640 1 -- 172.21.15.40:0/1209059357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 msgr2=0x556987ec3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.755+0000 7f29b3e00640 1 --2- 172.21.15.40:0/1209059357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.756+0000 7f29b3e00640 1 -- 172.21.15.40:0/1209059357 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556987bf4620 con 0x556987eb27d0 2024-04-27T00:07:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.756+0000 7f29b3e00640 1 --2- 172.21.15.40:0/1209059357 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987eb0e20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f29a400c6a0 tx=0x7f29a40ce2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c94d087e9bc4b78 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.756+0000 7f29b0c00640 1 -- 172.21.15.40:0/1209059357 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a409c340 con 0x556987eb27d0 2024-04-27T00:07:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.756+0000 7f29b0c00640 1 -- 172.21.15.40:0/1209059357 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29a4010720 con 0x556987eb27d0 2024-04-27T00:07:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.756+0000 7f29b0c00640 1 -- 172.21.15.40:0/1209059357 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a409c9e0 con 0x556987eb27d0 2024-04-27T00:07:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.757+0000 7f29c5da3740 1 -- 172.21.15.40:0/1209059357 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 msgr2=0x556987eb0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.757+0000 7f29c5da3740 1 --2- 172.21.15.40:0/1209059357 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987eb0e20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f29a400c6a0 tx=0x7f29a40ce2c0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.757+0000 7f29c5da3740 1 -- 172.21.15.40:0/1209059357 shutdown_connections 2024-04-27T00:07:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.757+0000 7f29c5da3740 1 --2- 172.21.15.40:0/1209059357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.757+0000 7f29c5da3740 1 --2- 172.21.15.40:0/1209059357 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987eb0e20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.757+0000 7f29c5da3740 1 --2- 172.21.15.40:0/1209059357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 0x556987eb08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.757+0000 7f29c5da3740 1 -- 172.21.15.40:0/1209059357 >> 172.21.15.40:0/1209059357 conn(0x556987ead450 msgr2=0x556987eb01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.757+0000 7f29c5da3740 1 -- 172.21.15.40:0/1209059357 shutdown_connections 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.757+0000 7f29c5da3740 1 -- 172.21.15.40:0/1209059357 wait complete. 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.757+0000 7f29c5da3740 1 -- start start 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.757+0000 7f29c5da3740 1 Processor -- start 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29c5da3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 0x556987ec0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29c5da3740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987ec11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29c5da3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29c5da3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1aba50 con 0x556987eb7940 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29c5da3740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1acec0 con 0x556987eb27d0 2024-04-27T00:07:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29c5da3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc1ac120 con 0x556987eb1de0 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29c5da3740 10 client.0 _ll_register_callbacks cb 0x7f29a40d4400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29c1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29c1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46466/0 (socket says 172.21.15.40:46466) 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29c1000640 1 -- 172.21.15.40:0/999165666 learned_addr learned my addr 172.21.15.40:0/999165666 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29b3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987ec11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29bbe00640 1 --2- 172.21.15.40:0/999165666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 0x556987ec0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29b3e00640 1 -- 172.21.15.40:0/999165666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 msgr2=0x556987ec0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29b3e00640 1 --2- 172.21.15.40:0/999165666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 0x556987ec0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29b3e00640 1 -- 172.21.15.40:0/999165666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 msgr2=0x556987ec2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.758+0000 7f29b3e00640 1 --2- 172.21.15.40:0/999165666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.759+0000 7f29b3e00640 1 -- 172.21.15.40:0/999165666 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556987dd0cb0 con 0x556987eb27d0 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.759+0000 7f29b3e00640 1 --2- 172.21.15.40:0/999165666 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987ec11c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x556987ebd140 tx=0x7f29a40a5650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d567a82fbb0a9944 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.759+0000 7f29aa000640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:07:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.759+0000 7f29aa000640 1 -- 172.21.15.40:0/999165666 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a409c340 con 0x556987eb27d0 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.759+0000 7f29c5da3740 1 -- 172.21.15.40:0/999165666 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556987e1de00 con 0x556987eb27d0 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.759+0000 7f29aa000640 1 -- 172.21.15.40:0/999165666 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f29a409c030 con 0x556987eb27d0 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.759+0000 7f29aa000640 1 -- 172.21.15.40:0/999165666 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f29a409c9e0 con 0x556987eb27d0 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.759+0000 7f29c5da3740 10 client.25805 fetch_fsmap learned FSMap version 24 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.759+0000 7f29c5da3740 1 -- 172.21.15.40:0/999165666 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556987ec3890 con 0x556987eb27d0 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.760+0000 7f29aa000640 1 -- 172.21.15.40:0/999165666 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556987e1de00 con 0x556987eb27d0 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.760+0000 7f29aa000640 1 -- 172.21.15.40:0/999165666 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f29a40a1f20 con 0x556987eb27d0 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.760+0000 7f29c5da3740 10 client.25805 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.760+0000 7f29c5da3740 10 client.25805 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.760+0000 7f29c5da3740 20 client.25805 populate_metadata read hostname 'smithi040' 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.760+0000 7f29c5da3740 1 --2- 172.21.15.40:0/999165666 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556987f2e4b0 0x556987f308d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.760+0000 7f29c5da3740 4 client.25805 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.760+0000 7f29c5da3740 1 -- 172.21.15.40:0/999165666 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556987d994b0 con 0x556987f2e4b0 2024-04-27T00:07:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.760+0000 7f29bbe00640 1 --2- 172.21.15.40:0/999165666 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556987f2e4b0 0x556987f308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.761+0000 7f29bbe00640 1 --2- 172.21.15.40:0/999165666 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556987f2e4b0 0x556987f308d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc28f99bfb95d9c server_cookie=ea427d6d3d6acaef in_seq=0 out_seq=0 2024-04-27T00:07:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.761+0000 7f29aa000640 10 client.25805 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:05.778 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29aa000640 1 -- 172.21.15.40:0/999165666 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556987d994b0 con 0x556987f2e4b0 2024-04-27T00:07:05.778 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29aa000640 10 client.25805 handle_command_reply: tid=0 2024-04-27T00:07:05.778 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29c5da3740 1 client.25805 shutdown 2024-04-27T00:07:05.778 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29c5da3740 1 -- 172.21.15.40:0/999165666 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 msgr2=0x556987ec11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:05.778 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29c5da3740 1 --2- 172.21.15.40:0/999165666 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987ec11c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x556987ebd140 tx=0x7f29a40a5650 comp rx=0 tx=0).stop 2024-04-27T00:07:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29c5da3740 1 -- 172.21.15.40:0/999165666 shutdown_connections 2024-04-27T00:07:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29c5da3740 1 --2- 172.21.15.40:0/999165666 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556987f2e4b0 0x556987f308d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29c5da3740 1 --2- 172.21.15.40:0/999165666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556987eb7940 0x556987ec2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29c5da3740 1 --2- 172.21.15.40:0/999165666 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556987eb27d0 0x556987ec11c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29c5da3740 1 --2- 172.21.15.40:0/999165666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556987eb1de0 0x556987ec0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29c5da3740 1 -- 172.21.15.40:0/999165666 >> 172.21.15.40:0/999165666 conn(0x556987ead450 msgr2=0x556987eb0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29aa000640 0 client.25805 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29c5da3740 1 -- 172.21.15.40:0/999165666 shutdown_connections 2024-04-27T00:07:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29c5da3740 1 -- 172.21.15.40:0/999165666 wait complete. 2024-04-27T00:07:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.777+0000 7f29c5da3740 20 client.25805 trim_cache size 0 max 16384 2024-04-27T00:07:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:07:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:07:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:07:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:07:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.778+0000 7f29c1a00640 1 -- 172.21.15.40:0/1592642236 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29a00a3c30 msgr2=0x7f29a00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.778+0000 7f29c1a00640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29a00a3c30 0x7f29a00a60f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0096c0 tx=0x7f29ac005a20 comp rx=0 tx=0).stop 2024-04-27T00:07:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.778+0000 7f29c1a00640 1 -- 172.21.15.40:0/1592642236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29bc105be0 msgr2=0x7f29bc19b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.778+0000 7f29c1a00640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29bc105be0 0x7f29bc19b170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f29a400e9f0 tx=0x7f29a400edf0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.778+0000 7f29c1a00640 1 -- 172.21.15.40:0/1592642236 shutdown_connections 2024-04-27T00:07:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.778+0000 7f29c1a00640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29bc1065a0 0x7f29bc1a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.778+0000 7f29c1a00640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f29a00a3c30 0x7f29a00a60f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.778+0000 7f29c1a00640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29bc105be0 0x7f29bc19b170 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.778+0000 7f29c1a00640 1 --2- 172.21.15.40:0/1592642236 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f29bc1049f0 0x7f29bc19ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.778+0000 7f29c1a00640 1 -- 172.21.15.40:0/1592642236 >> 172.21.15.40:0/1592642236 conn(0x7f29bc0fffa0 msgr2=0x7f29bc108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.778+0000 7f29c1a00640 1 -- 172.21.15.40:0/1592642236 shutdown_connections 2024-04-27T00:07:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:05.779+0000 7f29c1a00640 1 -- 172.21.15.40:0/1592642236 wait complete. 2024-04-27T00:07:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:06 smithi040 ceph-mon[28095]: pgmap v574: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:07:06.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:06 smithi080 ceph-mon[32362]: pgmap v574: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:07:06.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:06 smithi103 ceph-mon[32952]: pgmap v574: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:07:08.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:08 smithi040 ceph-mon[28095]: pgmap v575: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:08.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:08 smithi080 ceph-mon[32362]: pgmap v575: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:08.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:08 smithi103 ceph-mon[32952]: pgmap v575: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:10.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:10 smithi080 ceph-mon[32362]: pgmap v576: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:07:10.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:10 smithi103 ceph-mon[32952]: pgmap v576: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:07:10.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:10 smithi040 ceph-mon[28095]: pgmap v576: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:07:12.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:12 smithi080 ceph-mon[32362]: pgmap v577: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-27T00:07:12.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:12 smithi103 ceph-mon[32952]: pgmap v577: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-27T00:07:12.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:12 smithi040 ceph-mon[28095]: pgmap v577: 129 pgs: 129 active+clean; 3.4 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-27T00:07:14.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:14 smithi080 ceph-mon[32362]: pgmap v578: 129 pgs: 129 active+clean; 3.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:07:14.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:14 smithi103 ceph-mon[32952]: pgmap v578: 129 pgs: 129 active+clean; 3.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:07:14.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:14 smithi040 ceph-mon[28095]: pgmap v578: 129 pgs: 129 active+clean; 3.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:07:16.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:16 smithi080 ceph-mon[32362]: pgmap v579: 129 pgs: 129 active+clean; 3.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:07:16.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:16 smithi103 ceph-mon[32952]: pgmap v579: 129 pgs: 129 active+clean; 3.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:07:16.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:16 smithi040 ceph-mon[28095]: pgmap v579: 129 pgs: 129 active+clean; 3.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:07:18.519 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:18 smithi040 ceph-mon[28095]: pgmap v580: 129 pgs: 129 active+clean; 3.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:18.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:18 smithi080 ceph-mon[32362]: pgmap v580: 129 pgs: 129 active+clean; 3.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:18.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:18 smithi103 ceph-mon[32952]: pgmap v580: 129 pgs: 129 active+clean; 3.5 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.926+0000 7fdb25a00640 1 -- 172.21.15.40:0/1412509964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb201016b0 msgr2=0x7fdb20101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.926+0000 7fdb25a00640 1 --2- 172.21.15.40:0/1412509964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb201016b0 0x7fdb20101ad0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009900 tx=0x7fdb1002a3f0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.926+0000 7fdb25a00640 1 -- 172.21.15.40:0/1412509964 shutdown_connections 2024-04-27T00:07:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.926+0000 7fdb25a00640 1 --2- 172.21.15.40:0/1412509964 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb20103260 0x7fdb20109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.926+0000 7fdb25a00640 1 --2- 172.21.15.40:0/1412509964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb201028a0 0x7fdb20102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.926+0000 7fdb25a00640 1 --2- 172.21.15.40:0/1412509964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb201016b0 0x7fdb20101ad0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.926+0000 7fdb25a00640 1 -- 172.21.15.40:0/1412509964 >> 172.21.15.40:0/1412509964 conn(0x7fdb200fcc20 msgr2=0x7fdb200ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.926+0000 7fdb25a00640 1 -- 172.21.15.40:0/1412509964 shutdown_connections 2024-04-27T00:07:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.926+0000 7fdb25a00640 1 -- 172.21.15.40:0/1412509964 wait complete. 2024-04-27T00:07:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.927+0000 7fdb25a00640 1 Processor -- start 2024-04-27T00:07:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.928+0000 7fdb25a00640 1 -- start start 2024-04-27T00:07:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.928+0000 7fdb25a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb201016b0 0x7fdb20198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.928+0000 7fdb25a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb201028a0 0x7fdb20198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.928+0000 7fdb25a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb20103260 0x7fdb2019fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.928+0000 7fdb25a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2010e360 con 0x7fdb20103260 2024-04-27T00:07:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.928+0000 7fdb25a00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2010e1e0 con 0x7fdb201016b0 2024-04-27T00:07:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.928+0000 7fdb25a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2010e4e0 con 0x7fdb201028a0 2024-04-27T00:07:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.928+0000 7fdb1fe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb201016b0 0x7fdb20198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.928+0000 7fdb25000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb20103260 0x7fdb2019fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.928+0000 7fdb1fe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb201016b0 0x7fdb20198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:54812/0 (socket says 172.21.15.40:54812) 2024-04-27T00:07:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.928+0000 7fdb17400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb201028a0 0x7fdb20198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.928+0000 7fdb1fe00640 1 -- 172.21.15.40:0/2496830992 learned_addr learned my addr 172.21.15.40:0/2496830992 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.929+0000 7fdb1fe00640 1 -- 172.21.15.40:0/2496830992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb201028a0 msgr2=0x7fdb20198f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.929+0000 7fdb1fe00640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb201028a0 0x7fdb20198f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.929+0000 7fdb1fe00640 1 -- 172.21.15.40:0/2496830992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb20103260 msgr2=0x7fdb2019fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.929+0000 7fdb1fe00640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb20103260 0x7fdb2019fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.929+0000 7fdb1fe00640 1 -- 172.21.15.40:0/2496830992 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb201a0660 con 0x7fdb201016b0 2024-04-27T00:07:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.929+0000 7fdb17400640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb201028a0 0x7fdb20198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:07:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.929+0000 7fdb1fe00640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb201016b0 0x7fdb20198a40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdb10031500 tx=0x7fdb10003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.930+0000 7fdb1d600640 1 -- 172.21.15.40:0/2496830992 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1003f070 con 0x7fdb201016b0 2024-04-27T00:07:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.930+0000 7fdb25a00640 1 -- 172.21.15.40:0/2496830992 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb201a08f0 con 0x7fdb201016b0 2024-04-27T00:07:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.930+0000 7fdb1d600640 1 -- 172.21.15.40:0/2496830992 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb10031c50 con 0x7fdb201016b0 2024-04-27T00:07:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.930+0000 7fdb1d600640 1 -- 172.21.15.40:0/2496830992 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb10035030 con 0x7fdb201016b0 2024-04-27T00:07:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.930+0000 7fdb25a00640 1 -- 172.21.15.40:0/2496830992 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb201a0e30 con 0x7fdb201016b0 2024-04-27T00:07:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.933+0000 7fdb29df4740 1 Processor -- start 2024-04-27T00:07:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.933+0000 7fdb29df4740 1 -- start start 2024-04-27T00:07:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.933+0000 7fdb29df4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e064c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.933+0000 7fdb29df4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e064cbe0 0x55e8e06516e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.933+0000 7fdb1d600640 1 -- 172.21.15.40:0/2496830992 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdb10042070 con 0x7fdb201016b0 2024-04-27T00:07:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.933+0000 7fdb29df4740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e0651d50 0x55e8e06541e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.933+0000 7fdb29df4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e8e05ac380 con 0x55e8e064cbe0 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.933+0000 7fdb29df4740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55e8e0548c30 con 0x55e8e0651d50 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.933+0000 7fdb29df4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e8e05a8340 con 0x55e8e064c1f0 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.934+0000 7fdb1fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e064cbe0 0x55e8e06516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.934+0000 7fdb25000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e064c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.934+0000 7fdb25000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e064c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:53844/0 (socket says 172.21.15.40:53844) 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.934+0000 7fdb25000640 1 -- 172.21.15.40:0/3895575167 learned_addr learned my addr 172.21.15.40:0/3895575167 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.934+0000 7fdb17400640 1 --2- 172.21.15.40:0/3895575167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e0651d50 0x55e8e06541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.934+0000 7fdb25000640 1 -- 172.21.15.40:0/3895575167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e0651d50 msgr2=0x55e8e06541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.934+0000 7fdb25000640 1 --2- 172.21.15.40:0/3895575167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e0651d50 0x55e8e06541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.934+0000 7fdb25000640 1 -- 172.21.15.40:0/3895575167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e064cbe0 msgr2=0x55e8e06516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.934+0000 7fdb25000640 1 --2- 172.21.15.40:0/3895575167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e064cbe0 0x55e8e06516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.934+0000 7fdb25000640 1 -- 172.21.15.40:0/3895575167 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8e056a940 con 0x55e8e064c1f0 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.935+0000 7fdb25000640 1 --2- 172.21.15.40:0/3895575167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e064c610 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdb18011440 tx=0x7fdb18011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65d06d9e96597bff server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.935+0000 7fdb15600640 1 -- 172.21.15.40:0/3895575167 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18003c30 con 0x55e8e064c1f0 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.935+0000 7fdb15600640 1 -- 172.21.15.40:0/3895575167 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb18003dd0 con 0x55e8e064c1f0 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.935+0000 7fdb15600640 1 -- 172.21.15.40:0/3895575167 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1800fae0 con 0x55e8e064c1f0 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.935+0000 7fdb1d600640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdb000a3cb0 0x7fdb000a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb1d600640 1 -- 172.21.15.40:0/2496830992 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fdb100f0fb0 con 0x7fdb201016b0 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb1fe00640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdb000a3cb0 0x7fdb000a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb29df4740 1 -- 172.21.15.40:0/3895575167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 msgr2=0x55e8e064c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb29df4740 1 --2- 172.21.15.40:0/3895575167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e064c610 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdb18011440 tx=0x7fdb18011900 comp rx=0 tx=0).stop 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb29df4740 1 -- 172.21.15.40:0/3895575167 shutdown_connections 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb29df4740 1 --2- 172.21.15.40:0/3895575167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e0651d50 0x55e8e06541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb29df4740 1 --2- 172.21.15.40:0/3895575167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e064cbe0 0x55e8e06516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb29df4740 1 --2- 172.21.15.40:0/3895575167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e064c610 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb29df4740 1 -- 172.21.15.40:0/3895575167 >> 172.21.15.40:0/3895575167 conn(0x55e8e0643450 msgr2=0x55e8e0643850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb29df4740 1 -- 172.21.15.40:0/3895575167 shutdown_connections 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb29df4740 1 -- 172.21.15.40:0/3895575167 wait complete. 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb1fe00640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdb000a3cb0 0x7fdb000a6170 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x55e8e064cfe0 tx=0x7fdb100f4a60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.936+0000 7fdb29df4740 1 -- start start 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.937+0000 7fdb29df4740 1 Processor -- start 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.937+0000 7fdb29df4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e0656410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.937+0000 7fdb29df4740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 0x55e8e0656950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.937+0000 7fdb29df4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e0651d50 0x55e8e065d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.937+0000 7fdb29df4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e8e0654d00 con 0x55e8e0651d50 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.937+0000 7fdb29df4740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55e8e0533c60 con 0x55e8e064cbe0 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.937+0000 7fdb29df4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e8e0654e80 con 0x55e8e064c1f0 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.937+0000 7fdb29df4740 10 client.0 _ll_register_callbacks cb 0x55e8e055fe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:07:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.937+0000 7fdb17400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 0x55e8e0656950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb17400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 0x55e8e0656950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:54828/0 (socket says 172.21.15.40:54828) 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb17400640 1 -- 172.21.15.40:0/4046604007 learned_addr learned my addr 172.21.15.40:0/4046604007 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb25000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e0656410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb25000640 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e0651d50 0x55e8e065d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb17400640 1 -- 172.21.15.40:0/4046604007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 msgr2=0x55e8e0656410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb17400640 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e0656410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb17400640 1 -- 172.21.15.40:0/4046604007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e0651d50 msgr2=0x55e8e065d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb17400640 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e0651d50 0x55e8e065d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb17400640 1 -- 172.21.15.40:0/4046604007 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8e0542510 con 0x55e8e064cbe0 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb25000640 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e0651d50 0x55e8e065d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb17400640 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 0x55e8e0656950 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdb08012d20 tx=0x7fdb0800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e177408fb285d163 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb0e000640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.938+0000 7fdb0e000640 1 -- 172.21.15.40:0/4046604007 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb08012110 con 0x55e8e064cbe0 2024-04-27T00:07:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.939+0000 7fdb0e000640 1 -- 172.21.15.40:0/4046604007 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fdb08004030 con 0x55e8e064cbe0 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.939+0000 7fdb29df4740 1 -- 172.21.15.40:0/4046604007 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb18003dd0 con 0x55e8e064cbe0 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.939+0000 7fdb0e000640 1 -- 172.21.15.40:0/4046604007 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdb080122b0 con 0x55e8e064cbe0 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.939+0000 7fdb29df4740 10 client.25817 fetch_fsmap learned FSMap version 24 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.939+0000 7fdb29df4740 1 -- 172.21.15.40:0/4046604007 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e8e02be210 con 0x55e8e064cbe0 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.939+0000 7fdb0e000640 1 -- 172.21.15.40:0/4046604007 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb18003dd0 con 0x55e8e064cbe0 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.940+0000 7fdb0e000640 1 -- 172.21.15.40:0/4046604007 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdb08010430 con 0x55e8e064cbe0 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.940+0000 7fdb29df4740 10 client.25817 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.940+0000 7fdb29df4740 10 client.25817 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.940+0000 7fdb29df4740 20 client.25817 populate_metadata read hostname 'smithi040' 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.940+0000 7fdb29df4740 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e8e0661f50 0x55e8e0664340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.940+0000 7fdb29df4740 4 client.25817 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.940+0000 7fdb29df4740 1 -- 172.21.15.40:0/4046604007 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8e05b9f90 con 0x55e8e0661f50 2024-04-27T00:07:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.940+0000 7fdb25000640 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e8e0661f50 0x55e8e0664340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.941+0000 7fdb25000640 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e8e0661f50 0x55e8e0664340 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc41cb88362ecf51 server_cookie=a7295d927cd9bc81 in_seq=0 out_seq=0 2024-04-27T00:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.941+0000 7fdb0e000640 10 client.25817 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.941+0000 7fdb0e000640 1 -- 172.21.15.40:0/4046604007 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e8e05b9f90 con 0x55e8e0661f50 2024-04-27T00:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb0e000640 10 client.25817 handle_command_reply: tid=0 2024-04-27T00:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb29df4740 1 client.25817 shutdown 2024-04-27T00:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb29df4740 1 -- 172.21.15.40:0/4046604007 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 msgr2=0x55e8e0656950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb29df4740 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 0x55e8e0656950 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdb08012d20 tx=0x7fdb0800f790 comp rx=0 tx=0).stop 2024-04-27T00:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb29df4740 1 -- 172.21.15.40:0/4046604007 shutdown_connections 2024-04-27T00:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb29df4740 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e8e0661f50 0x55e8e0664340 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb29df4740 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e0651d50 0x55e8e065d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb29df4740 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 0x55e8e0656950 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb0e000640 0 client.25817 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb29df4740 1 --2- 172.21.15.40:0/4046604007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e0656410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb29df4740 1 -- 172.21.15.40:0/4046604007 >> 172.21.15.40:0/4046604007 conn(0x55e8e0643450 msgr2=0x55e8e0642650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb29df4740 1 -- 172.21.15.40:0/4046604007 shutdown_connections 2024-04-27T00:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb29df4740 1 -- 172.21.15.40:0/4046604007 wait complete. 2024-04-27T00:07:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.942+0000 7fdb29df4740 20 client.25817 trim_cache size 0 max 16384 2024-04-27T00:07:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.958+0000 7fdb29df4740 1 Processor -- start 2024-04-27T00:07:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.958+0000 7fdb29df4740 1 -- start start 2024-04-27T00:07:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.959+0000 7fdb29df4740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064c1f0 0x55e8e064ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.959+0000 7fdb29df4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e064cbe0 0x55e8e064e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.959+0000 7fdb29df4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e0651d50 0x55e8e065d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.959+0000 7fdb29df4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb201aba20 con 0x55e8e064cbe0 2024-04-27T00:07:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.959+0000 7fdb29df4740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fdb201a92f0 con 0x55e8e064c1f0 2024-04-27T00:07:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.959+0000 7fdb29df4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb201a7140 con 0x55e8e0651d50 2024-04-27T00:07:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.959+0000 7fdb17400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e064cbe0 0x55e8e064e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.959+0000 7fdb17400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e064cbe0 0x55e8e064e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46792/0 (socket says 172.21.15.40:46792) 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.959+0000 7fdb17400640 1 -- 172.21.15.40:0/939918439 learned_addr learned my addr 172.21.15.40:0/939918439 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.959+0000 7fdb25000640 1 --2- 172.21.15.40:0/939918439 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064c1f0 0x55e8e064ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.959+0000 7fdb25000640 1 --2- 172.21.15.40:0/939918439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e0651d50 0x55e8e065d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.960+0000 7fdb17400640 1 -- 172.21.15.40:0/939918439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e0651d50 msgr2=0x55e8e065d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.960+0000 7fdb17400640 1 --2- 172.21.15.40:0/939918439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e0651d50 0x55e8e065d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.960+0000 7fdb17400640 1 -- 172.21.15.40:0/939918439 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064c1f0 msgr2=0x55e8e064ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.960+0000 7fdb17400640 1 --2- 172.21.15.40:0/939918439 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064c1f0 0x55e8e064ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.960+0000 7fdb17400640 1 -- 172.21.15.40:0/939918439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8e06646b0 con 0x55e8e064cbe0 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.960+0000 7fdb25000640 1 --2- 172.21.15.40:0/939918439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e0651d50 0x55e8e065d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.960+0000 7fdb17400640 1 --2- 172.21.15.40:0/939918439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e064cbe0 0x55e8e064e300 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdb08000c00 tx=0x7fdb08012990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3057f7a0fd50c16 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.960+0000 7fdb14c00640 1 -- 172.21.15.40:0/939918439 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb08010430 con 0x55e8e064cbe0 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.960+0000 7fdb14c00640 1 -- 172.21.15.40:0/939918439 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb080105d0 con 0x55e8e064cbe0 2024-04-27T00:07:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.960+0000 7fdb14c00640 1 -- 172.21.15.40:0/939918439 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb08003850 con 0x55e8e064cbe0 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.961+0000 7fdb29df4740 1 -- 172.21.15.40:0/939918439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e064cbe0 msgr2=0x55e8e064e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.961+0000 7fdb29df4740 1 --2- 172.21.15.40:0/939918439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e064cbe0 0x55e8e064e300 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdb08000c00 tx=0x7fdb08012990 comp rx=0 tx=0).stop 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.961+0000 7fdb29df4740 1 -- 172.21.15.40:0/939918439 shutdown_connections 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.961+0000 7fdb29df4740 1 --2- 172.21.15.40:0/939918439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e0651d50 0x55e8e065d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.961+0000 7fdb29df4740 1 --2- 172.21.15.40:0/939918439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e064cbe0 0x55e8e064e300 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.961+0000 7fdb29df4740 1 --2- 172.21.15.40:0/939918439 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064c1f0 0x55e8e064ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.961+0000 7fdb29df4740 1 -- 172.21.15.40:0/939918439 >> 172.21.15.40:0/939918439 conn(0x55e8e0643450 msgr2=0x55e8e0656410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.961+0000 7fdb29df4740 1 -- 172.21.15.40:0/939918439 shutdown_connections 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.961+0000 7fdb29df4740 1 -- 172.21.15.40:0/939918439 wait complete. 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.962+0000 7fdb29df4740 1 -- start start 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.962+0000 7fdb29df4740 1 Processor -- start 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.962+0000 7fdb29df4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e065b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.962+0000 7fdb29df4740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 0x55e8e065b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.962+0000 7fdb29df4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e0651d50 0x55e8e065c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.962+0000 7fdb29df4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb201a77b0 con 0x55e8e0651d50 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.962+0000 7fdb29df4740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fdb201a8c20 con 0x55e8e064cbe0 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.962+0000 7fdb29df4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb201a7e80 con 0x55e8e064c1f0 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.962+0000 7fdb29df4740 10 client.0 _ll_register_callbacks cb 0x7fdb08009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.963+0000 7fdb17400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 0x55e8e065b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.963+0000 7fdb17400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 0x55e8e065b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:54844/0 (socket says 172.21.15.40:54844) 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.963+0000 7fdb17400640 1 -- 172.21.15.40:0/2442117948 learned_addr learned my addr 172.21.15.40:0/2442117948 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.963+0000 7fdb25000640 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e065b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.963+0000 7fdb25000640 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e0651d50 0x55e8e065c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.963+0000 7fdb17400640 1 -- 172.21.15.40:0/2442117948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 msgr2=0x55e8e065b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.963+0000 7fdb17400640 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e065b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.963+0000 7fdb17400640 1 -- 172.21.15.40:0/2442117948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e0651d50 msgr2=0x55e8e065c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.963+0000 7fdb17400640 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e0651d50 0x55e8e065c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.963+0000 7fdb17400640 1 -- 172.21.15.40:0/2442117948 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8e038a620 con 0x55e8e064cbe0 2024-04-27T00:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.963+0000 7fdb17400640 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 0x55e8e065b910 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdb080058c0 tx=0x7fdb08005300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7703b08f4a6c3a4e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.963+0000 7fdb0e000640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.964+0000 7fdb0e000640 1 -- 172.21.15.40:0/2442117948 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb08010430 con 0x55e8e064cbe0 2024-04-27T00:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.964+0000 7fdb29df4740 1 -- 172.21.15.40:0/2442117948 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb080105d0 con 0x55e8e064cbe0 2024-04-27T00:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.964+0000 7fdb0e000640 1 -- 172.21.15.40:0/2442117948 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fdb08003800 con 0x55e8e064cbe0 2024-04-27T00:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.964+0000 7fdb0e000640 1 -- 172.21.15.40:0/2442117948 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdb0803f050 con 0x55e8e064cbe0 2024-04-27T00:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.964+0000 7fdb29df4740 10 client.25823 fetch_fsmap learned FSMap version 24 2024-04-27T00:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.964+0000 7fdb29df4740 1 -- 172.21.15.40:0/2442117948 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e8e0566cb0 con 0x55e8e064cbe0 2024-04-27T00:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.964+0000 7fdb0e000640 1 -- 172.21.15.40:0/2442117948 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb080105d0 con 0x55e8e064cbe0 2024-04-27T00:07:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.964+0000 7fdb0e000640 1 -- 172.21.15.40:0/2442117948 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdb080297d0 con 0x55e8e064cbe0 2024-04-27T00:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.965+0000 7fdb29df4740 10 client.25823 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.965+0000 7fdb29df4740 10 client.25823 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.965+0000 7fdb29df4740 20 client.25823 populate_metadata read hostname 'smithi040' 2024-04-27T00:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.965+0000 7fdb29df4740 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e8e06c8cf0 0x55e8e06cb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.965+0000 7fdb29df4740 4 client.25823 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.965+0000 7fdb29df4740 1 -- 172.21.15.40:0/2442117948 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e8e05b3e00 con 0x55e8e06c8cf0 2024-04-27T00:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.965+0000 7fdb25000640 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e8e06c8cf0 0x55e8e06cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.965+0000 7fdb25000640 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e8e06c8cf0 0x55e8e06cb110 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78c46ce06c801e84 server_cookie=f1c5831af53280f0 in_seq=0 out_seq=0 2024-04-27T00:07:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.965+0000 7fdb0e000640 10 client.25823 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.981+0000 7fdb0e000640 1 -- 172.21.15.40:0/2442117948 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e8e05b3e00 con 0x55e8e06c8cf0 2024-04-27T00:07:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.981+0000 7fdb0e000640 10 client.25823 handle_command_reply: tid=0 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.981+0000 7fdb29df4740 1 client.25823 shutdown 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.981+0000 7fdb29df4740 1 -- 172.21.15.40:0/2442117948 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 msgr2=0x55e8e065b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.981+0000 7fdb29df4740 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 0x55e8e065b910 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdb080058c0 tx=0x7fdb08005300 comp rx=0 tx=0).stop 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.982+0000 7fdb29df4740 1 -- 172.21.15.40:0/2442117948 shutdown_connections 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.982+0000 7fdb29df4740 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e8e06c8cf0 0x55e8e06cb110 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.982+0000 7fdb29df4740 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8e0651d50 0x55e8e065c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.982+0000 7fdb29df4740 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e8e064cbe0 0x55e8e065b910 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.982+0000 7fdb29df4740 1 --2- 172.21.15.40:0/2442117948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8e064c1f0 0x55e8e065b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.982+0000 7fdb29df4740 1 -- 172.21.15.40:0/2442117948 >> 172.21.15.40:0/2442117948 conn(0x55e8e0643450 msgr2=0x55e8e064ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.982+0000 7fdb0e000640 0 client.25823 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.982+0000 7fdb29df4740 1 -- 172.21.15.40:0/2442117948 shutdown_connections 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.982+0000 7fdb29df4740 1 -- 172.21.15.40:0/2442117948 wait complete. 2024-04-27T00:07:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.982+0000 7fdb29df4740 20 client.25823 trim_cache size 0 max 16384 2024-04-27T00:07:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:07:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:07:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:07:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:07:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.983+0000 7fdb25a00640 1 -- 172.21.15.40:0/2496830992 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdb000a3cb0 msgr2=0x7fdb000a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.983+0000 7fdb25a00640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdb000a3cb0 0x7fdb000a6170 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x55e8e064cfe0 tx=0x7fdb100f4a60 comp rx=0 tx=0).stop 2024-04-27T00:07:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.983+0000 7fdb25a00640 1 -- 172.21.15.40:0/2496830992 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb201016b0 msgr2=0x7fdb20198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.983+0000 7fdb25a00640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb201016b0 0x7fdb20198a40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdb10031500 tx=0x7fdb10003c10 comp rx=0 tx=0).stop 2024-04-27T00:07:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.983+0000 7fdb25a00640 1 -- 172.21.15.40:0/2496830992 shutdown_connections 2024-04-27T00:07:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.983+0000 7fdb25a00640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb20103260 0x7fdb2019fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.983+0000 7fdb25a00640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdb000a3cb0 0x7fdb000a6170 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.983+0000 7fdb25a00640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb201028a0 0x7fdb20198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.983+0000 7fdb25a00640 1 --2- 172.21.15.40:0/2496830992 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb201016b0 0x7fdb20198a40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.983+0000 7fdb25a00640 1 -- 172.21.15.40:0/2496830992 >> 172.21.15.40:0/2496830992 conn(0x7fdb200fcc20 msgr2=0x7fdb20105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.983+0000 7fdb25a00640 1 -- 172.21.15.40:0/2496830992 shutdown_connections 2024-04-27T00:07:18.985 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:18.983+0000 7fdb25a00640 1 -- 172.21.15.40:0/2496830992 wait complete. 2024-04-27T00:07:20.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:20 smithi080 ceph-mon[32362]: pgmap v581: 129 pgs: 129 active+clean; 3.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:07:20.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:20 smithi103 ceph-mon[32952]: pgmap v581: 129 pgs: 129 active+clean; 3.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:07:20.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:20 smithi040 ceph-mon[28095]: pgmap v581: 129 pgs: 129 active+clean; 3.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:07:22.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:22 smithi040 ceph-mon[28095]: pgmap v582: 129 pgs: 129 active+clean; 3.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:07:22.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:22 smithi080 ceph-mon[32362]: pgmap v582: 129 pgs: 129 active+clean; 3.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:07:22.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:22 smithi103 ceph-mon[32952]: pgmap v582: 129 pgs: 129 active+clean; 3.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:07:24.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:24 smithi080 ceph-mon[32362]: pgmap v583: 129 pgs: 129 active+clean; 3.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:07:24.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:24 smithi103 ceph-mon[32952]: pgmap v583: 129 pgs: 129 active+clean; 3.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:07:24.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:24 smithi040 ceph-mon[28095]: pgmap v583: 129 pgs: 129 active+clean; 3.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:07:26.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:26 smithi080 ceph-mon[32362]: pgmap v584: 129 pgs: 129 active+clean; 3.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:26.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:26 smithi103 ceph-mon[32952]: pgmap v584: 129 pgs: 129 active+clean; 3.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:26.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:26 smithi040 ceph-mon[28095]: pgmap v584: 129 pgs: 129 active+clean; 3.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:28.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:28 smithi080 ceph-mon[32362]: pgmap v585: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:28.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:28 smithi103 ceph-mon[32952]: pgmap v585: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:28.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:28 smithi040 ceph-mon[28095]: pgmap v585: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:30.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:30 smithi080 ceph-mon[32362]: pgmap v586: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:30.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:30 smithi103 ceph-mon[32952]: pgmap v586: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:30.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:30 smithi040 ceph-mon[28095]: pgmap v586: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.132+0000 7f6777200640 1 -- 172.21.15.40:0/3270869552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770103250 msgr2=0x7f6770105640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.132+0000 7f6777200640 1 --2- 172.21.15.40:0/3270869552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770103250 0x7f6770105640 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6764009900 tx=0x7f676402a410 comp rx=0 tx=0).stop 2024-04-27T00:07:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.132+0000 7f6777200640 1 -- 172.21.15.40:0/3270869552 shutdown_connections 2024-04-27T00:07:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.132+0000 7f6777200640 1 --2- 172.21.15.40:0/3270869552 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67701084b0 0x7f677010a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.132+0000 7f6777200640 1 --2- 172.21.15.40:0/3270869552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6770105b80 0x7f6770107f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.132+0000 7f6777200640 1 --2- 172.21.15.40:0/3270869552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770103250 0x7f6770105640 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.132+0000 7f6777200640 1 -- 172.21.15.40:0/3270869552 >> 172.21.15.40:0/3270869552 conn(0x7f67700fcc60 msgr2=0x7f67700ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.133+0000 7f6777200640 1 -- 172.21.15.40:0/3270869552 shutdown_connections 2024-04-27T00:07:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.133+0000 7f6777200640 1 -- 172.21.15.40:0/3270869552 wait complete. 2024-04-27T00:07:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6777200640 1 Processor -- start 2024-04-27T00:07:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6777200640 1 -- start start 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6777200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770103250 0x7f677019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6777200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6770105b80 0x7f677019b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6777200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67701084b0 0x7f67701a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6777200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f677010e380 con 0x7f6770103250 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6777200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f677010e200 con 0x7f67701084b0 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6777200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f677010e500 con 0x7f6770105b80 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6776800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67701084b0 0x7f67701a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6775e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770103250 0x7f677019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6775400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6770105b80 0x7f677019b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6776800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67701084b0 0x7f67701a2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:34816/0 (socket says 172.21.15.40:34816) 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.134+0000 7f6776800640 1 -- 172.21.15.40:0/3022898348 learned_addr learned my addr 172.21.15.40:0/3022898348 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.135+0000 7f6776800640 1 -- 172.21.15.40:0/3022898348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6770105b80 msgr2=0x7f677019b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.135+0000 7f6776800640 1 --2- 172.21.15.40:0/3022898348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6770105b80 0x7f677019b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.135+0000 7f6776800640 1 -- 172.21.15.40:0/3022898348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770103250 msgr2=0x7f677019aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.135+0000 7f6776800640 1 --2- 172.21.15.40:0/3022898348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770103250 0x7f677019aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.135+0000 7f6776800640 1 -- 172.21.15.40:0/3022898348 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67701a28e0 con 0x7f67701084b0 2024-04-27T00:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.135+0000 7f6776800640 1 --2- 172.21.15.40:0/3022898348 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67701084b0 0x7f67701a2240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f676c00eb10 tx=0x7f676c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.135+0000 7f676aa00640 1 -- 172.21.15.40:0/3022898348 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c004b20 con 0x7f67701084b0 2024-04-27T00:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.135+0000 7f6777200640 1 -- 172.21.15.40:0/3022898348 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67701a2bd0 con 0x7f67701084b0 2024-04-27T00:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.135+0000 7f676aa00640 1 -- 172.21.15.40:0/3022898348 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f676c004cc0 con 0x7f67701084b0 2024-04-27T00:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.136+0000 7f676aa00640 1 -- 172.21.15.40:0/3022898348 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c010670 con 0x7f67701084b0 2024-04-27T00:07:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.136+0000 7f6777200640 1 -- 172.21.15.40:0/3022898348 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67701a30a0 con 0x7f67701084b0 2024-04-27T00:07:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.139+0000 7f677b5b5740 1 Processor -- start 2024-04-27T00:07:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.139+0000 7f676aa00640 1 -- 172.21.15.40:0/3022898348 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f676c005020 con 0x7f67701084b0 2024-04-27T00:07:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.139+0000 7f677b5b5740 1 -- start start 2024-04-27T00:07:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f677b5b5740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50acade0 0x563f50acb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f677b5b5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 0x563f50ad02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f677b5b5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50ad0940 0x563f50ad2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f677b5b5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563f50a2f380 con 0x563f50acb7d0 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f677b5b5740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x563f509cbc30 con 0x563f50acade0 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f677b5b5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563f50a2b340 con 0x563f50ad0940 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f6775e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 0x563f50ad02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f6775e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 0x563f50ad02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36384/0 (socket says 172.21.15.40:36384) 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f6775e00640 1 -- 172.21.15.40:0/1326636864 learned_addr learned my addr 172.21.15.40:0/1326636864 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f6775400640 1 --2- 172.21.15.40:0/1326636864 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50acade0 0x563f50acb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f6775400640 1 --2- 172.21.15.40:0/1326636864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50ad0940 0x563f50ad2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f6775e00640 1 -- 172.21.15.40:0/1326636864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50ad0940 msgr2=0x563f50ad2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f6775e00640 1 --2- 172.21.15.40:0/1326636864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50ad0940 0x563f50ad2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f6775e00640 1 -- 172.21.15.40:0/1326636864 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50acade0 msgr2=0x563f50acb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f6775e00640 1 --2- 172.21.15.40:0/1326636864 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50acade0 0x563f50acb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f6775e00640 1 -- 172.21.15.40:0/1326636864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f509ed940 con 0x563f50acb7d0 2024-04-27T00:07:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.140+0000 7f6775400640 1 --2- 172.21.15.40:0/1326636864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50ad0940 0x563f50ad2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.141+0000 7f6775e00640 1 --2- 172.21.15.40:0/1326636864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 0x563f50ad02d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6764006010 tx=0x7f6764004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dbec726f9104f80 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.141+0000 7f676aa00640 1 --2- 172.21.15.40:0/3022898348 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f67580a3c30 0x7f67580a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.141+0000 7f675f000640 1 -- 172.21.15.40:0/1326636864 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764035030 con 0x563f50acb7d0 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.141+0000 7f675f000640 1 -- 172.21.15.40:0/1326636864 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6764040070 con 0x563f50acb7d0 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.141+0000 7f676aa00640 1 -- 172.21.15.40:0/3022898348 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f676c0d0ef0 con 0x7f67701084b0 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.141+0000 7f675f000640 1 -- 172.21.15.40:0/1326636864 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764031920 con 0x563f50acb7d0 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.141+0000 7f6775400640 1 --2- 172.21.15.40:0/3022898348 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f67580a3c30 0x7f67580a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.142+0000 7f677b5b5740 1 -- 172.21.15.40:0/1326636864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 msgr2=0x563f50ad02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.142+0000 7f677b5b5740 1 --2- 172.21.15.40:0/1326636864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 0x563f50ad02d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6764006010 tx=0x7f6764004040 comp rx=0 tx=0).stop 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.142+0000 7f6775400640 1 --2- 172.21.15.40:0/3022898348 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f67580a3c30 0x7f67580a60f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x563f50acc030 tx=0x7f6760004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.142+0000 7f677b5b5740 1 -- 172.21.15.40:0/1326636864 shutdown_connections 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.142+0000 7f677b5b5740 1 --2- 172.21.15.40:0/1326636864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50ad0940 0x563f50ad2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.142+0000 7f677b5b5740 1 --2- 172.21.15.40:0/1326636864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 0x563f50ad02d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.142+0000 7f677b5b5740 1 --2- 172.21.15.40:0/1326636864 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50acade0 0x563f50acb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.142+0000 7f677b5b5740 1 -- 172.21.15.40:0/1326636864 >> 172.21.15.40:0/1326636864 conn(0x563f50ac6450 msgr2=0x563f50ac6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.142+0000 7f677b5b5740 1 -- 172.21.15.40:0/1326636864 shutdown_connections 2024-04-27T00:07:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.142+0000 7f677b5b5740 1 -- 172.21.15.40:0/1326636864 wait complete. 2024-04-27T00:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.143+0000 7f677b5b5740 1 -- start start 2024-04-27T00:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.143+0000 7f677b5b5740 1 Processor -- start 2024-04-27T00:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.143+0000 7f677b5b5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acade0 0x563f50ad4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.143+0000 7f677b5b5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acb7d0 0x563f50ad5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.143+0000 7f677b5b5740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50ad0940 0x563f50adc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f677b5b5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563f50ad3890 con 0x563f50acade0 2024-04-27T00:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f677b5b5740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x563f509b6c60 con 0x563f50ad0940 2024-04-27T00:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f677b5b5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563f50ad3a10 con 0x563f50acb7d0 2024-04-27T00:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f677b5b5740 10 client.0 _ll_register_callbacks cb 0x563f509e2e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:07:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f6775e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acade0 0x563f50ad4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f6775e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acade0 0x563f50ad4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36396/0 (socket says 172.21.15.40:36396) 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f6775e00640 1 -- 172.21.15.40:0/3178391696 learned_addr learned my addr 172.21.15.40:0/3178391696 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f6775400640 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acb7d0 0x563f50ad5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f6776800640 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50ad0940 0x563f50adc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f6775e00640 1 -- 172.21.15.40:0/3178391696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acb7d0 msgr2=0x563f50ad5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f6775e00640 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acb7d0 0x563f50ad5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f6775e00640 1 -- 172.21.15.40:0/3178391696 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50ad0940 msgr2=0x563f50adc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f6775e00640 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50ad0940 0x563f50adc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.144+0000 7f6775e00640 1 -- 172.21.15.40:0/3178391696 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f509c5510 con 0x563f50acade0 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.145+0000 7f6775e00640 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acade0 0x563f50ad4f40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f676402a450 tx=0x7f6764004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c6bf9b6981a71e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.145+0000 7f6753400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.145+0000 7f6753400640 1 -- 172.21.15.40:0/3178391696 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764052060 con 0x563f50acade0 2024-04-27T00:07:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.145+0000 7f6753400640 1 -- 172.21.15.40:0/3178391696 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f6764003b00 con 0x563f50acade0 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.145+0000 7f677b5b5740 1 -- 172.21.15.40:0/3178391696 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6764040070 con 0x563f50acade0 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.145+0000 7f6753400640 1 -- 172.21.15.40:0/3178391696 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6764059050 con 0x563f50acade0 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.145+0000 7f677b5b5740 10 client.15720 fetch_fsmap learned FSMap version 24 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.145+0000 7f677b5b5740 1 -- 172.21.15.40:0/3178391696 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563f50741210 con 0x563f50acade0 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.145+0000 7f6753400640 1 -- 172.21.15.40:0/3178391696 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6764040070 con 0x563f50acade0 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.146+0000 7f6753400640 1 -- 172.21.15.40:0/3178391696 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6764045600 con 0x563f50acade0 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.146+0000 7f677b5b5740 10 client.15720 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.146+0000 7f677b5b5740 10 client.15720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.146+0000 7f677b5b5740 20 client.15720 populate_metadata read hostname 'smithi040' 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.146+0000 7f677b5b5740 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563f50ae0590 0x563f50ae2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.146+0000 7f677b5b5740 4 client.15720 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.146+0000 7f677b5b5740 1 -- 172.21.15.40:0/3178391696 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f50a3cf90 con 0x563f50ae0590 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.146+0000 7f6775400640 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563f50ae0590 0x563f50ae2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.147+0000 7f6775400640 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563f50ae0590 0x563f50ae2980 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=941a043d1fb2d417 server_cookie=87708dbebf13cb09 in_seq=0 out_seq=0 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.147+0000 7f6753400640 10 client.15720 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f6753400640 1 -- 172.21.15.40:0/3178391696 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f50a3cf90 con 0x563f50ae0590 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f6753400640 10 client.15720 handle_command_reply: tid=0 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f677b5b5740 1 client.15720 shutdown 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f677b5b5740 1 -- 172.21.15.40:0/3178391696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acade0 msgr2=0x563f50ad4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f677b5b5740 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acade0 0x563f50ad4f40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f676402a450 tx=0x7f6764004550 comp rx=0 tx=0).stop 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f677b5b5740 1 -- 172.21.15.40:0/3178391696 shutdown_connections 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f677b5b5740 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563f50ae0590 0x563f50ae2980 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f677b5b5740 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50ad0940 0x563f50adc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f677b5b5740 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acb7d0 0x563f50ad5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f677b5b5740 1 --2- 172.21.15.40:0/3178391696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acade0 0x563f50ad4f40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f677b5b5740 1 -- 172.21.15.40:0/3178391696 >> 172.21.15.40:0/3178391696 conn(0x563f50ac6450 msgr2=0x563f50ac5650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f6753400640 0 client.15720 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f677b5b5740 1 -- 172.21.15.40:0/3178391696 shutdown_connections 2024-04-27T00:07:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f677b5b5740 1 -- 172.21.15.40:0/3178391696 wait complete. 2024-04-27T00:07:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.148+0000 7f677b5b5740 20 client.15720 trim_cache size 0 max 16384 2024-04-27T00:07:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f677b5b5740 1 Processor -- start 2024-04-27T00:07:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f677b5b5740 1 -- start start 2024-04-27T00:07:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f677b5b5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acade0 0x563f50ac98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f677b5b5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 0x563f50adc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f677b5b5740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50ad0940 0x563f50ad4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f677b5b5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f67701afdb0 con 0x563f50acb7d0 2024-04-27T00:07:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f677b5b5740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f67701ad680 con 0x563f50ad0940 2024-04-27T00:07:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f677b5b5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f67701ab4d0 con 0x563f50acade0 2024-04-27T00:07:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f6775400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 0x563f50adc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f6775e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acade0 0x563f50ac98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f6775400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 0x563f50adc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36402/0 (socket says 172.21.15.40:36402) 2024-04-27T00:07:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f6775400640 1 -- 172.21.15.40:0/3355074960 learned_addr learned my addr 172.21.15.40:0/3355074960 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.165+0000 7f6776800640 1 --2- 172.21.15.40:0/3355074960 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50ad0940 0x563f50ad4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.166+0000 7f6775e00640 1 -- 172.21.15.40:0/3355074960 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50ad0940 msgr2=0x563f50ad4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.166+0000 7f6775e00640 1 --2- 172.21.15.40:0/3355074960 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50ad0940 0x563f50ad4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.166+0000 7f6775e00640 1 -- 172.21.15.40:0/3355074960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 msgr2=0x563f50adc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.166+0000 7f6775e00640 1 --2- 172.21.15.40:0/3355074960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 0x563f50adc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.166+0000 7f6775e00640 1 -- 172.21.15.40:0/3355074960 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f5080d620 con 0x563f50acade0 2024-04-27T00:07:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.166+0000 7f6775400640 1 --2- 172.21.15.40:0/3355074960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 0x563f50adc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.166+0000 7f6775e00640 1 --2- 172.21.15.40:0/3355074960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acade0 0x563f50ac98e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f676402a930 tx=0x7f67640314a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6a73a51bf314831 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.166+0000 7f675e600640 1 -- 172.21.15.40:0/3355074960 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764052070 con 0x563f50acade0 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.166+0000 7f675e600640 1 -- 172.21.15.40:0/3355074960 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6764040040 con 0x563f50acade0 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.166+0000 7f675e600640 1 -- 172.21.15.40:0/3355074960 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764035070 con 0x563f50acade0 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.167+0000 7f677b5b5740 1 -- 172.21.15.40:0/3355074960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acade0 msgr2=0x563f50ac98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.167+0000 7f677b5b5740 1 --2- 172.21.15.40:0/3355074960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acade0 0x563f50ac98e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f676402a930 tx=0x7f67640314a0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.167+0000 7f677b5b5740 1 -- 172.21.15.40:0/3355074960 shutdown_connections 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.167+0000 7f677b5b5740 1 --2- 172.21.15.40:0/3355074960 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50ad0940 0x563f50ad4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.167+0000 7f677b5b5740 1 --2- 172.21.15.40:0/3355074960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50acb7d0 0x563f50adc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.167+0000 7f677b5b5740 1 --2- 172.21.15.40:0/3355074960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acade0 0x563f50ac98e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.167+0000 7f677b5b5740 1 -- 172.21.15.40:0/3355074960 >> 172.21.15.40:0/3355074960 conn(0x563f50ac6450 msgr2=0x563f50ac91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.167+0000 7f677b5b5740 1 -- 172.21.15.40:0/3355074960 shutdown_connections 2024-04-27T00:07:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.167+0000 7f677b5b5740 1 -- 172.21.15.40:0/3355074960 wait complete. 2024-04-27T00:07:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.168+0000 7f677b5b5740 1 -- start start 2024-04-27T00:07:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.168+0000 7f677b5b5740 1 Processor -- start 2024-04-27T00:07:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.168+0000 7f677b5b5740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50acade0 0x563f50ad96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.168+0000 7f677b5b5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acb7d0 0x563f50ad9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.168+0000 7f677b5b5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50ad0940 0x563f50adab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.168+0000 7f677b5b5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f67701abb40 con 0x563f50ad0940 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.168+0000 7f677b5b5740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f67701acfb0 con 0x563f50acade0 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.168+0000 7f677b5b5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f67701ac210 con 0x563f50acb7d0 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.168+0000 7f677b5b5740 10 client.0 _ll_register_callbacks cb 0x7f6764009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6775400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acb7d0 0x563f50ad9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6776800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50ad0940 0x563f50adab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6775400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acb7d0 0x563f50ad9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:57220/0 (socket says 172.21.15.40:57220) 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6775400640 1 -- 172.21.15.40:0/106521778 learned_addr learned my addr 172.21.15.40:0/106521778 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6775e00640 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50acade0 0x563f50ad96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6776800640 1 -- 172.21.15.40:0/106521778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acb7d0 msgr2=0x563f50ad9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6776800640 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acb7d0 0x563f50ad9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6776800640 1 -- 172.21.15.40:0/106521778 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50acade0 msgr2=0x563f50ad96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6776800640 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50acade0 0x563f50ad96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6776800640 1 -- 172.21.15.40:0/106521778 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f509e9cb0 con 0x563f50ad0940 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6775400640 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acb7d0 0x563f50ad9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6775e00640 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50acade0 0x563f50ad96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6776800640 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50ad0940 0x563f50adab40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f676c0ce4a0 tx=0x7f676c09c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b671e37106121f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.169+0000 7f6753400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.170+0000 7f6753400640 1 -- 172.21.15.40:0/106521778 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c005510 con 0x563f50ad0940 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.170+0000 7f6753400640 1 -- 172.21.15.40:0/106521778 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f676c09ced0 con 0x563f50ad0940 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.170+0000 7f677b5b5740 1 -- 172.21.15.40:0/106521778 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6764040040 con 0x563f50ad0940 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.170+0000 7f6753400640 1 -- 172.21.15.40:0/106521778 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f676c0a1020 con 0x563f50ad0940 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.170+0000 7f677b5b5740 10 client.15726 fetch_fsmap learned FSMap version 24 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.170+0000 7f677b5b5740 1 -- 172.21.15.40:0/106521778 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563f50adc2e0 con 0x563f50ad0940 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.170+0000 7f6753400640 1 -- 172.21.15.40:0/106521778 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6764040040 con 0x563f50ad0940 2024-04-27T00:07:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.170+0000 7f6753400640 1 -- 172.21.15.40:0/106521778 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f676c0989c0 con 0x563f50ad0940 2024-04-27T00:07:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.170+0000 7f677b5b5740 10 client.15726 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:07:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.170+0000 7f677b5b5740 10 client.15726 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:07:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.170+0000 7f677b5b5740 20 client.15726 populate_metadata read hostname 'smithi040' 2024-04-27T00:07:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.171+0000 7f677b5b5740 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563f50b46de0 0x563f50b49200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.171+0000 7f677b5b5740 4 client.15726 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:07:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.171+0000 7f677b5b5740 1 -- 172.21.15.40:0/106521778 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563f50a36e00 con 0x563f50b46de0 2024-04-27T00:07:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.171+0000 7f6775e00640 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563f50b46de0 0x563f50b49200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.171+0000 7f6775e00640 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563f50b46de0 0x563f50b49200 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1396f1b767531b4a server_cookie=e9ab458cb01cc1ef in_seq=0 out_seq=0 2024-04-27T00:07:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.171+0000 7f6753400640 10 client.15726 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:32.188 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.187+0000 7f6753400640 1 -- 172.21.15.40:0/106521778 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563f50a36e00 con 0x563f50b46de0 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.187+0000 7f6753400640 10 client.15726 handle_command_reply: tid=0 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.187+0000 7f677b5b5740 1 client.15726 shutdown 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.187+0000 7f677b5b5740 1 -- 172.21.15.40:0/106521778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50ad0940 msgr2=0x563f50adab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.187+0000 7f677b5b5740 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50ad0940 0x563f50adab40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f676c0ce4a0 tx=0x7f676c09c690 comp rx=0 tx=0).stop 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.187+0000 7f677b5b5740 1 -- 172.21.15.40:0/106521778 shutdown_connections 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.187+0000 7f677b5b5740 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563f50b46de0 0x563f50b49200 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.187+0000 7f677b5b5740 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563f50ad0940 0x563f50adab40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.187+0000 7f677b5b5740 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563f50acb7d0 0x563f50ad9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.187+0000 7f6753400640 0 client.15726 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.187+0000 7f677b5b5740 1 --2- 172.21.15.40:0/106521778 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563f50acade0 0x563f50ad96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.187+0000 7f677b5b5740 1 -- 172.21.15.40:0/106521778 >> 172.21.15.40:0/106521778 conn(0x563f50ac6450 msgr2=0x563f50ac4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.188+0000 7f677b5b5740 1 -- 172.21.15.40:0/106521778 shutdown_connections 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.188+0000 7f677b5b5740 1 -- 172.21.15.40:0/106521778 wait complete. 2024-04-27T00:07:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.188+0000 7f677b5b5740 20 client.15726 trim_cache size 0 max 16384 2024-04-27T00:07:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:07:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:07:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:07:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:07:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.189+0000 7f6777200640 1 -- 172.21.15.40:0/3022898348 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f67580a3c30 msgr2=0x7f67580a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.189+0000 7f6777200640 1 --2- 172.21.15.40:0/3022898348 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f67580a3c30 0x7f67580a60f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x563f50acc030 tx=0x7f6760004730 comp rx=0 tx=0).stop 2024-04-27T00:07:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.189+0000 7f6777200640 1 -- 172.21.15.40:0/3022898348 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67701084b0 msgr2=0x7f67701a2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.189+0000 7f6777200640 1 --2- 172.21.15.40:0/3022898348 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67701084b0 0x7f67701a2240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f676c00eb10 tx=0x7f676c00c2d0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.189+0000 7f6777200640 1 -- 172.21.15.40:0/3022898348 shutdown_connections 2024-04-27T00:07:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.189+0000 7f6777200640 1 --2- 172.21.15.40:0/3022898348 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f67701084b0 0x7f67701a2240 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.189+0000 7f6777200640 1 --2- 172.21.15.40:0/3022898348 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f67580a3c30 0x7f67580a60f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.189+0000 7f6777200640 1 --2- 172.21.15.40:0/3022898348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6770105b80 0x7f677019b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.189+0000 7f6777200640 1 --2- 172.21.15.40:0/3022898348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770103250 0x7f677019aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.189+0000 7f6777200640 1 -- 172.21.15.40:0/3022898348 >> 172.21.15.40:0/3022898348 conn(0x7f67700fcc60 msgr2=0x7f6770102810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.189+0000 7f6777200640 1 -- 172.21.15.40:0/3022898348 shutdown_connections 2024-04-27T00:07:32.192 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:32.189+0000 7f6777200640 1 -- 172.21.15.40:0/3022898348 wait complete. 2024-04-27T00:07:32.544 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.545 INFO:tasks.workunit.client.0.smithi040.stdout:189/600: open fd = 3 2024-04-27T00:07:32.545 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.545 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.545 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.545 INFO:tasks.workunit.client.0.smithi040.stdout:190/600: open fd = 3 2024-04-27T00:07:32.545 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.545 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.545 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.545 INFO:tasks.workunit.client.0.smithi040.stdout:191/600: open fd = 3 2024-04-27T00:07:32.545 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.545 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.545 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:192/600: open fd = 3 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:193/600: open fd = 3 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:194/600: open fd = 3 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:195/600: open fd = 3 2024-04-27T00:07:32.546 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:196/600: open fd = 3 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:197/600: open fd = 3 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:198/600: open fd = 3 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.547 INFO:tasks.workunit.client.0.smithi040.stdout:199/600: open fd = 3 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:200/600: open fd = 3 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:201/600: open fd = 3 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:202/600: open fd = 3 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.548 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.549 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.549 INFO:tasks.workunit.client.0.smithi040.stdout:203/600: open fd = 3 2024-04-27T00:07:32.549 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.549 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.549 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.550 INFO:tasks.workunit.client.0.smithi040.stdout:204/600: open fd = 3 2024-04-27T00:07:32.550 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.550 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.550 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.550 INFO:tasks.workunit.client.0.smithi040.stdout:205/600: open fd = 3 2024-04-27T00:07:32.550 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.550 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.550 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.550 INFO:tasks.workunit.client.0.smithi040.stdout:206/600: open fd = 3 2024-04-27T00:07:32.550 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.550 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:207/600: open fd = 3 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:208/600: open fd = 3 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:209/600: open fd = 3 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:210/600: open fd = 3 2024-04-27T00:07:32.551 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:211/600: open fd = 3 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:212/600: open fd = 3 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:213/600: open fd = 3 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.552 INFO:tasks.workunit.client.0.smithi040.stdout:214/600: open fd = 3 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:215/600: open fd = 3 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:216/600: open fd = 3 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:217/600: open fd = 3 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.553 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:218/600: open fd = 3 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:219/600: open fd = 3 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:220/600: open fd = 3 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:221/600: open fd = 3 2024-04-27T00:07:32.554 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:222/600: open fd = 3 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:223/600: open fd = 3 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:224/600: open fd = 3 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.555 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:225/600: open fd = 3 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:226/600: open fd = 3 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:227/600: open fd = 3 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:228/600: open fd = 3 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.556 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:229/600: open fd = 3 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:230/600: open fd = 3 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:231/600: open fd = 3 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:232/600: open fd = 3 2024-04-27T00:07:32.557 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:233/600: open fd = 3 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:234/600: open fd = 3 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:235/600: open fd = 3 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.558 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:236/600: open fd = 3 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:237/600: open fd = 3 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:238/600: open fd = 3 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:239/600: open fd = 3 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.559 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:240/600: open fd = 3 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:241/600: open fd = 3 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:242/600: open fd = 3 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.560 INFO:tasks.workunit.client.0.smithi040.stdout:243/600: open fd = 3 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:244/600: open fd = 3 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:245/600: open fd = 3 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:246/600: open fd = 3 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.561 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:247/600: open fd = 3 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:248/600: open fd = 3 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:249/600: open fd = 3 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:250/600: open fd = 3 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.562 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:07:32.563 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:32 smithi080 ceph-mon[32362]: pgmap v587: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-27T00:07:32.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:32 smithi103 ceph-mon[32952]: pgmap v587: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-27T00:07:32.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:32 smithi040 ceph-mon[28095]: pgmap v587: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 13 op/s 2024-04-27T00:07:34.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:34 smithi080 ceph-mon[32362]: pgmap v588: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:07:34.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:34 smithi103 ceph-mon[32952]: pgmap v588: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:07:34.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:34 smithi040 ceph-mon[28095]: pgmap v588: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:07:36.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:36 smithi080 ceph-mon[32362]: pgmap v589: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:07:36.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:36 smithi103 ceph-mon[32952]: pgmap v589: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:07:36.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:36 smithi040 ceph-mon[28095]: pgmap v589: 129 pgs: 129 active+clean; 3.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:07:38.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:38 smithi080 ceph-mon[32362]: pgmap v590: 129 pgs: 129 active+clean; 3.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:38.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:38 smithi103 ceph-mon[32952]: pgmap v590: 129 pgs: 129 active+clean; 3.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:38.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:38 smithi040 ceph-mon[28095]: pgmap v590: 129 pgs: 129 active+clean; 3.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:40.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:40 smithi103 ceph-mon[32952]: pgmap v591: 129 pgs: 129 active+clean; 3.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:40.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:40 smithi040 ceph-mon[28095]: pgmap v591: 129 pgs: 129 active+clean; 3.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:40.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:40 smithi080 ceph-mon[32362]: pgmap v591: 129 pgs: 129 active+clean; 3.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:42.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:42 smithi103 ceph-mon[32952]: pgmap v592: 129 pgs: 129 active+clean; 3.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:42.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:42 smithi040 ceph-mon[28095]: pgmap v592: 129 pgs: 129 active+clean; 3.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:42.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:42 smithi080 ceph-mon[32362]: pgmap v592: 129 pgs: 129 active+clean; 3.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:44.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:44 smithi103 ceph-mon[32952]: pgmap v593: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:07:44.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:44 smithi040 ceph-mon[28095]: pgmap v593: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:07:44.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:44 smithi080 ceph-mon[32362]: pgmap v593: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:07:45.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.341+0000 7f27a9000640 1 -- 172.21.15.40:0/463327837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a41028a0 msgr2=0x7f27a4102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:45.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.341+0000 7f27a9000640 1 --2- 172.21.15.40:0/463327837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a41028a0 0x7f27a4102d20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f278c009860 tx=0x7f278c02a340 comp rx=0 tx=0).stop 2024-04-27T00:07:45.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.342+0000 7f27a9000640 1 -- 172.21.15.40:0/463327837 shutdown_connections 2024-04-27T00:07:45.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.342+0000 7f27a9000640 1 --2- 172.21.15.40:0/463327837 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a4103260 0x7f27a4109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.342+0000 7f27a9000640 1 --2- 172.21.15.40:0/463327837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a41028a0 0x7f27a4102d20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.342+0000 7f27a9000640 1 --2- 172.21.15.40:0/463327837 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a41016b0 0x7f27a4101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.342+0000 7f27a9000640 1 -- 172.21.15.40:0/463327837 >> 172.21.15.40:0/463327837 conn(0x7f27a40fcc20 msgr2=0x7f27a40ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:45.342 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.342+0000 7f27a9000640 1 -- 172.21.15.40:0/463327837 shutdown_connections 2024-04-27T00:07:45.343 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.342+0000 7f27a9000640 1 -- 172.21.15.40:0/463327837 wait complete. 2024-04-27T00:07:45.344 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.343+0000 7f27a9000640 1 Processor -- start 2024-04-27T00:07:45.344 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.343+0000 7f27a9000640 1 -- start start 2024-04-27T00:07:45.344 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.344+0000 7f27a9000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a41016b0 0x7f27a419acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.344+0000 7f27a9000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a41028a0 0x7f27a419b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.344+0000 7f27a9000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a4103260 0x7f27a41a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.344+0000 7f27a9000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27a410e300 con 0x7f27a41028a0 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.344+0000 7f27a9000640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f27a410e180 con 0x7f27a4103260 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.344+0000 7f27a9000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f27a410e480 con 0x7f27a41016b0 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.344+0000 7f279aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a41028a0 0x7f27a419b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.344+0000 7f27a3400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a41016b0 0x7f27a419acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.344+0000 7f279aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a41028a0 0x7f27a419b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49188/0 (socket says 172.21.15.40:49188) 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.344+0000 7f279aa00640 1 -- 172.21.15.40:0/1627502566 learned_addr learned my addr 172.21.15.40:0/1627502566 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.344+0000 7f27a3e00640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a4103260 0x7f27a41a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.345+0000 7f27a3e00640 1 -- 172.21.15.40:0/1627502566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a41016b0 msgr2=0x7f27a419acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.345+0000 7f27a3e00640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a41016b0 0x7f27a419acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.345 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.345+0000 7f27a3e00640 1 -- 172.21.15.40:0/1627502566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a41028a0 msgr2=0x7f27a419b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:45.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.345+0000 7f27a3e00640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a41028a0 0x7f27a419b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.345+0000 7f27a3e00640 1 -- 172.21.15.40:0/1627502566 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a41a2870 con 0x7f27a4103260 2024-04-27T00:07:45.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.345+0000 7f279aa00640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a41028a0 0x7f27a419b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:07:45.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.345+0000 7f27a3e00640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a4103260 0x7f27a41a21d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f279c00e970 tx=0x7f279c00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:45.346 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.345+0000 7f27a0c00640 1 -- 172.21.15.40:0/1627502566 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c00ccc0 con 0x7f27a4103260 2024-04-27T00:07:45.347 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.345+0000 7f27a0c00640 1 -- 172.21.15.40:0/1627502566 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f279c005020 con 0x7f27a4103260 2024-04-27T00:07:45.347 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.345+0000 7f27a0c00640 1 -- 172.21.15.40:0/1627502566 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c004a00 con 0x7f27a4103260 2024-04-27T00:07:45.347 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.345+0000 7f27a9000640 1 -- 172.21.15.40:0/1627502566 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a41a2b60 con 0x7f27a4103260 2024-04-27T00:07:45.347 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.345+0000 7f27a9000640 1 -- 172.21.15.40:0/1627502566 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a41a3060 con 0x7f27a4103260 2024-04-27T00:07:45.348 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.349+0000 7f27a0c00640 1 -- 172.21.15.40:0/1627502566 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f279c004ba0 con 0x7f27a4103260 2024-04-27T00:07:45.348 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.349+0000 7f27ad3ec740 1 Processor -- start 2024-04-27T00:07:45.348 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.349+0000 7f27ad3ec740 1 -- start start 2024-04-27T00:07:45.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.349+0000 7f27ad3ec740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76fcde0 0x55eda76fd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.349+0000 7f27ad3ec740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda77022d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.350 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.349+0000 7f27ad3ec740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda7702940 0x55eda7704dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.349+0000 7f27ad3ec740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55eda7661380 con 0x55eda7702940 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.349+0000 7f27ad3ec740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55eda75fdc30 con 0x55eda76fcde0 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.349+0000 7f27ad3ec740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55eda765d340 con 0x55eda76fd7d0 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.349+0000 7f279aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda7702940 0x55eda7704dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.350+0000 7f279aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda7702940 0x55eda7704dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49198/0 (socket says 172.21.15.40:49198) 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.350+0000 7f279aa00640 1 -- 172.21.15.40:0/816554229 learned_addr learned my addr 172.21.15.40:0/816554229 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.350+0000 7f279aa00640 1 --2- 172.21.15.40:0/816554229 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76fcde0 0x55eda76fd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.350+0000 7f27a3400640 1 --2- 172.21.15.40:0/816554229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda77022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.350+0000 7f279aa00640 1 -- 172.21.15.40:0/816554229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 msgr2=0x55eda77022d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.350+0000 7f279aa00640 1 --2- 172.21.15.40:0/816554229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda77022d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.350+0000 7f279aa00640 1 -- 172.21.15.40:0/816554229 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76fcde0 msgr2=0x55eda76fd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.350+0000 7f279aa00640 1 --2- 172.21.15.40:0/816554229 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76fcde0 0x55eda76fd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.350+0000 7f279aa00640 1 -- 172.21.15.40:0/816554229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eda761f940 con 0x55eda7702940 2024-04-27T00:07:45.351 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.350+0000 7f27a0c00640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f27880a3c30 0x7f27880a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.350+0000 7f279aa00640 1 --2- 172.21.15.40:0/816554229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda7702940 0x55eda7704dd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f278c005b60 tx=0x7f278c005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d75e0ce561f1440 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:45.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.351+0000 7f27a0c00640 1 -- 172.21.15.40:0/1627502566 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f279c0d0900 con 0x7f27a4103260 2024-04-27T00:07:45.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.351+0000 7f2798c00640 1 -- 172.21.15.40:0/816554229 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c037040 con 0x55eda7702940 2024-04-27T00:07:45.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.351+0000 7f2798c00640 1 -- 172.21.15.40:0/816554229 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f278c004030 con 0x55eda7702940 2024-04-27T00:07:45.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.351+0000 7f2798c00640 1 -- 172.21.15.40:0/816554229 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c032be0 con 0x55eda7702940 2024-04-27T00:07:45.352 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.351+0000 7f27a3400640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f27880a3c30 0x7f27880a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.351+0000 7f27ad3ec740 1 -- 172.21.15.40:0/816554229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda7702940 msgr2=0x55eda7704dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:45.355 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.351+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/816554229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda7702940 0x55eda7704dd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f278c005b60 tx=0x7f278c005a90 comp rx=0 tx=0).stop 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.352+0000 7f27ad3ec740 1 -- 172.21.15.40:0/816554229 shutdown_connections 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.352+0000 7f27a3400640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f27880a3c30 0x7f27880a60f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55eda76fdbd0 tx=0x7f2794010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.352+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/816554229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda7702940 0x55eda7704dd0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.352+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/816554229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda77022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.352+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/816554229 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76fcde0 0x55eda76fd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.352+0000 7f27ad3ec740 1 -- 172.21.15.40:0/816554229 >> 172.21.15.40:0/816554229 conn(0x55eda76f8450 msgr2=0x55eda76f8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.352+0000 7f27ad3ec740 1 -- 172.21.15.40:0/816554229 shutdown_connections 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.352+0000 7f27ad3ec740 1 -- 172.21.15.40:0/816554229 wait complete. 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.352+0000 7f27ad3ec740 1 -- start start 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.352+0000 7f27ad3ec740 1 Processor -- start 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.353+0000 7f27ad3ec740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 0x55eda77070b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.353+0000 7f27ad3ec740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda77075f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.353+0000 7f27ad3ec740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda7702940 0x55eda770e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.353+0000 7f27ad3ec740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55eda7705890 con 0x55eda76fcde0 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.353+0000 7f27ad3ec740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55eda75e8c60 con 0x55eda7702940 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.353+0000 7f27ad3ec740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55eda7705a10 con 0x55eda76fd7d0 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.353+0000 7f27ad3ec740 10 client.0 _ll_register_callbacks cb 0x55eda7614e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.353+0000 7f27a3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 0x55eda77070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.353+0000 7f279aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda77075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.353+0000 7f279aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda77075f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:58696/0 (socket says 172.21.15.40:58696) 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.353+0000 7f279aa00640 1 -- 172.21.15.40:0/2541014497 learned_addr learned my addr 172.21.15.40:0/2541014497 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.353+0000 7f27a3e00640 1 --2- 172.21.15.40:0/2541014497 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda7702940 0x55eda770e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.354+0000 7f27a3400640 1 -- 172.21.15.40:0/2541014497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 msgr2=0x55eda77075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.354+0000 7f27a3400640 1 --2- 172.21.15.40:0/2541014497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda77075f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.354+0000 7f27a3400640 1 -- 172.21.15.40:0/2541014497 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda7702940 msgr2=0x55eda770e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.354+0000 7f27a3400640 1 --2- 172.21.15.40:0/2541014497 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda7702940 0x55eda770e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.354+0000 7f27a3400640 1 -- 172.21.15.40:0/2541014497 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eda75f7510 con 0x55eda76fcde0 2024-04-27T00:07:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.354+0000 7f27a3400640 1 --2- 172.21.15.40:0/2541014497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 0x55eda77070b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55eda76fe240 tx=0x7f2794004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6e6e8fa6800264 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.354+0000 7f2791600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.354+0000 7f2791600640 1 -- 172.21.15.40:0/2541014497 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279400de60 con 0x55eda76fcde0 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.354+0000 7f2791600640 1 -- 172.21.15.40:0/2541014497 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f2794004b20 con 0x55eda76fcde0 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.354+0000 7f27ad3ec740 1 -- 172.21.15.40:0/2541014497 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f278c004030 con 0x55eda76fcde0 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.355+0000 7f2791600640 1 -- 172.21.15.40:0/2541014497 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2794004d20 con 0x55eda76fcde0 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.355+0000 7f27ad3ec740 10 client.15738 fetch_fsmap learned FSMap version 24 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.355+0000 7f27ad3ec740 1 -- 172.21.15.40:0/2541014497 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55eda7373210 con 0x55eda76fcde0 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.355+0000 7f2791600640 1 -- 172.21.15.40:0/2541014497 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f278c004030 con 0x55eda76fcde0 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.355+0000 7f2791600640 1 -- 172.21.15.40:0/2541014497 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f279400b040 con 0x55eda76fcde0 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.355+0000 7f27ad3ec740 10 client.15738 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.355+0000 7f27ad3ec740 10 client.15738 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.355+0000 7f27ad3ec740 20 client.15738 populate_metadata read hostname 'smithi040' 2024-04-27T00:07:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.355+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/2541014497 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55eda7712b80 0x55eda7714fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.355+0000 7f27ad3ec740 4 client.15738 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.355+0000 7f27ad3ec740 1 -- 172.21.15.40:0/2541014497 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eda766ef90 con 0x55eda7712b80 2024-04-27T00:07:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.356+0000 7f279aa00640 1 --2- 172.21.15.40:0/2541014497 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55eda7712b80 0x55eda7714fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.356+0000 7f279aa00640 1 --2- 172.21.15.40:0/2541014497 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55eda7712b80 0x55eda7714fb0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84f0afe317965486 server_cookie=d0d9d351d61d726c in_seq=0 out_seq=0 2024-04-27T00:07:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.356+0000 7f2791600640 10 client.15738 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.357+0000 7f2791600640 1 -- 172.21.15.40:0/2541014497 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eda766ef90 con 0x55eda7712b80 2024-04-27T00:07:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.357+0000 7f2791600640 10 client.15738 handle_command_reply: tid=0 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.357+0000 7f27ad3ec740 1 client.15738 shutdown 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.358+0000 7f27ad3ec740 1 -- 172.21.15.40:0/2541014497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 msgr2=0x55eda77070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.358+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/2541014497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 0x55eda77070b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55eda76fe240 tx=0x7f2794004ab0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.358+0000 7f27ad3ec740 1 -- 172.21.15.40:0/2541014497 shutdown_connections 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.358+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/2541014497 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55eda7712b80 0x55eda7714fb0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.358+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/2541014497 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda7702940 0x55eda770e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.358+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/2541014497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda77075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.358+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/2541014497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 0x55eda77070b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.358+0000 7f2791600640 0 client.15738 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.358+0000 7f27ad3ec740 1 -- 172.21.15.40:0/2541014497 >> 172.21.15.40:0/2541014497 conn(0x55eda76f8450 msgr2=0x55eda76f7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.358+0000 7f27ad3ec740 1 -- 172.21.15.40:0/2541014497 shutdown_connections 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.358+0000 7f27ad3ec740 1 -- 172.21.15.40:0/2541014497 wait complete. 2024-04-27T00:07:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.358+0000 7f27ad3ec740 20 client.15738 trim_cache size 0 max 16384 2024-04-27T00:07:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.374+0000 7f27ad3ec740 1 Processor -- start 2024-04-27T00:07:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f27ad3ec740 1 -- start start 2024-04-27T00:07:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f27ad3ec740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76f8450 0x55eda76fbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f27ad3ec740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 0x55eda770e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f27ad3ec740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda7706cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f27ad3ec740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27a41afe10 con 0x55eda76fcde0 2024-04-27T00:07:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f27ad3ec740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f27a41ad6e0 con 0x55eda76f8450 2024-04-27T00:07:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f27ad3ec740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f27a41ab530 con 0x55eda76fd7d0 2024-04-27T00:07:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f279aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 0x55eda770e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f27a3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda7706cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f27a3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda7706cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:58700/0 (socket says 172.21.15.40:58700) 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f279aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 0x55eda770e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49214/0 (socket says 172.21.15.40:49214) 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f27a3e00640 1 -- 172.21.15.40:0/3094780199 learned_addr learned my addr 172.21.15.40:0/3094780199 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.375+0000 7f27a3400640 1 --2- 172.21.15.40:0/3094780199 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76f8450 0x55eda76fbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.376+0000 7f27a3e00640 1 -- 172.21.15.40:0/3094780199 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76f8450 msgr2=0x55eda76fbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.376+0000 7f27a3e00640 1 --2- 172.21.15.40:0/3094780199 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76f8450 0x55eda76fbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.376+0000 7f27a3e00640 1 -- 172.21.15.40:0/3094780199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 msgr2=0x55eda770e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.376+0000 7f27a3e00640 1 --2- 172.21.15.40:0/3094780199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 0x55eda770e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.376+0000 7f27a3e00640 1 -- 172.21.15.40:0/3094780199 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eda743f620 con 0x55eda76fd7d0 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.376+0000 7f279aa00640 1 --2- 172.21.15.40:0/3094780199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 0x55eda770e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.376+0000 7f27a3e00640 1 --2- 172.21.15.40:0/3094780199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda7706cc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f279c09c3e0 tx=0x7f279c004e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b7a2e3fd6fe791d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.376+0000 7f2793e00640 1 -- 172.21.15.40:0/3094780199 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c0b5e60 con 0x55eda76fd7d0 2024-04-27T00:07:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.376+0000 7f2793e00640 1 -- 172.21.15.40:0/3094780199 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f279c0b2070 con 0x55eda76fd7d0 2024-04-27T00:07:45.378 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.376+0000 7f2793e00640 1 -- 172.21.15.40:0/3094780199 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c01c040 con 0x55eda76fd7d0 2024-04-27T00:07:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.377+0000 7f27ad3ec740 1 -- 172.21.15.40:0/3094780199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 msgr2=0x55eda7706cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.377+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/3094780199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda7706cc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f279c09c3e0 tx=0x7f279c004e90 comp rx=0 tx=0).stop 2024-04-27T00:07:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.377+0000 7f27ad3ec740 1 -- 172.21.15.40:0/3094780199 shutdown_connections 2024-04-27T00:07:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.377+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/3094780199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fd7d0 0x55eda7706cc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.377+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/3094780199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76fcde0 0x55eda770e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.377+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/3094780199 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76f8450 0x55eda76fbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.377+0000 7f27ad3ec740 1 -- 172.21.15.40:0/3094780199 >> 172.21.15.40:0/3094780199 conn(0x55eda7702410 msgr2=0x55eda76fb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.377+0000 7f27ad3ec740 1 -- 172.21.15.40:0/3094780199 shutdown_connections 2024-04-27T00:07:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.377+0000 7f27ad3ec740 1 -- 172.21.15.40:0/3094780199 wait complete. 2024-04-27T00:07:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.377+0000 7f27ad3ec740 1 -- start start 2024-04-27T00:07:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f27ad3ec740 1 Processor -- start 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f27ad3ec740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76f8450 0x55eda770bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f27ad3ec740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fcde0 0x55eda770c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f27ad3ec740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76fd7d0 0x55eda770d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f27ad3ec740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f27a41abba0 con 0x55eda76f8450 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f27ad3ec740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f27a41ad010 con 0x55eda76fd7d0 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f27ad3ec740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f27a41ac270 con 0x55eda76fcde0 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f27ad3ec740 10 client.0 _ll_register_callbacks cb 0x7f2794009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f27a3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76f8450 0x55eda770bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f27a3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76fd7d0 0x55eda770d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f27a3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76fd7d0 0x55eda770d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:46032/0 (socket says 172.21.15.40:46032) 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f27a3e00640 1 -- 172.21.15.40:0/116230875 learned_addr learned my addr 172.21.15.40:0/116230875 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.378+0000 7f279aa00640 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fcde0 0x55eda770c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f27a3e00640 1 -- 172.21.15.40:0/116230875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fcde0 msgr2=0x55eda770c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f27a3e00640 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fcde0 0x55eda770c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f27a3e00640 1 -- 172.21.15.40:0/116230875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76f8450 msgr2=0x55eda770bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f27a3e00640 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76f8450 0x55eda770bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f27a3e00640 1 -- 172.21.15.40:0/116230875 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eda761bcb0 con 0x55eda76fd7d0 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f279aa00640 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fcde0 0x55eda770c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f27a3e00640 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76fd7d0 0x55eda770d4b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f279c009230 tx=0x7f279c087120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a29548876346bffb server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f2791600640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f2791600640 1 -- 172.21.15.40:0/116230875 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c0b9030 con 0x55eda76fd7d0 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f27ad3ec740 1 -- 172.21.15.40:0/116230875 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f279c0b2070 con 0x55eda76fd7d0 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f2791600640 1 -- 172.21.15.40:0/116230875 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f279c0054f0 con 0x55eda76fd7d0 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f2791600640 1 -- 172.21.15.40:0/116230875 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f279c0b5e60 con 0x55eda76fd7d0 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f27ad3ec740 10 client.25859 fetch_fsmap learned FSMap version 24 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.379+0000 7f27ad3ec740 1 -- 172.21.15.40:0/116230875 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55eda7779270 con 0x55eda76fd7d0 2024-04-27T00:07:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.380+0000 7f2791600640 1 -- 172.21.15.40:0/116230875 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f279c0b2070 con 0x55eda76fd7d0 2024-04-27T00:07:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.380+0000 7f2791600640 1 -- 172.21.15.40:0/116230875 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f279c0983a0 con 0x55eda76fd7d0 2024-04-27T00:07:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.380+0000 7f27ad3ec740 10 client.25859 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:07:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.380+0000 7f27ad3ec740 10 client.25859 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:07:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.380+0000 7f27ad3ec740 20 client.25859 populate_metadata read hostname 'smithi040' 2024-04-27T00:07:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.380+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55eda7702940 0x55eda777bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.380+0000 7f27ad3ec740 4 client.25859 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:07:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.380+0000 7f27ad3ec740 1 -- 172.21.15.40:0/116230875 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eda7668e00 con 0x55eda7702940 2024-04-27T00:07:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.380+0000 7f27a3400640 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55eda7702940 0x55eda777bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.381+0000 7f27a3400640 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55eda7702940 0x55eda777bb60 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3645629b3184ce6b server_cookie=f3a2a8ad2959150d in_seq=0 out_seq=0 2024-04-27T00:07:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.381+0000 7f2791600640 10 client.25859 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f2791600640 1 -- 172.21.15.40:0/116230875 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eda7668e00 con 0x55eda7702940 2024-04-27T00:07:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f2791600640 10 client.25859 handle_command_reply: tid=0 2024-04-27T00:07:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f27ad3ec740 1 client.25859 shutdown 2024-04-27T00:07:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f27ad3ec740 1 -- 172.21.15.40:0/116230875 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76fd7d0 msgr2=0x55eda770d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76fd7d0 0x55eda770d4b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f279c009230 tx=0x7f279c087120 comp rx=0 tx=0).stop 2024-04-27T00:07:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f27ad3ec740 1 -- 172.21.15.40:0/116230875 shutdown_connections 2024-04-27T00:07:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55eda7702940 0x55eda777bb60 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55eda76fd7d0 0x55eda770d4b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55eda76fcde0 0x55eda770c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f27ad3ec740 1 --2- 172.21.15.40:0/116230875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eda76f8450 0x55eda770bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f27ad3ec740 1 -- 172.21.15.40:0/116230875 >> 172.21.15.40:0/116230875 conn(0x55eda7702410 msgr2=0x55eda76fb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:45.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f2791600640 0 client.25859 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.397+0000 7f27ad3ec740 1 -- 172.21.15.40:0/116230875 shutdown_connections 2024-04-27T00:07:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.398+0000 7f27ad3ec740 1 -- 172.21.15.40:0/116230875 wait complete. 2024-04-27T00:07:45.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.398+0000 7f27ad3ec740 20 client.25859 trim_cache size 0 max 16384 2024-04-27T00:07:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:07:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:07:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:07:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:07:45.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.398+0000 7f27a9000640 1 -- 172.21.15.40:0/1627502566 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f27880a3c30 msgr2=0x7f27880a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:45.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.398+0000 7f27a9000640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f27880a3c30 0x7f27880a60f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55eda76fdbd0 tx=0x7f2794010040 comp rx=0 tx=0).stop 2024-04-27T00:07:45.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.398+0000 7f27a9000640 1 -- 172.21.15.40:0/1627502566 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a4103260 msgr2=0x7f27a41a21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:45.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.398+0000 7f27a9000640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a4103260 0x7f27a41a21d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f279c00e970 tx=0x7f279c00ee30 comp rx=0 tx=0).stop 2024-04-27T00:07:45.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.399+0000 7f27a9000640 1 -- 172.21.15.40:0/1627502566 shutdown_connections 2024-04-27T00:07:45.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.399+0000 7f27a9000640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f27a4103260 0x7f27a41a21d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.399+0000 7f27a9000640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27a41028a0 0x7f27a419b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.399+0000 7f27a9000640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f27880a3c30 0x7f27880a60f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.399+0000 7f27a9000640 1 --2- 172.21.15.40:0/1627502566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f27a41016b0 0x7f27a419acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:45.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.399+0000 7f27a9000640 1 -- 172.21.15.40:0/1627502566 >> 172.21.15.40:0/1627502566 conn(0x7f27a40fcc20 msgr2=0x7f27a410a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:45.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.399+0000 7f27a9000640 1 -- 172.21.15.40:0/1627502566 shutdown_connections 2024-04-27T00:07:45.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:45.399+0000 7f27a9000640 1 -- 172.21.15.40:0/1627502566 wait complete. 2024-04-27T00:07:46.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:46 smithi103 ceph-mon[32952]: pgmap v594: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:46.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:46 smithi040 ceph-mon[28095]: pgmap v594: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:46.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:46 smithi080 ceph-mon[32362]: pgmap v594: 129 pgs: 129 active+clean; 3.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:47.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:47 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:07:47.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:47 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:07:47.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:47 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:07:48.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:48 smithi103 ceph-mon[32952]: pgmap v595: 129 pgs: 129 active+clean; 3.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:07:48.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:48.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:48.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:48.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:48 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:48.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:48 smithi040 ceph-mon[28095]: pgmap v595: 129 pgs: 129 active+clean; 3.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:07:48.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:48.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:48.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:48.618 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:48 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:48.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:48 smithi080 ceph-mon[32362]: pgmap v595: 129 pgs: 129 active+clean; 3.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:07:48.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:48.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:48.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:48.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:48 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:49 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi080 to 176.2M 2024-04-27T00:07:49.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:49 smithi103 ceph-mon[32952]: Unable to set osd_memory_target on smithi080 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-27T00:07:49.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:49 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi040 to 177.5M 2024-04-27T00:07:49.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:49 smithi103 ceph-mon[32952]: Unable to set osd_memory_target on smithi040 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-27T00:07:49.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:07:49.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:07:49.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:07:49.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:49 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi080 to 176.2M 2024-04-27T00:07:49.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:49 smithi040 ceph-mon[28095]: Unable to set osd_memory_target on smithi080 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-27T00:07:49.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:49 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi040 to 177.5M 2024-04-27T00:07:49.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:49 smithi040 ceph-mon[28095]: Unable to set osd_memory_target on smithi040 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-27T00:07:49.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:07:49.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:07:49.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:07:49.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:49 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi080 to 176.2M 2024-04-27T00:07:49.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:49 smithi080 ceph-mon[32362]: Unable to set osd_memory_target on smithi080 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-27T00:07:49.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:07:49.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:49 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi040 to 177.5M 2024-04-27T00:07:49.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:49 smithi080 ceph-mon[32362]: Unable to set osd_memory_target on smithi040 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-27T00:07:49.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:07:49.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:07:49.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:07:50.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:50 smithi040 ceph-mon[28095]: pgmap v596: 129 pgs: 129 active+clean; 3.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:07:51.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:50 smithi080 ceph-mon[32362]: pgmap v596: 129 pgs: 129 active+clean; 3.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:07:51.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:50 smithi103 ceph-mon[32952]: pgmap v596: 129 pgs: 129 active+clean; 3.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:07:52.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:52 smithi040 ceph-mon[28095]: pgmap v597: 129 pgs: 129 active+clean; 3.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:53.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:52 smithi080 ceph-mon[32362]: pgmap v597: 129 pgs: 129 active+clean; 3.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:53.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:52 smithi103 ceph-mon[32952]: pgmap v597: 129 pgs: 129 active+clean; 3.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:54.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:54 smithi040 ceph-mon[28095]: pgmap v598: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:07:55.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:54 smithi080 ceph-mon[32362]: pgmap v598: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:07:55.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:54 smithi103 ceph-mon[32952]: pgmap v598: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:07:56.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:56 smithi040 ceph-mon[28095]: pgmap v599: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:57.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:56 smithi080 ceph-mon[32362]: pgmap v599: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:57.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:56 smithi103 ceph-mon[32952]: pgmap v599: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:07:58.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:07:57 smithi040 ceph-mon[28095]: pgmap v600: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:58.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:07:57 smithi080 ceph-mon[32362]: pgmap v600: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:58.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:07:57 smithi103 ceph-mon[32952]: pgmap v600: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:07:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.547+0000 7ff12fe00640 1 -- 172.21.15.40:0/2766421469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff130108570 msgr2=0x7ff13010a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.547+0000 7ff12fe00640 1 --2- 172.21.15.40:0/2766421469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff130108570 0x7ff13010a960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff128009ca0 tx=0x7ff12802a8b0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.547+0000 7ff12fe00640 1 -- 172.21.15.40:0/2766421469 shutdown_connections 2024-04-27T00:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.547+0000 7ff12fe00640 1 --2- 172.21.15.40:0/2766421469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff130108570 0x7ff13010a960 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.547+0000 7ff12fe00640 1 --2- 172.21.15.40:0/2766421469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff130105c40 0x7ff130108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.547+0000 7ff12fe00640 1 --2- 172.21.15.40:0/2766421469 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff130103310 0x7ff130105700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.547+0000 7ff12fe00640 1 -- 172.21.15.40:0/2766421469 >> 172.21.15.40:0/2766421469 conn(0x7ff1300fcc60 msgr2=0x7ff1300ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.547+0000 7ff12fe00640 1 -- 172.21.15.40:0/2766421469 shutdown_connections 2024-04-27T00:07:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.547+0000 7ff12fe00640 1 -- 172.21.15.40:0/2766421469 wait complete. 2024-04-27T00:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.548+0000 7ff12fe00640 1 Processor -- start 2024-04-27T00:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.549+0000 7ff12fe00640 1 -- start start 2024-04-27T00:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.549+0000 7ff12fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff130103310 0x7ff13019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.549+0000 7ff12fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff130105c40 0x7ff13019b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.549+0000 7ff12fe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff130108570 0x7ff1301a2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.549+0000 7ff12fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff13010e360 con 0x7ff130105c40 2024-04-27T00:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.549+0000 7ff12fe00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7ff13010e1e0 con 0x7ff130108570 2024-04-27T00:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.549+0000 7ff12fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff13010e4e0 con 0x7ff130103310 2024-04-27T00:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.549+0000 7ff12ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff130103310 0x7ff13019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.549+0000 7ff12ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff130103310 0x7ff13019abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47952/0 (socket says 172.21.15.40:47952) 2024-04-27T00:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.549+0000 7ff12ea00640 1 -- 172.21.15.40:0/213338725 learned_addr learned my addr 172.21.15.40:0/213338725 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.549+0000 7ff12f400640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff130108570 0x7ff1301a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.549+0000 7ff12e000640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff130105c40 0x7ff13019b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.550+0000 7ff12ea00640 1 -- 172.21.15.40:0/213338725 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff130108570 msgr2=0x7ff1301a2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.550+0000 7ff12ea00640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff130108570 0x7ff1301a2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.550+0000 7ff12ea00640 1 -- 172.21.15.40:0/213338725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff130105c40 msgr2=0x7ff13019b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.550+0000 7ff12ea00640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff130105c40 0x7ff13019b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.550+0000 7ff12ea00640 1 -- 172.21.15.40:0/213338725 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1301a27f0 con 0x7ff130103310 2024-04-27T00:07:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.550+0000 7ff12ea00640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff130103310 0x7ff13019abd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff12000d950 tx=0x7ff12000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:58.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.550+0000 7ff127400640 1 -- 172.21.15.40:0/213338725 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120017070 con 0x7ff130103310 2024-04-27T00:07:58.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.551+0000 7ff127400640 1 -- 172.21.15.40:0/213338725 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff120009c30 con 0x7ff130103310 2024-04-27T00:07:58.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.551+0000 7ff127400640 1 -- 172.21.15.40:0/213338725 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12000f620 con 0x7ff130103310 2024-04-27T00:07:58.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.551+0000 7ff12fe00640 1 -- 172.21.15.40:0/213338725 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1301a2ae0 con 0x7ff130103310 2024-04-27T00:07:58.553 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.551+0000 7ff12fe00640 1 -- 172.21.15.40:0/213338725 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1301a3080 con 0x7ff130103310 2024-04-27T00:07:58.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.555+0000 7ff138645740 1 Processor -- start 2024-04-27T00:07:58.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.555+0000 7ff138645740 1 -- start start 2024-04-27T00:07:58.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff138645740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 0x5635fe570200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff138645740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe5707d0 0x5635fe5752d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff138645740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe575940 0x5635fe577dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.559 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff138645740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5635fe4d4380 con 0x5635fe56fde0 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff138645740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5635fe470c30 con 0x5635fe5707d0 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff138645740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5635fe4d0340 con 0x5635fe575940 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff127400640 1 -- 172.21.15.40:0/213338725 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff120010460 con 0x7ff130103310 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff12f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 0x5635fe570200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff12f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 0x5635fe570200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46562/0 (socket says 172.21.15.40:46562) 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff12f400640 1 -- 172.21.15.40:0/516106119 learned_addr learned my addr 172.21.15.40:0/516106119 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff12e000640 1 --2- 172.21.15.40:0/516106119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe575940 0x5635fe577dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff12ea00640 1 --2- 172.21.15.40:0/516106119 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe5707d0 0x5635fe5752d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff12f400640 1 -- 172.21.15.40:0/516106119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe575940 msgr2=0x5635fe577dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff12f400640 1 --2- 172.21.15.40:0/516106119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe575940 0x5635fe577dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff12f400640 1 -- 172.21.15.40:0/516106119 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe5707d0 msgr2=0x5635fe5752d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff12f400640 1 --2- 172.21.15.40:0/516106119 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe5707d0 0x5635fe5752d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff12f400640 1 -- 172.21.15.40:0/516106119 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635fe492940 con 0x5635fe56fde0 2024-04-27T00:07:58.561 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.556+0000 7ff12ea00640 1 --2- 172.21.15.40:0/516106119 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe5707d0 0x5635fe5752d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T00:07:58.561 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.557+0000 7ff12f400640 1 --2- 172.21.15.40:0/516106119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 0x5635fe570200 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff128009ca0 tx=0x7ff128009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2112be822722316 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:58.561 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.557+0000 7ff11fe00640 1 -- 172.21.15.40:0/516106119 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff128030900 con 0x5635fe56fde0 2024-04-27T00:07:58.561 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.557+0000 7ff11fe00640 1 -- 172.21.15.40:0/516106119 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff128036030 con 0x5635fe56fde0 2024-04-27T00:07:58.561 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.557+0000 7ff11fe00640 1 -- 172.21.15.40:0/516106119 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff128004520 con 0x5635fe56fde0 2024-04-27T00:07:58.561 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.558+0000 7ff127400640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff1140a3cb0 0x7ff1140a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.558+0000 7ff138645740 1 -- 172.21.15.40:0/516106119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 msgr2=0x5635fe570200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.558+0000 7ff138645740 1 --2- 172.21.15.40:0/516106119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 0x5635fe570200 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff128009ca0 tx=0x7ff128009a20 comp rx=0 tx=0).stop 2024-04-27T00:07:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.558+0000 7ff127400640 1 -- 172.21.15.40:0/213338725 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7ff1200d0430 con 0x7ff130103310 2024-04-27T00:07:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.558+0000 7ff12ea00640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff1140a3cb0 0x7ff1140a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.558+0000 7ff138645740 1 -- 172.21.15.40:0/516106119 shutdown_connections 2024-04-27T00:07:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.558+0000 7ff138645740 1 --2- 172.21.15.40:0/516106119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe575940 0x5635fe577dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.558+0000 7ff138645740 1 --2- 172.21.15.40:0/516106119 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe5707d0 0x5635fe5752d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.558+0000 7ff138645740 1 --2- 172.21.15.40:0/516106119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 0x5635fe570200 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.558+0000 7ff138645740 1 -- 172.21.15.40:0/516106119 >> 172.21.15.40:0/516106119 conn(0x5635fe56b450 msgr2=0x5635fe56b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:58.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.558+0000 7ff138645740 1 -- 172.21.15.40:0/516106119 shutdown_connections 2024-04-27T00:07:58.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.558+0000 7ff138645740 1 -- 172.21.15.40:0/516106119 wait complete. 2024-04-27T00:07:58.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.559+0000 7ff12ea00640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff1140a3cb0 0x7ff1140a6170 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5635fe571240 tx=0x7ff120010cb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:58.563 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.559+0000 7ff138645740 1 -- start start 2024-04-27T00:07:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.559+0000 7ff138645740 1 Processor -- start 2024-04-27T00:07:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.559+0000 7ff138645740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 0x5635fe579f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.559+0000 7ff138645740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe5707d0 0x5635fe57a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff138645740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe575940 0x5635fe5814c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff138645740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5635fe578890 con 0x5635fe56fde0 2024-04-27T00:07:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff138645740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5635fe45bc60 con 0x5635fe575940 2024-04-27T00:07:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff138645740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5635fe578a10 con 0x5635fe5707d0 2024-04-27T00:07:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff138645740 10 client.0 _ll_register_callbacks cb 0x5635fe487e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:07:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff12f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 0x5635fe579f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff12f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 0x5635fe579f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46564/0 (socket says 172.21.15.40:46564) 2024-04-27T00:07:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff12f400640 1 -- 172.21.15.40:0/655352255 learned_addr learned my addr 172.21.15.40:0/655352255 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff12e000640 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe5707d0 0x5635fe57a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff12f400640 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe575940 0x5635fe5814c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff12f400640 1 -- 172.21.15.40:0/655352255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe5707d0 msgr2=0x5635fe57a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff12f400640 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe5707d0 0x5635fe57a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff12f400640 1 -- 172.21.15.40:0/655352255 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe575940 msgr2=0x5635fe5814c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff12f400640 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe575940 0x5635fe5814c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.560+0000 7ff12f400640 1 -- 172.21.15.40:0/655352255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635fe46a510 con 0x5635fe56fde0 2024-04-27T00:07:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.561+0000 7ff12f400640 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 0x5635fe579f40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff128030db0 tx=0x7ff128030d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe36d5bb8b4235e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.561+0000 7ff11cc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:07:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.561+0000 7ff11cc00640 1 -- 172.21.15.40:0/655352255 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff128059060 con 0x5635fe56fde0 2024-04-27T00:07:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.561+0000 7ff11cc00640 1 -- 172.21.15.40:0/655352255 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7ff128004260 con 0x5635fe56fde0 2024-04-27T00:07:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.561+0000 7ff138645740 1 -- 172.21.15.40:0/655352255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff128036030 con 0x5635fe56fde0 2024-04-27T00:07:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.561+0000 7ff11cc00640 1 -- 172.21.15.40:0/655352255 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff128004460 con 0x5635fe56fde0 2024-04-27T00:07:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.561+0000 7ff138645740 10 client.15756 fetch_fsmap learned FSMap version 24 2024-04-27T00:07:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.561+0000 7ff138645740 1 -- 172.21.15.40:0/655352255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5635fe1e6210 con 0x5635fe56fde0 2024-04-27T00:07:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.562+0000 7ff11cc00640 1 -- 172.21.15.40:0/655352255 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff128036030 con 0x5635fe56fde0 2024-04-27T00:07:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.562+0000 7ff11cc00640 1 -- 172.21.15.40:0/655352255 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff128050070 con 0x5635fe56fde0 2024-04-27T00:07:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.562+0000 7ff138645740 10 client.15756 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:07:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.562+0000 7ff138645740 10 client.15756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:07:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.562+0000 7ff138645740 20 client.15756 populate_metadata read hostname 'smithi040' 2024-04-27T00:07:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.562+0000 7ff138645740 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5635fe585590 0x5635fe587980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.562+0000 7ff138645740 4 client.15756 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:07:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.562+0000 7ff138645740 1 -- 172.21.15.40:0/655352255 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635fe4e1f90 con 0x5635fe585590 2024-04-27T00:07:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.562+0000 7ff12e000640 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5635fe585590 0x5635fe587980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.563+0000 7ff12e000640 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5635fe585590 0x5635fe587980 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a40032a501fbc81c server_cookie=dea0762476d9840 in_seq=0 out_seq=0 2024-04-27T00:07:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.563+0000 7ff11cc00640 10 client.15756 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff11cc00640 1 -- 172.21.15.40:0/655352255 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5635fe4e1f90 con 0x5635fe585590 2024-04-27T00:07:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff11cc00640 10 client.15756 handle_command_reply: tid=0 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff138645740 1 client.15756 shutdown 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff138645740 1 -- 172.21.15.40:0/655352255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 msgr2=0x5635fe579f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff138645740 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 0x5635fe579f40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff128030db0 tx=0x7ff128030d00 comp rx=0 tx=0).stop 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff138645740 1 -- 172.21.15.40:0/655352255 shutdown_connections 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff138645740 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5635fe585590 0x5635fe587980 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff138645740 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe575940 0x5635fe5814c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff138645740 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe5707d0 0x5635fe57a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff138645740 1 --2- 172.21.15.40:0/655352255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe56fde0 0x5635fe579f40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff138645740 1 -- 172.21.15.40:0/655352255 >> 172.21.15.40:0/655352255 conn(0x5635fe56b450 msgr2=0x5635fe56a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff11cc00640 0 client.15756 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff138645740 1 -- 172.21.15.40:0/655352255 shutdown_connections 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff138645740 1 -- 172.21.15.40:0/655352255 wait complete. 2024-04-27T00:07:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.564+0000 7ff138645740 20 client.15756 trim_cache size 0 max 16384 2024-04-27T00:07:58.582 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.581+0000 7ff138645740 1 Processor -- start 2024-04-27T00:07:58.582 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.581+0000 7ff138645740 1 -- start start 2024-04-27T00:07:58.582 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.581+0000 7ff138645740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe56fde0 0x5635fe56eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.582 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.581+0000 7ff138645740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe5707d0 0x5635fe581450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.583 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.581+0000 7ff138645740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe575940 0x5635fe579b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.583 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.581+0000 7ff138645740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff1301afda0 con 0x5635fe575940 2024-04-27T00:07:58.583 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.581+0000 7ff138645740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7ff1301ad670 con 0x5635fe56fde0 2024-04-27T00:07:58.583 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.581+0000 7ff138645740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff1301ab4c0 con 0x5635fe5707d0 2024-04-27T00:07:58.584 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.582+0000 7ff12f400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe56fde0 0x5635fe56eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.584 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.582+0000 7ff12f400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe56fde0 0x5635fe56eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:38064/0 (socket says 172.21.15.40:38064) 2024-04-27T00:07:58.584 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.582+0000 7ff12f400640 1 -- 172.21.15.40:0/1124638499 learned_addr learned my addr 172.21.15.40:0/1124638499 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.582+0000 7ff12f400640 1 --2- 172.21.15.40:0/1124638499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe575940 0x5635fe579b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.582+0000 7ff12e000640 1 --2- 172.21.15.40:0/1124638499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe5707d0 0x5635fe581450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.582+0000 7ff12f400640 1 -- 172.21.15.40:0/1124638499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe5707d0 msgr2=0x5635fe581450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.582+0000 7ff12f400640 1 --2- 172.21.15.40:0/1124638499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe5707d0 0x5635fe581450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.582+0000 7ff12f400640 1 -- 172.21.15.40:0/1124638499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe575940 msgr2=0x5635fe579b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.582+0000 7ff12f400640 1 --2- 172.21.15.40:0/1124638499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe575940 0x5635fe579b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.582+0000 7ff12f400640 1 -- 172.21.15.40:0/1124638499 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635fe2b2620 con 0x5635fe56fde0 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.583+0000 7ff12f400640 1 --2- 172.21.15.40:0/1124638499 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe56fde0 0x5635fe56eb10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff128009430 tx=0x7ff128009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4c85ef02a83fe2f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.583+0000 7ff11f400640 1 -- 172.21.15.40:0/1124638499 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff128059030 con 0x5635fe56fde0 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.583+0000 7ff11f400640 1 -- 172.21.15.40:0/1124638499 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff128036070 con 0x5635fe56fde0 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.583+0000 7ff11f400640 1 -- 172.21.15.40:0/1124638499 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff128004600 con 0x5635fe56fde0 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.584+0000 7ff138645740 1 -- 172.21.15.40:0/1124638499 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe56fde0 msgr2=0x5635fe56eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.584+0000 7ff138645740 1 --2- 172.21.15.40:0/1124638499 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe56fde0 0x5635fe56eb10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff128009430 tx=0x7ff128009a20 comp rx=0 tx=0).stop 2024-04-27T00:07:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.584+0000 7ff138645740 1 -- 172.21.15.40:0/1124638499 shutdown_connections 2024-04-27T00:07:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.584+0000 7ff138645740 1 --2- 172.21.15.40:0/1124638499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe575940 0x5635fe579b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.584+0000 7ff138645740 1 --2- 172.21.15.40:0/1124638499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe5707d0 0x5635fe581450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.584+0000 7ff138645740 1 --2- 172.21.15.40:0/1124638499 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe56fde0 0x5635fe56eb10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.584+0000 7ff138645740 1 -- 172.21.15.40:0/1124638499 >> 172.21.15.40:0/1124638499 conn(0x5635fe56b450 msgr2=0x5635fe56e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.584+0000 7ff138645740 1 -- 172.21.15.40:0/1124638499 shutdown_connections 2024-04-27T00:07:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.584+0000 7ff138645740 1 -- 172.21.15.40:0/1124638499 wait complete. 2024-04-27T00:07:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.584+0000 7ff138645740 1 -- start start 2024-04-27T00:07:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.585+0000 7ff138645740 1 Processor -- start 2024-04-27T00:07:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.585+0000 7ff138645740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe56fde0 0x5635fe57e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.585+0000 7ff138645740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe5707d0 0x5635fe57e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.585+0000 7ff138645740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe575940 0x5635fe5e99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.585+0000 7ff138645740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff1301abb30 con 0x5635fe5707d0 2024-04-27T00:07:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.585+0000 7ff138645740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7ff1301acfa0 con 0x5635fe575940 2024-04-27T00:07:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.585+0000 7ff138645740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff1301ac200 con 0x5635fe56fde0 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.585+0000 7ff138645740 10 client.0 _ll_register_callbacks cb 0x7ff11800b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.585+0000 7ff12e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe5707d0 0x5635fe57e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.585+0000 7ff12e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe5707d0 0x5635fe57e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46572/0 (socket says 172.21.15.40:46572) 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.585+0000 7ff12e000640 1 -- 172.21.15.40:0/6432170 learned_addr learned my addr 172.21.15.40:0/6432170 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.585+0000 7ff12f400640 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe56fde0 0x5635fe57e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.586+0000 7ff12f400640 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe575940 0x5635fe5e99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.586+0000 7ff12e000640 1 -- 172.21.15.40:0/6432170 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe56fde0 msgr2=0x5635fe57e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.586+0000 7ff12e000640 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe56fde0 0x5635fe57e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.586+0000 7ff12e000640 1 -- 172.21.15.40:0/6432170 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe575940 msgr2=0x5635fe5e99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.586+0000 7ff12e000640 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe575940 0x5635fe5e99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.586+0000 7ff12e000640 1 -- 172.21.15.40:0/6432170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635fe48ecb0 con 0x5635fe5707d0 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.586+0000 7ff12f400640 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe575940 0x5635fe5e99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:07:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.586+0000 7ff12e000640 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe5707d0 0x5635fe57e600 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff11800b7e0 tx=0x7ff1180138f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2a516b81e7f647 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.586+0000 7ff11cc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.587+0000 7ff11cc00640 1 -- 172.21.15.40:0/6432170 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff118005220 con 0x5635fe5707d0 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.587+0000 7ff11cc00640 1 -- 172.21.15.40:0/6432170 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7ff1180053c0 con 0x5635fe5707d0 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.587+0000 7ff138645740 1 -- 172.21.15.40:0/6432170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff128036070 con 0x5635fe5707d0 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.587+0000 7ff11cc00640 1 -- 172.21.15.40:0/6432170 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff11802f050 con 0x5635fe5707d0 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.587+0000 7ff138645740 10 client.15762 fetch_fsmap learned FSMap version 24 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.587+0000 7ff138645740 1 -- 172.21.15.40:0/6432170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5635fe5eae80 con 0x5635fe5707d0 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.587+0000 7ff11cc00640 1 -- 172.21.15.40:0/6432170 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff128036070 con 0x5635fe5707d0 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.587+0000 7ff11cc00640 1 -- 172.21.15.40:0/6432170 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff118020350 con 0x5635fe5707d0 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.588+0000 7ff138645740 10 client.15762 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.588+0000 7ff138645740 10 client.15762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.588+0000 7ff138645740 20 client.15762 populate_metadata read hostname 'smithi040' 2024-04-27T00:07:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.588+0000 7ff138645740 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5635fe5eb8b0 0x5635fe5edcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:07:58.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.588+0000 7ff138645740 4 client.15762 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:07:58.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.588+0000 7ff138645740 1 -- 172.21.15.40:0/6432170 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5635fe4dbe00 con 0x5635fe5eb8b0 2024-04-27T00:07:58.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.588+0000 7ff12f400640 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5635fe5eb8b0 0x5635fe5edcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:07:58.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.589+0000 7ff12f400640 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5635fe5eb8b0 0x5635fe5edcd0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=234054001b94ef75 server_cookie=776364f75f6798c0 in_seq=0 out_seq=0 2024-04-27T00:07:58.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.589+0000 7ff11cc00640 10 client.15762 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.604+0000 7ff11cc00640 1 -- 172.21.15.40:0/6432170 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5635fe4dbe00 con 0x5635fe5eb8b0 2024-04-27T00:07:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.604+0000 7ff11cc00640 10 client.15762 handle_command_reply: tid=0 2024-04-27T00:07:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.604+0000 7ff138645740 1 client.15762 shutdown 2024-04-27T00:07:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.605+0000 7ff138645740 1 -- 172.21.15.40:0/6432170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe5707d0 msgr2=0x5635fe57e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.605+0000 7ff138645740 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe5707d0 0x5635fe57e600 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff11800b7e0 tx=0x7ff1180138f0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.605+0000 7ff138645740 1 -- 172.21.15.40:0/6432170 shutdown_connections 2024-04-27T00:07:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.605+0000 7ff138645740 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5635fe5eb8b0 0x5635fe5edcd0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.605+0000 7ff138645740 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5635fe575940 0x5635fe5e99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.605+0000 7ff138645740 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5635fe5707d0 0x5635fe57e600 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.605+0000 7ff11cc00640 0 client.15762 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:07:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.605+0000 7ff138645740 1 --2- 172.21.15.40:0/6432170 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635fe56fde0 0x5635fe57e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.605+0000 7ff138645740 1 -- 172.21.15.40:0/6432170 >> 172.21.15.40:0/6432170 conn(0x5635fe56b450 msgr2=0x5635fe579b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.605+0000 7ff138645740 1 -- 172.21.15.40:0/6432170 shutdown_connections 2024-04-27T00:07:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.605+0000 7ff138645740 1 -- 172.21.15.40:0/6432170 wait complete. 2024-04-27T00:07:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.605+0000 7ff138645740 20 client.15762 trim_cache size 0 max 16384 2024-04-27T00:07:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:07:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:07:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:07:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:07:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.606+0000 7ff12fe00640 1 -- 172.21.15.40:0/213338725 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff1140a3cb0 msgr2=0x7ff1140a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.606+0000 7ff12fe00640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff1140a3cb0 0x7ff1140a6170 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5635fe571240 tx=0x7ff120010cb0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.606+0000 7ff12fe00640 1 -- 172.21.15.40:0/213338725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff130103310 msgr2=0x7ff13019abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:07:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.606+0000 7ff12fe00640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff130103310 0x7ff13019abd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff12000d950 tx=0x7ff12000de10 comp rx=0 tx=0).stop 2024-04-27T00:07:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.606+0000 7ff12fe00640 1 -- 172.21.15.40:0/213338725 shutdown_connections 2024-04-27T00:07:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.606+0000 7ff12fe00640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7ff130108570 0x7ff1301a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.606+0000 7ff12fe00640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff130105c40 0x7ff13019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.606+0000 7ff12fe00640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7ff1140a3cb0 0x7ff1140a6170 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.606+0000 7ff12fe00640 1 --2- 172.21.15.40:0/213338725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff130103310 0x7ff13019abd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:07:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.606+0000 7ff12fe00640 1 -- 172.21.15.40:0/213338725 >> 172.21.15.40:0/213338725 conn(0x7ff1300fcc60 msgr2=0x7ff130102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:07:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.606+0000 7ff12fe00640 1 -- 172.21.15.40:0/213338725 shutdown_connections 2024-04-27T00:07:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:07:58.607+0000 7ff12fe00640 1 -- 172.21.15.40:0/213338725 wait complete. 2024-04-27T00:08:00.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:00 smithi080 ceph-mon[32362]: pgmap v601: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:08:00.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:00 smithi103 ceph-mon[32952]: pgmap v601: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:08:00.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:00 smithi040 ceph-mon[28095]: pgmap v601: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:08:02.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:02 smithi080 ceph-mon[32362]: pgmap v602: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:08:02.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:02 smithi103 ceph-mon[32952]: pgmap v602: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:08:02.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:02 smithi040 ceph-mon[28095]: pgmap v602: 129 pgs: 129 active+clean; 4.0 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:08:04.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:04 smithi080 ceph-mon[32362]: pgmap v603: 129 pgs: 129 active+clean; 4.1 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:08:04.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:04 smithi103 ceph-mon[32952]: pgmap v603: 129 pgs: 129 active+clean; 4.1 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:08:04.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:04 smithi040 ceph-mon[28095]: pgmap v603: 129 pgs: 129 active+clean; 4.1 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:08:06.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:06 smithi080 ceph-mon[32362]: pgmap v604: 129 pgs: 129 active+clean; 4.1 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:08:06.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:06 smithi103 ceph-mon[32952]: pgmap v604: 129 pgs: 129 active+clean; 4.1 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:08:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:06 smithi040 ceph-mon[28095]: pgmap v604: 129 pgs: 129 active+clean; 4.1 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:08:08.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:08 smithi103 ceph-mon[32952]: pgmap v605: 129 pgs: 129 active+clean; 4.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:08:08.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:08 smithi040 ceph-mon[28095]: pgmap v605: 129 pgs: 129 active+clean; 4.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:08:08.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:08 smithi080 ceph-mon[32362]: pgmap v605: 129 pgs: 129 active+clean; 4.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:08:10.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:10 smithi103 ceph-mon[32952]: pgmap v606: 129 pgs: 129 active+clean; 4.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:08:10.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:10 smithi040 ceph-mon[28095]: pgmap v606: 129 pgs: 129 active+clean; 4.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:08:10.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:10 smithi080 ceph-mon[32362]: pgmap v606: 129 pgs: 129 active+clean; 4.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:08:11.760 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.759+0000 7f54a2e00640 1 -- 172.21.15.40:0/154806423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f549c078ed0 msgr2=0x7f549c079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:11.760 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.759+0000 7f54a2e00640 1 --2- 172.21.15.40:0/154806423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f549c078ed0 0x7f549c079770 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f548c009860 tx=0x7f548c02a340 comp rx=0 tx=0).stop 2024-04-27T00:08:11.760 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.759+0000 7f54a2e00640 1 -- 172.21.15.40:0/154806423 shutdown_connections 2024-04-27T00:08:11.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.759+0000 7f54a2e00640 1 --2- 172.21.15.40:0/154806423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f549c078ed0 0x7f549c079770 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.759+0000 7f54a2e00640 1 --2- 172.21.15.40:0/154806423 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c078530 0x7f549c078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.759+0000 7f54a2e00640 1 --2- 172.21.15.40:0/154806423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c0772f0 0x7f549c077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.759+0000 7f54a2e00640 1 -- 172.21.15.40:0/154806423 >> 172.21.15.40:0/154806423 conn(0x7f549c100150 msgr2=0x7f549c102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:11.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.759+0000 7f54a2e00640 1 -- 172.21.15.40:0/154806423 shutdown_connections 2024-04-27T00:08:11.761 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.760+0000 7f54a2e00640 1 -- 172.21.15.40:0/154806423 wait complete. 2024-04-27T00:08:11.762 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.760+0000 7f54a2e00640 1 Processor -- start 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f54a2e00640 1 -- start start 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f54a2e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c0772f0 0x7f549c19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f54a2e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c078530 0x7f549c19f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f54a2e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f549c078ed0 0x7f549c1a6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f54a2e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f549c114b00 con 0x7f549c078ed0 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f54a2e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f549c114980 con 0x7f549c078530 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f54a2e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f549c114c80 con 0x7f549c0772f0 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f54a0e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f549c078ed0 0x7f549c1a6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f549b400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c078530 0x7f549c19f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f549be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c0772f0 0x7f549c19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f54a0e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f549c078ed0 0x7f549c1a6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59252/0 (socket says 172.21.15.40:59252) 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f549b400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c078530 0x7f549c19f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:39856/0 (socket says 172.21.15.40:39856) 2024-04-27T00:08:11.763 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f549be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c0772f0 0x7f549c19f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:41404/0 (socket says 172.21.15.40:41404) 2024-04-27T00:08:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.761+0000 7f54a0e00640 1 -- 172.21.15.40:0/1278061312 learned_addr learned my addr 172.21.15.40:0/1278061312 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.762+0000 7f549b400640 1 -- 172.21.15.40:0/1278061312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c0772f0 msgr2=0x7f549c19f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.762+0000 7f549b400640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c0772f0 0x7f549c19f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.762+0000 7f549b400640 1 -- 172.21.15.40:0/1278061312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f549c078ed0 msgr2=0x7f549c1a6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.762+0000 7f549b400640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f549c078ed0 0x7f549c1a6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.762+0000 7f549b400640 1 -- 172.21.15.40:0/1278061312 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549c1a6cd0 con 0x7f549c078530 2024-04-27T00:08:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.762+0000 7f54a0e00640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f549c078ed0 0x7f549c1a6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:08:11.764 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.762+0000 7f549b400640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c078530 0x7f549c19f640 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f548800e9f0 tx=0x7f548800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.762+0000 7f5498c00640 1 -- 172.21.15.40:0/1278061312 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548800ccc0 con 0x7f549c078530 2024-04-27T00:08:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.763+0000 7f5498c00640 1 -- 172.21.15.40:0/1278061312 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54880108e0 con 0x7f549c078530 2024-04-27T00:08:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.763+0000 7f5498c00640 1 -- 172.21.15.40:0/1278061312 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5488018880 con 0x7f549c078530 2024-04-27T00:08:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.763+0000 7f54a2e00640 1 -- 172.21.15.40:0/1278061312 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f549c1a6fc0 con 0x7f549c078530 2024-04-27T00:08:11.765 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.763+0000 7f54a2e00640 1 -- 172.21.15.40:0/1278061312 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f549c1a7480 con 0x7f549c078530 2024-04-27T00:08:11.766 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.766+0000 7f54a5c78740 1 Processor -- start 2024-04-27T00:08:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.766+0000 7f54a5c78740 1 -- start start 2024-04-27T00:08:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.766+0000 7f54a5c78740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53dbd0 0x55d27e53dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.766+0000 7f54a5c78740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 0x55d27e5430c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.766+0000 7f54a5c78740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e545bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f54a5c78740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d27e4a2380 con 0x55d27e543730 2024-04-27T00:08:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f54a5c78740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55d27e43ec30 con 0x55d27e53dbd0 2024-04-27T00:08:11.767 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f54a5c78740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d27e49e340 con 0x55d27e53e5c0 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f5498c00640 1 -- 172.21.15.40:0/1278061312 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5488010430 con 0x7f549c078530 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f54a0e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53dbd0 0x55d27e53dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f549be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 0x55d27e5430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f549be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 0x55d27e5430c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:41414/0 (socket says 172.21.15.40:41414) 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f549be00640 1 -- 172.21.15.40:0/632644624 learned_addr learned my addr 172.21.15.40:0/632644624 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f54a0e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53dbd0 0x55d27e53dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:39872/0 (socket says 172.21.15.40:39872) 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f549b400640 1 --2- 172.21.15.40:0/632644624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e545bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f549be00640 1 -- 172.21.15.40:0/632644624 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53dbd0 msgr2=0x55d27e53dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f549be00640 1 --2- 172.21.15.40:0/632644624 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53dbd0 0x55d27e53dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f549be00640 1 -- 172.21.15.40:0/632644624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 msgr2=0x55d27e545bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f549be00640 1 --2- 172.21.15.40:0/632644624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e545bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.767+0000 7f549be00640 1 -- 172.21.15.40:0/632644624 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d27e460940 con 0x55d27e53e5c0 2024-04-27T00:08:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.768+0000 7f549be00640 1 --2- 172.21.15.40:0/632644624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 0x55d27e5430c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f549400e580 tx=0x7f54940095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed827cc56d94e75a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.768+0000 7f5491600640 1 -- 172.21.15.40:0/632644624 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494003970 con 0x55d27e53e5c0 2024-04-27T00:08:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.768+0000 7f5491600640 1 -- 172.21.15.40:0/632644624 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f549400d8b0 con 0x55d27e53e5c0 2024-04-27T00:08:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.768+0000 7f5491600640 1 -- 172.21.15.40:0/632644624 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494004530 con 0x55d27e53e5c0 2024-04-27T00:08:11.773 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f54a5c78740 1 -- 172.21.15.40:0/632644624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 msgr2=0x55d27e5430c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:11.773 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f54a5c78740 1 --2- 172.21.15.40:0/632644624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 0x55d27e5430c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f549400e580 tx=0x7f54940095c0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.773 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f54a5c78740 1 -- 172.21.15.40:0/632644624 shutdown_connections 2024-04-27T00:08:11.773 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f54a5c78740 1 --2- 172.21.15.40:0/632644624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e545bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f54a5c78740 1 --2- 172.21.15.40:0/632644624 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 0x55d27e5430c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f54a5c78740 1 --2- 172.21.15.40:0/632644624 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53dbd0 0x55d27e53dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f54a5c78740 1 -- 172.21.15.40:0/632644624 >> 172.21.15.40:0/632644624 conn(0x55d27e539240 msgr2=0x55d27e539640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f54a5c78740 1 -- 172.21.15.40:0/632644624 shutdown_connections 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f54a5c78740 1 -- 172.21.15.40:0/632644624 wait complete. 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f5498c00640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f54800a3c00 0x7f54800a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f5498c00640 1 -- 172.21.15.40:0/1278061312 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f54880d0c60 con 0x7f549c078530 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f549be00640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f54800a3c00 0x7f54800a60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f54a5c78740 1 -- start start 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.769+0000 7f54a5c78740 1 Processor -- start 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.770+0000 7f549be00640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f54800a3c00 0x7f54800a60c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5494009b70 tx=0x7f5494005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.770+0000 7f54a5c78740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53dbd0 0x55d27e547df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.770+0000 7f54a5c78740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 0x55d27e548330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.770+0000 7f54a5c78740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e54f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.770+0000 7f54a5c78740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d27e5466e0 con 0x55d27e543730 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.770+0000 7f54a5c78740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55d27e429c60 con 0x55d27e53dbd0 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.770+0000 7f54a5c78740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d27e546860 con 0x55d27e53e5c0 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.770+0000 7f54a5c78740 10 client.0 _ll_register_callbacks cb 0x55d27e455e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.770+0000 7f549be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53dbd0 0x55d27e547df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.770+0000 7f54a0e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e54f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.770+0000 7f549b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 0x55d27e548330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.771+0000 7f549be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53dbd0 0x55d27e547df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:39876/0 (socket says 172.21.15.40:39876) 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.771+0000 7f549b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 0x55d27e548330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:41428/0 (socket says 172.21.15.40:41428) 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.771+0000 7f549be00640 1 -- 172.21.15.40:0/374626461 learned_addr learned my addr 172.21.15.40:0/374626461 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.771+0000 7f549b400640 1 -- 172.21.15.40:0/374626461 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53dbd0 msgr2=0x55d27e547df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.771+0000 7f549b400640 1 --2- 172.21.15.40:0/374626461 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53dbd0 0x55d27e547df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.771+0000 7f549b400640 1 -- 172.21.15.40:0/374626461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 msgr2=0x55d27e54f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.771+0000 7f549b400640 1 --2- 172.21.15.40:0/374626461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e54f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.771+0000 7f549b400640 1 -- 172.21.15.40:0/374626461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d27e438510 con 0x55d27e53e5c0 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.771+0000 7f54a0e00640 1 --2- 172.21.15.40:0/374626461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e54f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.771+0000 7f549b400640 1 --2- 172.21.15.40:0/374626461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 0x55d27e548330 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f54880ce210 tx=0x7f5488010720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d015330e7f44939f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.771+0000 7f5486000640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.771+0000 7f5486000640 1 -- 172.21.15.40:0/374626461 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54880a0d90 con 0x55d27e53e5c0 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f54a5c78740 1 -- 172.21.15.40:0/374626461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f549400d8b0 con 0x55d27e53e5c0 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f5486000640 1 -- 172.21.15.40:0/374626461 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f54880ce460 con 0x55d27e53e5c0 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f5486000640 1 -- 172.21.15.40:0/374626461 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f54880a5650 con 0x55d27e53e5c0 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f54a5c78740 10 client.25915 fetch_fsmap learned FSMap version 24 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f54a5c78740 1 -- 172.21.15.40:0/374626461 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d27e1b4210 con 0x55d27e53e5c0 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f5486000640 1 -- 172.21.15.40:0/374626461 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f549400d8b0 con 0x55d27e53e5c0 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f5486000640 1 -- 172.21.15.40:0/374626461 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f54880989e0 con 0x55d27e53e5c0 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f54a5c78740 10 client.25915 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:08:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f54a5c78740 10 client.25915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:08:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f54a5c78740 20 client.25915 populate_metadata read hostname 'smithi040' 2024-04-27T00:08:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f54a5c78740 1 --2- 172.21.15.40:0/374626461 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d27e553870 0x55d27e555d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f54a5c78740 4 client.25915 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:08:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.772+0000 7f54a5c78740 1 -- 172.21.15.40:0/374626461 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d27e4aff90 con 0x55d27e553870 2024-04-27T00:08:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.773+0000 7f549be00640 1 --2- 172.21.15.40:0/374626461 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d27e553870 0x55d27e555d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.773+0000 7f549be00640 1 --2- 172.21.15.40:0/374626461 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d27e553870 0x55d27e555d00 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0f99ff1850426be server_cookie=4046cf86d0aad75 in_seq=0 out_seq=0 2024-04-27T00:08:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.773+0000 7f5486000640 10 client.25915 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.774+0000 7f5486000640 1 -- 172.21.15.40:0/374626461 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d27e4aff90 con 0x55d27e553870 2024-04-27T00:08:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.774+0000 7f5486000640 10 client.25915 handle_command_reply: tid=0 2024-04-27T00:08:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.774+0000 7f54a5c78740 1 client.25915 shutdown 2024-04-27T00:08:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.775+0000 7f54a5c78740 1 -- 172.21.15.40:0/374626461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 msgr2=0x55d27e548330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.775+0000 7f54a5c78740 1 --2- 172.21.15.40:0/374626461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 0x55d27e548330 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f54880ce210 tx=0x7f5488010720 comp rx=0 tx=0).stop 2024-04-27T00:08:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.775+0000 7f54a5c78740 1 -- 172.21.15.40:0/374626461 shutdown_connections 2024-04-27T00:08:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.775+0000 7f54a5c78740 1 --2- 172.21.15.40:0/374626461 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d27e553870 0x55d27e555d00 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.775+0000 7f54a5c78740 1 --2- 172.21.15.40:0/374626461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e54f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.775+0000 7f54a5c78740 1 --2- 172.21.15.40:0/374626461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53e5c0 0x55d27e548330 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.775+0000 7f5486000640 0 client.25915 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.775+0000 7f54a5c78740 1 --2- 172.21.15.40:0/374626461 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53dbd0 0x55d27e547df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.775+0000 7f54a5c78740 1 -- 172.21.15.40:0/374626461 >> 172.21.15.40:0/374626461 conn(0x55d27e539240 msgr2=0x55d27e538440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.775+0000 7f54a5c78740 1 -- 172.21.15.40:0/374626461 shutdown_connections 2024-04-27T00:08:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.775+0000 7f54a5c78740 1 -- 172.21.15.40:0/374626461 wait complete. 2024-04-27T00:08:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.775+0000 7f54a5c78740 20 client.25915 trim_cache size 0 max 16384 2024-04-27T00:08:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.791+0000 7f54a5c78740 1 Processor -- start 2024-04-27T00:08:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.791+0000 7f54a5c78740 1 -- start start 2024-04-27T00:08:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.791+0000 7f54a5c78740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e53dbd0 0x55d27e537bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.791+0000 7f54a5c78740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53e5c0 0x55d27e54f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f54a5c78740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e543730 0x55d27e5380f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f54a5c78740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f549c1b4280 con 0x55d27e53dbd0 2024-04-27T00:08:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f54a5c78740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f549c1b1b50 con 0x55d27e53e5c0 2024-04-27T00:08:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f54a5c78740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f549c1af9a0 con 0x55d27e543730 2024-04-27T00:08:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f549be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e53dbd0 0x55d27e537bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f549be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e53dbd0 0x55d27e537bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59292/0 (socket says 172.21.15.40:59292) 2024-04-27T00:08:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f549be00640 1 -- 172.21.15.40:0/23804280 learned_addr learned my addr 172.21.15.40:0/23804280 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f54a0e00640 1 --2- 172.21.15.40:0/23804280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e543730 0x55d27e5380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f549b400640 1 --2- 172.21.15.40:0/23804280 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53e5c0 0x55d27e54f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f549be00640 1 -- 172.21.15.40:0/23804280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e543730 msgr2=0x55d27e5380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f549be00640 1 --2- 172.21.15.40:0/23804280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e543730 0x55d27e5380f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f549be00640 1 -- 172.21.15.40:0/23804280 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53e5c0 msgr2=0x55d27e54f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f549be00640 1 --2- 172.21.15.40:0/23804280 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53e5c0 0x55d27e54f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.792+0000 7f549be00640 1 -- 172.21.15.40:0/23804280 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d27e280620 con 0x55d27e53dbd0 2024-04-27T00:08:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.793+0000 7f549be00640 1 --2- 172.21.15.40:0/23804280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e53dbd0 0x55d27e537bb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f549400e520 tx=0x7f549401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f544d2151d6946ca server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.793+0000 7f5490c00640 1 -- 172.21.15.40:0/23804280 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494027070 con 0x55d27e53dbd0 2024-04-27T00:08:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.793+0000 7f5490c00640 1 -- 172.21.15.40:0/23804280 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f549400dc80 con 0x55d27e53dbd0 2024-04-27T00:08:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.793+0000 7f5490c00640 1 -- 172.21.15.40:0/23804280 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494005540 con 0x55d27e53dbd0 2024-04-27T00:08:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.794+0000 7f54a5c78740 1 -- 172.21.15.40:0/23804280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e53dbd0 msgr2=0x55d27e537bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.794+0000 7f54a5c78740 1 --2- 172.21.15.40:0/23804280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e53dbd0 0x55d27e537bb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f549400e520 tx=0x7f549401f000 comp rx=0 tx=0).stop 2024-04-27T00:08:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.794+0000 7f54a5c78740 1 -- 172.21.15.40:0/23804280 shutdown_connections 2024-04-27T00:08:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.794+0000 7f54a5c78740 1 --2- 172.21.15.40:0/23804280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e543730 0x55d27e5380f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.794+0000 7f54a5c78740 1 --2- 172.21.15.40:0/23804280 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53e5c0 0x55d27e54f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.794+0000 7f54a5c78740 1 --2- 172.21.15.40:0/23804280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e53dbd0 0x55d27e537bb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.794+0000 7f54a5c78740 1 -- 172.21.15.40:0/23804280 >> 172.21.15.40:0/23804280 conn(0x55d27e539240 msgr2=0x55d27e53f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.794+0000 7f54a5c78740 1 -- 172.21.15.40:0/23804280 shutdown_connections 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.794+0000 7f54a5c78740 1 -- 172.21.15.40:0/23804280 wait complete. 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.794+0000 7f54a5c78740 1 -- start start 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.794+0000 7f54a5c78740 1 Processor -- start 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.795+0000 7f54a5c78740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53dbd0 0x55d27e54c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.795+0000 7f54a5c78740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53e5c0 0x55d27e54c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.795+0000 7f54a5c78740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e54d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.795+0000 7f54a5c78740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f549c1b0010 con 0x55d27e543730 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.795+0000 7f54a5c78740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f549c1b1480 con 0x55d27e53e5c0 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.795+0000 7f54a5c78740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f549c1b06e0 con 0x55d27e53dbd0 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.795+0000 7f54a5c78740 10 client.0 _ll_register_callbacks cb 0x7f54880d4240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.795+0000 7f549be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53dbd0 0x55d27e54c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.795+0000 7f54a0e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e54d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.795+0000 7f549be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53dbd0 0x55d27e54c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:41440/0 (socket says 172.21.15.40:41440) 2024-04-27T00:08:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.795+0000 7f549be00640 1 -- 172.21.15.40:0/3991254890 learned_addr learned my addr 172.21.15.40:0/3991254890 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.795+0000 7f549b400640 1 --2- 172.21.15.40:0/3991254890 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53e5c0 0x55d27e54c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f549be00640 1 -- 172.21.15.40:0/3991254890 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53e5c0 msgr2=0x55d27e54c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f549be00640 1 --2- 172.21.15.40:0/3991254890 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53e5c0 0x55d27e54c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f549be00640 1 -- 172.21.15.40:0/3991254890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 msgr2=0x55d27e54d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f549be00640 1 --2- 172.21.15.40:0/3991254890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e54d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f549be00640 1 -- 172.21.15.40:0/3991254890 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d27e45ccb0 con 0x55d27e53dbd0 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f549be00640 1 --2- 172.21.15.40:0/3991254890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53dbd0 0x55d27e54c260 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5494038ff0 tx=0x7f549400eed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=101e61c434d1ac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f5486000640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f5486000640 1 -- 172.21.15.40:0/3991254890 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494027040 con 0x55d27e53dbd0 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f54a5c78740 1 -- 172.21.15.40:0/3991254890 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f549400dc80 con 0x55d27e53dbd0 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f5486000640 1 -- 172.21.15.40:0/3991254890 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f5494036550 con 0x55d27e53dbd0 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f5486000640 1 -- 172.21.15.40:0/3991254890 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5494009d00 con 0x55d27e53dbd0 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f54a5c78740 10 client.25921 fetch_fsmap learned FSMap version 24 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.796+0000 7f54a5c78740 1 -- 172.21.15.40:0/3991254890 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d27e54eee0 con 0x55d27e53dbd0 2024-04-27T00:08:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.797+0000 7f5486000640 1 -- 172.21.15.40:0/3991254890 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f549400dc80 con 0x55d27e53dbd0 2024-04-27T00:08:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.797+0000 7f5486000640 1 -- 172.21.15.40:0/3991254890 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f549403e030 con 0x55d27e53dbd0 2024-04-27T00:08:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.797+0000 7f54a5c78740 10 client.25921 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:08:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.797+0000 7f54a5c78740 10 client.25921 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:08:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.797+0000 7f54a5c78740 20 client.25921 populate_metadata read hostname 'smithi040' 2024-04-27T00:08:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.797+0000 7f54a5c78740 1 --2- 172.21.15.40:0/3991254890 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d27e5b9e80 0x55d27e5bc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.797+0000 7f54a5c78740 4 client.25921 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:08:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.797+0000 7f54a5c78740 1 -- 172.21.15.40:0/3991254890 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d27e4a9e00 con 0x55d27e5b9e80 2024-04-27T00:08:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.797+0000 7f549b400640 1 --2- 172.21.15.40:0/3991254890 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d27e5b9e80 0x55d27e5bc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.798+0000 7f549b400640 1 --2- 172.21.15.40:0/3991254890 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d27e5b9e80 0x55d27e5bc270 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f91a856093eec10c server_cookie=f9a3ea180a2fc524 in_seq=0 out_seq=0 2024-04-27T00:08:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.798+0000 7f5486000640 10 client.25921 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f5486000640 1 -- 172.21.15.40:0/3991254890 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d27e4a9e00 con 0x55d27e5b9e80 2024-04-27T00:08:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f5486000640 10 client.25921 handle_command_reply: tid=0 2024-04-27T00:08:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f54a5c78740 1 client.25921 shutdown 2024-04-27T00:08:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f54a5c78740 1 -- 172.21.15.40:0/3991254890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53dbd0 msgr2=0x55d27e54c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f54a5c78740 1 --2- 172.21.15.40:0/3991254890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53dbd0 0x55d27e54c260 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5494038ff0 tx=0x7f549400eed0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f54a5c78740 1 -- 172.21.15.40:0/3991254890 shutdown_connections 2024-04-27T00:08:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f54a5c78740 1 --2- 172.21.15.40:0/3991254890 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d27e5b9e80 0x55d27e5bc270 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f54a5c78740 1 --2- 172.21.15.40:0/3991254890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d27e543730 0x55d27e54d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f54a5c78740 1 --2- 172.21.15.40:0/3991254890 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d27e53e5c0 0x55d27e54c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f5486000640 0 client.25921 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f54a5c78740 1 --2- 172.21.15.40:0/3991254890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d27e53dbd0 0x55d27e54c260 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f54a5c78740 1 -- 172.21.15.40:0/3991254890 >> 172.21.15.40:0/3991254890 conn(0x55d27e539240 msgr2=0x55d27e53bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f54a5c78740 1 -- 172.21.15.40:0/3991254890 shutdown_connections 2024-04-27T00:08:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.814+0000 7f54a5c78740 1 -- 172.21.15.40:0/3991254890 wait complete. 2024-04-27T00:08:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.815+0000 7f54a5c78740 20 client.25921 trim_cache size 0 max 16384 2024-04-27T00:08:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:08:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:08:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:08:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:08:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.816+0000 7f54a2e00640 1 -- 172.21.15.40:0/1278061312 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f54800a3c00 msgr2=0x7f54800a60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.816+0000 7f54a2e00640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f54800a3c00 0x7f54800a60c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5494009b70 tx=0x7f5494005bd0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.816+0000 7f54a2e00640 1 -- 172.21.15.40:0/1278061312 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c078530 msgr2=0x7f549c19f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.816+0000 7f54a2e00640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c078530 0x7f549c19f640 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f548800e9f0 tx=0x7f548800eeb0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.816+0000 7f54a2e00640 1 -- 172.21.15.40:0/1278061312 shutdown_connections 2024-04-27T00:08:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.816+0000 7f54a2e00640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f549c078ed0 0x7f549c1a6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.816+0000 7f54a2e00640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f549c078530 0x7f549c19f640 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.816+0000 7f54a2e00640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f54800a3c00 0x7f54800a60c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.816+0000 7f54a2e00640 1 --2- 172.21.15.40:0/1278061312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f549c0772f0 0x7f549c19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.816+0000 7f54a2e00640 1 -- 172.21.15.40:0/1278061312 >> 172.21.15.40:0/1278061312 conn(0x7f549c100150 msgr2=0x7f549c101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.816+0000 7f54a2e00640 1 -- 172.21.15.40:0/1278061312 shutdown_connections 2024-04-27T00:08:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:11.816+0000 7f54a2e00640 1 -- 172.21.15.40:0/1278061312 wait complete. 2024-04-27T00:08:12.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:12 smithi103 ceph-mon[32952]: pgmap v607: 129 pgs: 129 active+clean; 4.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:08:12.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:12 smithi040 ceph-mon[28095]: pgmap v607: 129 pgs: 129 active+clean; 4.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:08:12.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:12 smithi080 ceph-mon[32362]: pgmap v607: 129 pgs: 129 active+clean; 4.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:08:14.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:14 smithi103 ceph-mon[32952]: pgmap v608: 129 pgs: 129 active+clean; 4.2 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:08:14.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:14 smithi040 ceph-mon[28095]: pgmap v608: 129 pgs: 129 active+clean; 4.2 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:08:14.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:14 smithi080 ceph-mon[32362]: pgmap v608: 129 pgs: 129 active+clean; 4.2 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:08:16.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:16 smithi103 ceph-mon[32952]: pgmap v609: 129 pgs: 129 active+clean; 4.2 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:08:16.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:16 smithi040 ceph-mon[28095]: pgmap v609: 129 pgs: 129 active+clean; 4.2 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:08:16.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:16 smithi080 ceph-mon[32362]: pgmap v609: 129 pgs: 129 active+clean; 4.2 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:08:18.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:18 smithi103 ceph-mon[32952]: pgmap v610: 129 pgs: 129 active+clean; 4.3 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:08:18.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:18 smithi040 ceph-mon[28095]: pgmap v610: 129 pgs: 129 active+clean; 4.3 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:08:18.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:18 smithi080 ceph-mon[32362]: pgmap v610: 129 pgs: 129 active+clean; 4.3 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:08:20.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:20 smithi103 ceph-mon[32952]: pgmap v611: 129 pgs: 129 active+clean; 4.3 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:08:20.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:20 smithi040 ceph-mon[28095]: pgmap v611: 129 pgs: 129 active+clean; 4.3 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:08:20.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:20 smithi080 ceph-mon[32362]: pgmap v611: 129 pgs: 129 active+clean; 4.3 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:08:22.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:22 smithi103 ceph-mon[32952]: pgmap v612: 129 pgs: 129 active+clean; 4.3 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:08:22.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:22 smithi040 ceph-mon[28095]: pgmap v612: 129 pgs: 129 active+clean; 4.3 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:08:22.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:22 smithi080 ceph-mon[32362]: pgmap v612: 129 pgs: 129 active+clean; 4.3 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:08:24.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:24 smithi103 ceph-mon[32952]: pgmap v613: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:08:24.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:24 smithi040 ceph-mon[28095]: pgmap v613: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:08:24.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:24 smithi080 ceph-mon[32362]: pgmap v613: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:08:24.966 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.965+0000 7f6b4be00640 1 -- 172.21.15.40:0/2705967857 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b4c109fd0 msgr2=0x7f6b4c10a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:24.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.965+0000 7f6b4be00640 1 --2- 172.21.15.40:0/2705967857 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b4c109fd0 0x7f6b4c10a450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009900 tx=0x7f6b3402a410 comp rx=0 tx=0).stop 2024-04-27T00:08:24.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.965+0000 7f6b4be00640 1 -- 172.21.15.40:0/2705967857 shutdown_connections 2024-04-27T00:08:24.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.965+0000 7f6b4be00640 1 --2- 172.21.15.40:0/2705967857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b4c10a990 0x7f6b4c111220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.965+0000 7f6b4be00640 1 --2- 172.21.15.40:0/2705967857 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b4c109fd0 0x7f6b4c10a450 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.965+0000 7f6b4be00640 1 --2- 172.21.15.40:0/2705967857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b4c108de0 0x7f6b4c109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.965+0000 7f6b4be00640 1 -- 172.21.15.40:0/2705967857 >> 172.21.15.40:0/2705967857 conn(0x7f6b4c077dc0 msgr2=0x7f6b4c07a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:24.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.965+0000 7f6b4be00640 1 -- 172.21.15.40:0/2705967857 shutdown_connections 2024-04-27T00:08:24.967 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.965+0000 7f6b4be00640 1 -- 172.21.15.40:0/2705967857 wait complete. 2024-04-27T00:08:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.966+0000 7f6b4be00640 1 Processor -- start 2024-04-27T00:08:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.967+0000 7f6b4be00640 1 -- start start 2024-04-27T00:08:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.967+0000 7f6b4be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b4c108de0 0x7f6b4c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.967+0000 7f6b4be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b4c109fd0 0x7f6b4c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.967+0000 7f6b4be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b4c10a990 0x7f6b4c1a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.967+0000 7f6b4be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c114910 con 0x7f6b4c109fd0 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.967+0000 7f6b4be00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c114790 con 0x7f6b4c10a990 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.967+0000 7f6b4be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c114a90 con 0x7f6b4c108de0 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.967+0000 7f6b43e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b4c109fd0 0x7f6b4c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.967+0000 7f6b43e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b4c109fd0 0x7f6b4c19f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53242/0 (socket says 172.21.15.40:53242) 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.967+0000 7f6b4aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b4c108de0 0x7f6b4c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.967+0000 7f6b43e00640 1 -- 172.21.15.40:0/271076986 learned_addr learned my addr 172.21.15.40:0/271076986 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.967+0000 7f6b4b400640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b4c10a990 0x7f6b4c1a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.968+0000 7f6b4aa00640 1 -- 172.21.15.40:0/271076986 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b4c10a990 msgr2=0x7f6b4c1a6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.968+0000 7f6b4aa00640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b4c10a990 0x7f6b4c1a6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.968+0000 7f6b4aa00640 1 -- 172.21.15.40:0/271076986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b4c109fd0 msgr2=0x7f6b4c19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.968+0000 7f6b4aa00640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b4c109fd0 0x7f6b4c19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.968+0000 7f6b4aa00640 1 -- 172.21.15.40:0/271076986 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4c1a6c00 con 0x7f6b4c108de0 2024-04-27T00:08:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.968+0000 7f6b43e00640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b4c109fd0 0x7f6b4c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:08:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.968+0000 7f6b4aa00640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b4c108de0 0x7f6b4c19efe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00e9c0 tx=0x7f6b3c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.968+0000 7f6b43400640 1 -- 172.21.15.40:0/271076986 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c00ccc0 con 0x7f6b4c108de0 2024-04-27T00:08:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.968+0000 7f6b43400640 1 -- 172.21.15.40:0/271076986 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b3c0035f0 con 0x7f6b4c108de0 2024-04-27T00:08:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.968+0000 7f6b43400640 1 -- 172.21.15.40:0/271076986 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c00ce60 con 0x7f6b4c108de0 2024-04-27T00:08:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.968+0000 7f6b4be00640 1 -- 172.21.15.40:0/271076986 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b4c1a6ef0 con 0x7f6b4c108de0 2024-04-27T00:08:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.968+0000 7f6b4be00640 1 -- 172.21.15.40:0/271076986 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b4c1a7300 con 0x7f6b4c108de0 2024-04-27T00:08:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.972+0000 7f6b5446e740 1 Processor -- start 2024-04-27T00:08:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.972+0000 7f6b5446e740 1 -- start start 2024-04-27T00:08:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.972+0000 7f6b43400640 1 -- 172.21.15.40:0/271076986 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6b3c004000 con 0x7f6b4c108de0 2024-04-27T00:08:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.972+0000 7f6b5446e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 0x56404d244200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.972+0000 7f6b5446e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d2447d0 0x56404d2492d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.972+0000 7f6b5446e740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d249940 0x56404d24bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.972+0000 7f6b5446e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56404d1a8380 con 0x56404d243de0 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.972+0000 7f6b5446e740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x56404d144c30 con 0x56404d249940 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.972+0000 7f6b5446e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56404d1a4340 con 0x56404d2447d0 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.973+0000 7f6b4b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 0x56404d244200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.973+0000 7f6b4b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 0x56404d244200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53254/0 (socket says 172.21.15.40:53254) 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.973+0000 7f6b4b400640 1 -- 172.21.15.40:0/2809722751 learned_addr learned my addr 172.21.15.40:0/2809722751 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.973+0000 7f6b43e00640 1 --2- 172.21.15.40:0/2809722751 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d249940 0x56404d24bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.973+0000 7f6b4aa00640 1 --2- 172.21.15.40:0/2809722751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d2447d0 0x56404d2492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.973+0000 7f6b4b400640 1 -- 172.21.15.40:0/2809722751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d2447d0 msgr2=0x56404d2492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.973+0000 7f6b4b400640 1 --2- 172.21.15.40:0/2809722751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d2447d0 0x56404d2492d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.973+0000 7f6b4b400640 1 -- 172.21.15.40:0/2809722751 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d249940 msgr2=0x56404d24bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.973+0000 7f6b4b400640 1 --2- 172.21.15.40:0/2809722751 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d249940 0x56404d24bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.973+0000 7f6b4b400640 1 -- 172.21.15.40:0/2809722751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56404d166940 con 0x56404d243de0 2024-04-27T00:08:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.973+0000 7f6b4b400640 1 --2- 172.21.15.40:0/2809722751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 0x56404d244200 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009fd0 tx=0x7f6b440118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4493083d0fb8e592 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:24.990 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.974+0000 7f6b3b400640 1 -- 172.21.15.40:0/2809722751 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44004d20 con 0x56404d243de0 2024-04-27T00:08:24.990 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.974+0000 7f6b3b400640 1 -- 172.21.15.40:0/2809722751 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b44004050 con 0x56404d243de0 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.974+0000 7f6b43400640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b300a3c30 0x7f6b300a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.974+0000 7f6b3b400640 1 -- 172.21.15.40:0/2809722751 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44005280 con 0x56404d243de0 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.974+0000 7f6b43400640 1 -- 172.21.15.40:0/271076986 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f6b3c0d0da0 con 0x7f6b4c108de0 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.974+0000 7f6b4aa00640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b300a3c30 0x7f6b300a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.975+0000 7f6b5446e740 1 -- 172.21.15.40:0/2809722751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 msgr2=0x56404d244200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.975+0000 7f6b5446e740 1 --2- 172.21.15.40:0/2809722751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 0x56404d244200 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009fd0 tx=0x7f6b440118d0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.975+0000 7f6b5446e740 1 -- 172.21.15.40:0/2809722751 shutdown_connections 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.975+0000 7f6b5446e740 1 --2- 172.21.15.40:0/2809722751 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d249940 0x56404d24bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.975+0000 7f6b5446e740 1 --2- 172.21.15.40:0/2809722751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d2447d0 0x56404d2492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.975+0000 7f6b5446e740 1 --2- 172.21.15.40:0/2809722751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 0x56404d244200 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.975+0000 7f6b5446e740 1 -- 172.21.15.40:0/2809722751 >> 172.21.15.40:0/2809722751 conn(0x56404d23f450 msgr2=0x56404d23f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.975+0000 7f6b5446e740 1 -- 172.21.15.40:0/2809722751 shutdown_connections 2024-04-27T00:08:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.975+0000 7f6b4aa00640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b300a3c30 0x7f6b300a60f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x56404d244bd0 tx=0x7f6b3c0d4850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.975+0000 7f6b5446e740 1 -- 172.21.15.40:0/2809722751 wait complete. 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.975+0000 7f6b5446e740 1 -- start start 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.975+0000 7f6b5446e740 1 Processor -- start 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.976+0000 7f6b5446e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 0x56404d24df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.976+0000 7f6b5446e740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d2447d0 0x56404d24e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.976+0000 7f6b5446e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d249940 0x56404d255510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.976+0000 7f6b5446e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56404d24c8f0 con 0x56404d243de0 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.976+0000 7f6b5446e740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x56404d12fc60 con 0x56404d2447d0 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.976+0000 7f6b5446e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56404d24ca70 con 0x56404d249940 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.976+0000 7f6b5446e740 10 client.0 _ll_register_callbacks cb 0x56404d15be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.976+0000 7f6b4b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 0x56404d24df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.976+0000 7f6b4b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 0x56404d24df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53270/0 (socket says 172.21.15.40:53270) 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.976+0000 7f6b4b400640 1 -- 172.21.15.40:0/1785161361 learned_addr learned my addr 172.21.15.40:0/1785161361 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:24.992 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.976+0000 7f6b43e00640 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d2447d0 0x56404d24e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.976+0000 7f6b4b400640 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d249940 0x56404d255510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.977+0000 7f6b43e00640 1 -- 172.21.15.40:0/1785161361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d249940 msgr2=0x56404d255510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.977+0000 7f6b43e00640 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d249940 0x56404d255510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.977+0000 7f6b43e00640 1 -- 172.21.15.40:0/1785161361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 msgr2=0x56404d24df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.977+0000 7f6b43e00640 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 0x56404d24df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.977+0000 7f6b43e00640 1 -- 172.21.15.40:0/1785161361 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56404d13e510 con 0x56404d2447d0 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.977+0000 7f6b43e00640 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d2447d0 0x56404d24e4d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6b34005940 tx=0x7f6b340317d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=893a43537b4d88f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.977+0000 7f6b2fe00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.977+0000 7f6b2fe00640 1 -- 172.21.15.40:0/1785161361 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34035040 con 0x56404d2447d0 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.978+0000 7f6b5446e740 1 -- 172.21.15.40:0/1785161361 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b44004050 con 0x56404d2447d0 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.978+0000 7f6b2fe00640 1 -- 172.21.15.40:0/1785161361 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f6b34004450 con 0x56404d2447d0 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.978+0000 7f6b2fe00640 1 -- 172.21.15.40:0/1785161361 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6b340035f0 con 0x56404d2447d0 2024-04-27T00:08:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.978+0000 7f6b5446e740 10 client.25904 fetch_fsmap learned FSMap version 24 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.978+0000 7f6b5446e740 1 -- 172.21.15.40:0/1785161361 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56404ceba210 con 0x56404d2447d0 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.978+0000 7f6b2fe00640 1 -- 172.21.15.40:0/1785161361 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b44004050 con 0x56404d2447d0 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.978+0000 7f6b2fe00640 1 -- 172.21.15.40:0/1785161361 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6b34044070 con 0x56404d2447d0 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.978+0000 7f6b5446e740 10 client.25904 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.978+0000 7f6b5446e740 10 client.25904 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.978+0000 7f6b5446e740 20 client.25904 populate_metadata read hostname 'smithi040' 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.979+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56404d2599b0 0x56404d25be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.979+0000 7f6b5446e740 4 client.25904 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.979+0000 7f6b5446e740 1 -- 172.21.15.40:0/1785161361 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56404d1b5f90 con 0x56404d2599b0 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.979+0000 7f6b4b400640 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56404d2599b0 0x56404d25be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.980+0000 7f6b4b400640 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56404d2599b0 0x56404d25be20 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e14e370826e092e server_cookie=64f4ee0a049d9cb1 in_seq=0 out_seq=0 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.980+0000 7f6b2fe00640 10 client.25904 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.980+0000 7f6b2fe00640 1 -- 172.21.15.40:0/1785161361 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56404d1b5f90 con 0x56404d2599b0 2024-04-27T00:08:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.980+0000 7f6b2fe00640 10 client.25904 handle_command_reply: tid=0 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b5446e740 1 client.25904 shutdown 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b5446e740 1 -- 172.21.15.40:0/1785161361 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d2447d0 msgr2=0x56404d24e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d2447d0 0x56404d24e4d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6b34005940 tx=0x7f6b340317d0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b5446e740 1 -- 172.21.15.40:0/1785161361 shutdown_connections 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56404d2599b0 0x56404d25be20 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d249940 0x56404d255510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d2447d0 0x56404d24e4d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b2fe00640 0 client.25904 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1785161361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d243de0 0x56404d24df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b5446e740 1 -- 172.21.15.40:0/1785161361 >> 172.21.15.40:0/1785161361 conn(0x56404d23f450 msgr2=0x56404d23e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b5446e740 1 -- 172.21.15.40:0/1785161361 shutdown_connections 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b5446e740 1 -- 172.21.15.40:0/1785161361 wait complete. 2024-04-27T00:08:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.981+0000 7f6b5446e740 20 client.25904 trim_cache size 0 max 16384 2024-04-27T00:08:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.998+0000 7f6b5446e740 1 Processor -- start 2024-04-27T00:08:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.998+0000 7f6b5446e740 1 -- start start 2024-04-27T00:08:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.998+0000 7f6b5446e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d243de0 0x56404d2459b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.998+0000 7f6b5446e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d2447d0 0x56404d245ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.998+0000 7f6b5446e740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d249940 0x56404d2554d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:24.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.998+0000 7f6b5446e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c1b41b0 con 0x56404d2447d0 2024-04-27T00:08:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.998+0000 7f6b5446e740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c1b1a80 con 0x56404d249940 2024-04-27T00:08:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.998+0000 7f6b5446e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c1af8d0 con 0x56404d243de0 2024-04-27T00:08:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.999+0000 7f6b4b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d243de0 0x56404d2459b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.999+0000 7f6b43e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d2447d0 0x56404d245ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.999+0000 7f6b4b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d243de0 0x56404d2459b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36734/0 (socket says 172.21.15.40:36734) 2024-04-27T00:08:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.999+0000 7f6b4b400640 1 -- 172.21.15.40:0/1222742086 learned_addr learned my addr 172.21.15.40:0/1222742086 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.999+0000 7f6b4b400640 1 --2- 172.21.15.40:0/1222742086 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d249940 0x56404d2554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.999+0000 7f6b43e00640 1 -- 172.21.15.40:0/1222742086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d243de0 msgr2=0x56404d2459b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.999+0000 7f6b43e00640 1 --2- 172.21.15.40:0/1222742086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d243de0 0x56404d2459b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.999+0000 7f6b43e00640 1 -- 172.21.15.40:0/1222742086 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d249940 msgr2=0x56404d2554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.999+0000 7f6b43e00640 1 --2- 172.21.15.40:0/1222742086 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d249940 0x56404d2554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.999+0000 7f6b43e00640 1 -- 172.21.15.40:0/1222742086 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56404cf86620 con 0x56404d2447d0 2024-04-27T00:08:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:24.999+0000 7f6b4b400640 1 --2- 172.21.15.40:0/1222742086 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d249940 0x56404d2554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:08:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.000+0000 7f6b43e00640 1 --2- 172.21.15.40:0/1222742086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d2447d0 0x56404d245ef0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6b34000c00 tx=0x7f6b3402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f2c9f7d38a600f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.000+0000 7f6b3aa00640 1 -- 172.21.15.40:0/1222742086 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34035070 con 0x56404d2447d0 2024-04-27T00:08:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.000+0000 7f6b3aa00640 1 -- 172.21.15.40:0/1222742086 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b3403c640 con 0x56404d2447d0 2024-04-27T00:08:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.000+0000 7f6b3aa00640 1 -- 172.21.15.40:0/1222742086 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b340039f0 con 0x56404d2447d0 2024-04-27T00:08:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.001+0000 7f6b5446e740 1 -- 172.21.15.40:0/1222742086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d2447d0 msgr2=0x56404d245ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.001+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1222742086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d2447d0 0x56404d245ef0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6b34000c00 tx=0x7f6b3402a410 comp rx=0 tx=0).stop 2024-04-27T00:08:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.001+0000 7f6b5446e740 1 -- 172.21.15.40:0/1222742086 shutdown_connections 2024-04-27T00:08:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.001+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1222742086 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d249940 0x56404d2554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.001+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1222742086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d2447d0 0x56404d245ef0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.001+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1222742086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d243de0 0x56404d2459b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.001+0000 7f6b5446e740 1 -- 172.21.15.40:0/1222742086 >> 172.21.15.40:0/1222742086 conn(0x56404d23f450 msgr2=0x56404d248fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.001+0000 7f6b5446e740 1 -- 172.21.15.40:0/1222742086 shutdown_connections 2024-04-27T00:08:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.001+0000 7f6b5446e740 1 -- 172.21.15.40:0/1222742086 wait complete. 2024-04-27T00:08:25.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.001+0000 7f6b5446e740 1 -- start start 2024-04-27T00:08:25.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.002+0000 7f6b5446e740 1 Processor -- start 2024-04-27T00:08:25.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.002+0000 7f6b5446e740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d243de0 0x56404d252800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:25.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.002+0000 7f6b5446e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d2447d0 0x56404d252d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.002+0000 7f6b5446e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d249940 0x56404d253ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.002+0000 7f6b5446e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c1aff40 con 0x56404d249940 2024-04-27T00:08:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.002+0000 7f6b5446e740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c1b13b0 con 0x56404d243de0 2024-04-27T00:08:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.002+0000 7f6b5446e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c1b0610 con 0x56404d2447d0 2024-04-27T00:08:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.002+0000 7f6b5446e740 10 client.0 _ll_register_callbacks cb 0x7f6b34009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:08:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.002+0000 7f6b4b400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d243de0 0x56404d252800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.002+0000 7f6b43e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d2447d0 0x56404d252d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.003+0000 7f6b4b400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d243de0 0x56404d252800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:39204/0 (socket says 172.21.15.40:39204) 2024-04-27T00:08:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.003+0000 7f6b43e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d2447d0 0x56404d252d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36742/0 (socket says 172.21.15.40:36742) 2024-04-27T00:08:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.003+0000 7f6b4b400640 1 -- 172.21.15.40:0/1163503389 learned_addr learned my addr 172.21.15.40:0/1163503389 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.003+0000 7f6b4b400640 1 --2- 172.21.15.40:0/1163503389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d249940 0x56404d253ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.003+0000 7f6b43e00640 1 -- 172.21.15.40:0/1163503389 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d243de0 msgr2=0x56404d252800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.003+0000 7f6b43e00640 1 --2- 172.21.15.40:0/1163503389 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d243de0 0x56404d252800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.003+0000 7f6b43e00640 1 -- 172.21.15.40:0/1163503389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d249940 msgr2=0x56404d253ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.003+0000 7f6b43e00640 1 --2- 172.21.15.40:0/1163503389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d249940 0x56404d253ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.003+0000 7f6b43e00640 1 -- 172.21.15.40:0/1163503389 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56404d162cb0 con 0x56404d2447d0 2024-04-27T00:08:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.003+0000 7f6b43e00640 1 --2- 172.21.15.40:0/1163503389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d2447d0 0x56404d252d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x56404d25bff0 tx=0x7f6b340039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=467d66cb0bdae695 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.003+0000 7f6b2fe00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:08:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.004+0000 7f6b2fe00640 1 -- 172.21.15.40:0/1163503389 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34035040 con 0x56404d2447d0 2024-04-27T00:08:25.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.004+0000 7f6b2fe00640 1 -- 172.21.15.40:0/1163503389 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f6b3403c440 con 0x56404d2447d0 2024-04-27T00:08:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.004+0000 7f6b5446e740 1 -- 172.21.15.40:0/1163503389 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b3403c640 con 0x56404d2447d0 2024-04-27T00:08:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.004+0000 7f6b2fe00640 1 -- 172.21.15.40:0/1163503389 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6b34066050 con 0x56404d2447d0 2024-04-27T00:08:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.004+0000 7f6b5446e740 10 client.25939 fetch_fsmap learned FSMap version 24 2024-04-27T00:08:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.004+0000 7f6b5446e740 1 -- 172.21.15.40:0/1163503389 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56404d255410 con 0x56404d2447d0 2024-04-27T00:08:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.004+0000 7f6b2fe00640 1 -- 172.21.15.40:0/1163503389 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b3403c640 con 0x56404d2447d0 2024-04-27T00:08:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.004+0000 7f6b2fe00640 1 -- 172.21.15.40:0/1163503389 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6b34044040 con 0x56404d2447d0 2024-04-27T00:08:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.005+0000 7f6b5446e740 10 client.25939 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:08:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.005+0000 7f6b5446e740 10 client.25939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:08:25.009 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.005+0000 7f6b5446e740 20 client.25939 populate_metadata read hostname 'smithi040' 2024-04-27T00:08:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.005+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1163503389 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56404d2c0190 0x56404d2c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.005+0000 7f6b5446e740 4 client.25939 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:08:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.005+0000 7f6b5446e740 1 -- 172.21.15.40:0/1163503389 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56404d1afe00 con 0x56404d2c0190 2024-04-27T00:08:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.005+0000 7f6b4b400640 1 --2- 172.21.15.40:0/1163503389 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56404d2c0190 0x56404d2c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.005+0000 7f6b4b400640 1 --2- 172.21.15.40:0/1163503389 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56404d2c0190 0x56404d2c25b0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1156725abf81f2be server_cookie=8ac63f9e8fded935 in_seq=0 out_seq=0 2024-04-27T00:08:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.006+0000 7f6b2fe00640 10 client.25939 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.022+0000 7f6b2fe00640 1 -- 172.21.15.40:0/1163503389 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56404d1afe00 con 0x56404d2c0190 2024-04-27T00:08:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.022+0000 7f6b2fe00640 10 client.25939 handle_command_reply: tid=0 2024-04-27T00:08:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.022+0000 7f6b5446e740 1 client.25939 shutdown 2024-04-27T00:08:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.023+0000 7f6b5446e740 1 -- 172.21.15.40:0/1163503389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d2447d0 msgr2=0x56404d252d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.023+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1163503389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d2447d0 0x56404d252d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x56404d25bff0 tx=0x7f6b340039c0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.023+0000 7f6b5446e740 1 -- 172.21.15.40:0/1163503389 shutdown_connections 2024-04-27T00:08:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.023+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1163503389 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56404d2c0190 0x56404d2c25b0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.023+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1163503389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56404d249940 0x56404d253ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.023+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1163503389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56404d2447d0 0x56404d252d40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.023+0000 7f6b5446e740 1 --2- 172.21.15.40:0/1163503389 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x56404d243de0 0x56404d252800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.024 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.023+0000 7f6b5446e740 1 -- 172.21.15.40:0/1163503389 >> 172.21.15.40:0/1163503389 conn(0x56404d23f450 msgr2=0x56404d24df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:25.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.023+0000 7f6b2fe00640 0 client.25939 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:25.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.023+0000 7f6b5446e740 1 -- 172.21.15.40:0/1163503389 shutdown_connections 2024-04-27T00:08:25.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.023+0000 7f6b5446e740 1 -- 172.21.15.40:0/1163503389 wait complete. 2024-04-27T00:08:25.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.023+0000 7f6b5446e740 20 client.25939 trim_cache size 0 max 16384 2024-04-27T00:08:25.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:08:25.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:08:25.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:08:25.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:08:25.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.024+0000 7f6b4be00640 1 -- 172.21.15.40:0/271076986 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b300a3c30 msgr2=0x7f6b300a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:25.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.024+0000 7f6b4be00640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b300a3c30 0x7f6b300a60f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x56404d244bd0 tx=0x7f6b3c0d4850 comp rx=0 tx=0).stop 2024-04-27T00:08:25.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.024+0000 7f6b4be00640 1 -- 172.21.15.40:0/271076986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b4c108de0 msgr2=0x7f6b4c19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:25.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.024+0000 7f6b4be00640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b4c108de0 0x7f6b4c19efe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00e9c0 tx=0x7f6b3c00ee80 comp rx=0 tx=0).stop 2024-04-27T00:08:25.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.024+0000 7f6b4be00640 1 -- 172.21.15.40:0/271076986 shutdown_connections 2024-04-27T00:08:25.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.024+0000 7f6b4be00640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f6b4c10a990 0x7f6b4c1a6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.024+0000 7f6b4be00640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b4c109fd0 0x7f6b4c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.027 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.024+0000 7f6b4be00640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f6b300a3c30 0x7f6b300a60f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.027 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.024+0000 7f6b4be00640 1 --2- 172.21.15.40:0/271076986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b4c108de0 0x7f6b4c19efe0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:25.027 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.024+0000 7f6b4be00640 1 -- 172.21.15.40:0/271076986 >> 172.21.15.40:0/271076986 conn(0x7f6b4c077dc0 msgr2=0x7f6b4c10d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:25.027 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.024+0000 7f6b4be00640 1 -- 172.21.15.40:0/271076986 shutdown_connections 2024-04-27T00:08:25.027 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:25.024+0000 7f6b4be00640 1 -- 172.21.15.40:0/271076986 wait complete. 2024-04-27T00:08:26.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:26 smithi040 ceph-mon[28095]: pgmap v614: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:08:26.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:26 smithi080 ceph-mon[32362]: pgmap v614: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:08:26.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:26 smithi103 ceph-mon[32952]: pgmap v614: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:08:28.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:28 smithi040 ceph-mon[28095]: pgmap v615: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:08:28.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:28 smithi080 ceph-mon[32362]: pgmap v615: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:08:28.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:28 smithi103 ceph-mon[32952]: pgmap v615: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:08:30.399 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:30 smithi103 ceph-mon[32952]: pgmap v616: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-27T00:08:30.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:30 smithi040 ceph-mon[28095]: pgmap v616: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-27T00:08:30.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:30 smithi080 ceph-mon[32362]: pgmap v616: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-27T00:08:32.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:32 smithi040 ceph-mon[28095]: pgmap v617: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:08:32.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:32 smithi080 ceph-mon[32362]: pgmap v617: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:08:32.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:32 smithi103 ceph-mon[32952]: pgmap v617: 129 pgs: 129 active+clean; 4.4 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:08:34.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:34 smithi040 ceph-mon[28095]: pgmap v618: 129 pgs: 129 active+clean; 4.5 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:08:34.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:34 smithi080 ceph-mon[32362]: pgmap v618: 129 pgs: 129 active+clean; 4.5 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:08:34.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:34 smithi103 ceph-mon[32952]: pgmap v618: 129 pgs: 129 active+clean; 4.5 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:08:35.101 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.101 INFO:tasks.workunit.client.0.smithi040.stdout:251/600: open fd = 3 2024-04-27T00:08:35.101 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.101 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.102 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.102 INFO:tasks.workunit.client.0.smithi040.stdout:252/600: open fd = 3 2024-04-27T00:08:35.102 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.102 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.102 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.102 INFO:tasks.workunit.client.0.smithi040.stdout:253/600: open fd = 3 2024-04-27T00:08:35.102 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.102 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.102 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.102 INFO:tasks.workunit.client.0.smithi040.stdout:254/600: open fd = 3 2024-04-27T00:08:35.102 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.102 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:255/600: open fd = 3 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:256/600: open fd = 3 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:257/600: open fd = 3 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.103 INFO:tasks.workunit.client.0.smithi040.stdout:258/600: open fd = 3 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:259/600: open fd = 3 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:260/600: open fd = 3 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:261/600: open fd = 3 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.104 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:262/600: open fd = 3 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:263/600: open fd = 3 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:264/600: open fd = 3 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:265/600: open fd = 3 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.105 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:266/600: open fd = 3 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:267/600: open fd = 3 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:268/600: open fd = 3 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:269/600: open fd = 3 2024-04-27T00:08:35.106 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:270/600: open fd = 3 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:271/600: open fd = 3 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:272/600: open fd = 3 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.107 INFO:tasks.workunit.client.0.smithi040.stdout:273/600: open fd = 3 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:274/600: open fd = 3 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:275/600: open fd = 3 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:276/600: open fd = 3 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.108 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:277/600: open fd = 3 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:278/600: open fd = 3 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:279/600: open fd = 3 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:280/600: open fd = 3 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.109 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:281/600: open fd = 3 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:282/600: open fd = 3 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:283/600: open fd = 3 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.110 INFO:tasks.workunit.client.0.smithi040.stdout:284/600: open fd = 3 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:285/600: open fd = 3 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:286/600: open fd = 3 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:287/600: open fd = 3 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.111 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:288/600: open fd = 3 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:289/600: open fd = 3 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:290/600: open fd = 3 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:291/600: open fd = 3 2024-04-27T00:08:35.112 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:292/600: open fd = 3 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:293/600: open fd = 3 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:294/600: open fd = 3 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.113 INFO:tasks.workunit.client.0.smithi040.stdout:295/600: open fd = 3 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:296/600: open fd = 3 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:297/600: open fd = 3 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:298/600: open fd = 3 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.114 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:299/600: open fd = 3 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:300/600: open fd = 3 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:301/600: open fd = 3 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.115 INFO:tasks.workunit.client.0.smithi040.stdout:302/600: open fd = 3 2024-04-27T00:08:35.116 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.116 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.116 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.116 INFO:tasks.workunit.client.0.smithi040.stdout:303/600: open fd = 3 2024-04-27T00:08:35.116 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.116 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.116 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.116 INFO:tasks.workunit.client.0.smithi040.stdout:304/600: open fd = 3 2024-04-27T00:08:35.116 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:305/600: open fd = 3 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:306/600: open fd = 3 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:307/600: open fd = 3 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.117 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:308/600: open fd = 3 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:309/600: open fd = 3 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:310/600: open fd = 3 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.118 INFO:tasks.workunit.client.0.smithi040.stdout:311/600: open fd = 3 2024-04-27T00:08:35.119 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.119 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.119 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:08:35.119 INFO:tasks.workunit.client.0.smithi040.stdout:312/600: open fd = 3 2024-04-27T00:08:35.119 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:35.119 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:08:36.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:36 smithi040 ceph-mon[28095]: pgmap v619: 129 pgs: 129 active+clean; 4.5 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-27T00:08:36.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:36 smithi080 ceph-mon[32362]: pgmap v619: 129 pgs: 129 active+clean; 4.5 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-27T00:08:36.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:36 smithi103 ceph-mon[32952]: pgmap v619: 129 pgs: 129 active+clean; 4.5 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-27T00:08:38.179 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.178+0000 7fb123e00640 1 -- 172.21.15.40:0/2107009727 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb1241049f0 msgr2=0x7fb124104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.178+0000 7fb123e00640 1 --2- 172.21.15.40:0/2107009727 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb1241049f0 0x7fb124104e10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb114009900 tx=0x7fb11402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.178+0000 7fb123e00640 1 -- 172.21.15.40:0/2107009727 shutdown_connections 2024-04-27T00:08:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.178+0000 7fb123e00640 1 --2- 172.21.15.40:0/2107009727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1241065a0 0x7fb12410ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.178+0000 7fb123e00640 1 --2- 172.21.15.40:0/2107009727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb124105be0 0x7fb124106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.178+0000 7fb123e00640 1 --2- 172.21.15.40:0/2107009727 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb1241049f0 0x7fb124104e10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.178+0000 7fb123e00640 1 -- 172.21.15.40:0/2107009727 >> 172.21.15.40:0/2107009727 conn(0x7fb1240fffa0 msgr2=0x7fb1241023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.179+0000 7fb123e00640 1 -- 172.21.15.40:0/2107009727 shutdown_connections 2024-04-27T00:08:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.179+0000 7fb123e00640 1 -- 172.21.15.40:0/2107009727 wait complete. 2024-04-27T00:08:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.180+0000 7fb123e00640 1 Processor -- start 2024-04-27T00:08:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.180+0000 7fb123e00640 1 -- start start 2024-04-27T00:08:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb123e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1241049f0 0x7fb12419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb123e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb124105be0 0x7fb12419b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.181 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb123e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb1241065a0 0x7fb1241a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb123e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241104f0 con 0x7fb1241049f0 2024-04-27T00:08:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb123e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb124110370 con 0x7fb1241065a0 2024-04-27T00:08:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb123e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb124110670 con 0x7fb124105be0 2024-04-27T00:08:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb122a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1241049f0 0x7fb12419abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb11a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb124105be0 0x7fb12419b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb11a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb124105be0 0x7fb12419b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:59670/0 (socket says 172.21.15.40:59670) 2024-04-27T00:08:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb11a000640 1 -- 172.21.15.40:0/2323256870 learned_addr learned my addr 172.21.15.40:0/2323256870 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb123400640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb1241065a0 0x7fb1241a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb11a000640 1 -- 172.21.15.40:0/2323256870 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb1241065a0 msgr2=0x7fb1241a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb11a000640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb1241065a0 0x7fb1241a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb11a000640 1 -- 172.21.15.40:0/2323256870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1241049f0 msgr2=0x7fb12419abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb11a000640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1241049f0 0x7fb12419abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.181+0000 7fb11a000640 1 -- 172.21.15.40:0/2323256870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1241a2810 con 0x7fb124105be0 2024-04-27T00:08:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.182+0000 7fb11a000640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb124105be0 0x7fb12419b130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb10c00ebc0 tx=0x7fb10c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.182+0000 7fb11be00640 1 -- 172.21.15.40:0/2323256870 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c010630 con 0x7fb124105be0 2024-04-27T00:08:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.182+0000 7fb11be00640 1 -- 172.21.15.40:0/2323256870 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb10c010c80 con 0x7fb124105be0 2024-04-27T00:08:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.182+0000 7fb11be00640 1 -- 172.21.15.40:0/2323256870 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c018b60 con 0x7fb124105be0 2024-04-27T00:08:38.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.182+0000 7fb123e00640 1 -- 172.21.15.40:0/2323256870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1241a2b00 con 0x7fb124105be0 2024-04-27T00:08:38.185 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.182+0000 7fb123e00640 1 -- 172.21.15.40:0/2323256870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1241a30a0 con 0x7fb124105be0 2024-04-27T00:08:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.185+0000 7fb12c57a740 1 Processor -- start 2024-04-27T00:08:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.185+0000 7fb11be00640 1 -- 172.21.15.40:0/2323256870 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb10c018d00 con 0x7fb124105be0 2024-04-27T00:08:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb12c57a740 1 -- start start 2024-04-27T00:08:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb12c57a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e2919f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb12c57a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb12c57a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 0x556e291a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb12c57a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556e29103380 con 0x556e291a4940 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb12c57a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x556e2909fc30 con 0x556e2919f7d0 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb12c57a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556e290ff340 con 0x556e2919ede0 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb11a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 0x556e291a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb123400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e2919f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb122a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb123400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e2919f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:59674/0 (socket says 172.21.15.40:59674) 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb122a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291a42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:52736/0 (socket says 172.21.15.40:52736) 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb123400640 1 -- 172.21.15.40:0/677010747 learned_addr learned my addr 172.21.15.40:0/677010747 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.186+0000 7fb11a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 0x556e291a6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51892/0 (socket says 172.21.15.40:51892) 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.187+0000 7fb123400640 1 -- 172.21.15.40:0/677010747 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 msgr2=0x556e291a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.187+0000 7fb123400640 1 --2- 172.21.15.40:0/677010747 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291a42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.187+0000 7fb123400640 1 -- 172.21.15.40:0/677010747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 msgr2=0x556e291a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.187+0000 7fb123400640 1 --2- 172.21.15.40:0/677010747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 0x556e291a6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.187+0000 7fb123400640 1 -- 172.21.15.40:0/677010747 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e290c1940 con 0x556e2919ede0 2024-04-27T00:08:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.187+0000 7fb123400640 1 --2- 172.21.15.40:0/677010747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e2919f200 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009ca0 tx=0x7fb11c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ae5f01cb056ff19 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.187+0000 7fb11be00640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb1080a3cb0 0x7fb1080a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.187+0000 7fb113e00640 1 -- 172.21.15.40:0/677010747 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c01e070 con 0x556e2919ede0 2024-04-27T00:08:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.187+0000 7fb11be00640 1 -- 172.21.15.40:0/2323256870 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fb10c0d0ed0 con 0x7fb124105be0 2024-04-27T00:08:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.187+0000 7fb113e00640 1 -- 172.21.15.40:0/677010747 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb11c009e50 con 0x556e2919ede0 2024-04-27T00:08:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.187+0000 7fb113e00640 1 -- 172.21.15.40:0/677010747 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c00ea10 con 0x556e2919ede0 2024-04-27T00:08:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.188+0000 7fb122a00640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb1080a3cb0 0x7fb1080a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.188+0000 7fb12c57a740 1 -- 172.21.15.40:0/677010747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 msgr2=0x556e2919f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:38.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.188+0000 7fb12c57a740 1 --2- 172.21.15.40:0/677010747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e2919f200 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009ca0 tx=0x7fb11c011810 comp rx=0 tx=0).stop 2024-04-27T00:08:38.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.188+0000 7fb122a00640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb1080a3cb0 0x7fb1080a6170 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb114009230 tx=0x7fb1140314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:38.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.188+0000 7fb12c57a740 1 -- 172.21.15.40:0/677010747 shutdown_connections 2024-04-27T00:08:38.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.188+0000 7fb12c57a740 1 --2- 172.21.15.40:0/677010747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 0x556e291a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.188+0000 7fb12c57a740 1 --2- 172.21.15.40:0/677010747 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291a42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.188+0000 7fb12c57a740 1 --2- 172.21.15.40:0/677010747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e2919f200 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.188+0000 7fb12c57a740 1 -- 172.21.15.40:0/677010747 >> 172.21.15.40:0/677010747 conn(0x556e2919a450 msgr2=0x556e2919a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.188+0000 7fb12c57a740 1 -- 172.21.15.40:0/677010747 shutdown_connections 2024-04-27T00:08:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.188+0000 7fb12c57a740 1 -- 172.21.15.40:0/677010747 wait complete. 2024-04-27T00:08:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.189+0000 7fb12c57a740 1 -- start start 2024-04-27T00:08:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.189+0000 7fb12c57a740 1 Processor -- start 2024-04-27T00:08:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb12c57a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e2919ede0 0x556e291a90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb12c57a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291a95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb12c57a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e291a4940 0x556e291b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb12c57a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556e291a78f0 con 0x556e2919ede0 2024-04-27T00:08:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb12c57a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x556e2908ac60 con 0x556e2919f7d0 2024-04-27T00:08:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb12c57a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556e291a7a70 con 0x556e291a4940 2024-04-27T00:08:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb12c57a740 10 client.0 _ll_register_callbacks cb 0x556e290b6e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:08:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb122a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e2919ede0 0x556e291a90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb123400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e291a4940 0x556e291b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb11a000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291a95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb123400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e291a4940 0x556e291b05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:59690/0 (socket says 172.21.15.40:59690) 2024-04-27T00:08:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb123400640 1 -- 172.21.15.40:0/1800468583 learned_addr learned my addr 172.21.15.40:0/1800468583 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb123400640 1 -- 172.21.15.40:0/1800468583 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 msgr2=0x556e291a95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb123400640 1 --2- 172.21.15.40:0/1800468583 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291a95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb123400640 1 -- 172.21.15.40:0/1800468583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e2919ede0 msgr2=0x556e291a90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb123400640 1 --2- 172.21.15.40:0/1800468583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e2919ede0 0x556e291a90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.190+0000 7fb123400640 1 -- 172.21.15.40:0/1800468583 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e29099510 con 0x556e291a4940 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.191+0000 7fb123400640 1 --2- 172.21.15.40:0/1800468583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e291a4940 0x556e291b05c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb11c011d10 tx=0x7fb11c00e430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47b697672a42c8ee server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.191+0000 7fb110c00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.191+0000 7fb110c00640 1 -- 172.21.15.40:0/1800468583 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c01e030 con 0x556e291a4940 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.191+0000 7fb110c00640 1 -- 172.21.15.40:0/1800468583 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fb11c005460 con 0x556e291a4940 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.191+0000 7fb12c57a740 1 -- 172.21.15.40:0/1800468583 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb11c009e50 con 0x556e291a4940 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.191+0000 7fb110c00640 1 -- 172.21.15.40:0/1800468583 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb11c031050 con 0x556e291a4940 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.191+0000 7fb12c57a740 10 client.25951 fetch_fsmap learned FSMap version 24 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.191+0000 7fb12c57a740 1 -- 172.21.15.40:0/1800468583 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556e28e15210 con 0x556e291a4940 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.192+0000 7fb110c00640 1 -- 172.21.15.40:0/1800468583 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb11c009e50 con 0x556e291a4940 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.192+0000 7fb110c00640 1 -- 172.21.15.40:0/1800468583 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb11c004a60 con 0x556e291a4940 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.192+0000 7fb12c57a740 10 client.25951 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:08:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.192+0000 7fb12c57a740 10 client.25951 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:08:38.202 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.192+0000 7fb12c57a740 20 client.25951 populate_metadata read hostname 'smithi040' 2024-04-27T00:08:38.202 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.192+0000 7fb12c57a740 1 --2- 172.21.15.40:0/1800468583 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556e291b4930 0x556e291b6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.202 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.192+0000 7fb12c57a740 4 client.25951 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:08:38.202 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.192+0000 7fb12c57a740 1 -- 172.21.15.40:0/1800468583 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e29110f90 con 0x556e291b4930 2024-04-27T00:08:38.202 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.193+0000 7fb122a00640 1 --2- 172.21.15.40:0/1800468583 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556e291b4930 0x556e291b6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.202 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.193+0000 7fb122a00640 1 --2- 172.21.15.40:0/1800468583 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556e291b4930 0x556e291b6d20 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c62a8e050f06339 server_cookie=fa769ad91789e26c in_seq=0 out_seq=0 2024-04-27T00:08:38.202 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.193+0000 7fb110c00640 10 client.25951 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:38.202 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb110c00640 1 -- 172.21.15.40:0/1800468583 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e29110f90 con 0x556e291b4930 2024-04-27T00:08:38.202 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb110c00640 10 client.25951 handle_command_reply: tid=0 2024-04-27T00:08:38.202 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb12c57a740 1 client.25951 shutdown 2024-04-27T00:08:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb12c57a740 1 -- 172.21.15.40:0/1800468583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e291a4940 msgr2=0x556e291b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb12c57a740 1 --2- 172.21.15.40:0/1800468583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e291a4940 0x556e291b05c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb11c011d10 tx=0x7fb11c00e430 comp rx=0 tx=0).stop 2024-04-27T00:08:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb12c57a740 1 -- 172.21.15.40:0/1800468583 shutdown_connections 2024-04-27T00:08:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb12c57a740 1 --2- 172.21.15.40:0/1800468583 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556e291b4930 0x556e291b6d20 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb12c57a740 1 --2- 172.21.15.40:0/1800468583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e291a4940 0x556e291b05c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb12c57a740 1 --2- 172.21.15.40:0/1800468583 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291a95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb12c57a740 1 --2- 172.21.15.40:0/1800468583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e2919ede0 0x556e291a90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb12c57a740 1 -- 172.21.15.40:0/1800468583 >> 172.21.15.40:0/1800468583 conn(0x556e2919a450 msgr2=0x556e29199650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb110c00640 0 client.25951 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb12c57a740 1 -- 172.21.15.40:0/1800468583 shutdown_connections 2024-04-27T00:08:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb12c57a740 1 -- 172.21.15.40:0/1800468583 wait complete. 2024-04-27T00:08:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.194+0000 7fb12c57a740 20 client.25951 trim_cache size 0 max 16384 2024-04-27T00:08:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.211+0000 7fb12c57a740 1 Processor -- start 2024-04-27T00:08:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.211+0000 7fb12c57a740 1 -- start start 2024-04-27T00:08:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.211+0000 7fb12c57a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e29198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.211+0000 7fb12c57a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291b0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.211+0000 7fb12c57a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 0x556e29199300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.211+0000 7fb12c57a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241afdc0 con 0x556e291a4940 2024-04-27T00:08:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.211+0000 7fb12c57a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241ad690 con 0x556e2919f7d0 2024-04-27T00:08:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.211+0000 7fb12c57a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241ab4e0 con 0x556e2919ede0 2024-04-27T00:08:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.212+0000 7fb123400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 0x556e29199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.212+0000 7fb123400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 0x556e29199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51918/0 (socket says 172.21.15.40:51918) 2024-04-27T00:08:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.212+0000 7fb123400640 1 -- 172.21.15.40:0/2584918995 learned_addr learned my addr 172.21.15.40:0/2584918995 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.212+0000 7fb122a00640 1 --2- 172.21.15.40:0/2584918995 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e29198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.212+0000 7fb11a000640 1 --2- 172.21.15.40:0/2584918995 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.212+0000 7fb123400640 1 -- 172.21.15.40:0/2584918995 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 msgr2=0x556e29198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.212+0000 7fb123400640 1 --2- 172.21.15.40:0/2584918995 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e29198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.212+0000 7fb123400640 1 -- 172.21.15.40:0/2584918995 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 msgr2=0x556e291b0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.212+0000 7fb123400640 1 --2- 172.21.15.40:0/2584918995 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291b0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.212+0000 7fb123400640 1 -- 172.21.15.40:0/2584918995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e28ee1620 con 0x556e291a4940 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.212+0000 7fb123400640 1 --2- 172.21.15.40:0/2584918995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 0x556e29199300 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb11c005d00 tx=0x7fb11c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f2fadd429dc0b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.213+0000 7fb113400640 1 -- 172.21.15.40:0/2584918995 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c03a070 con 0x556e291a4940 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.213+0000 7fb113400640 1 -- 172.21.15.40:0/2584918995 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb11c028070 con 0x556e291a4940 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.213+0000 7fb113400640 1 -- 172.21.15.40:0/2584918995 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c01e040 con 0x556e291a4940 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.213+0000 7fb12c57a740 1 -- 172.21.15.40:0/2584918995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 msgr2=0x556e29199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.213+0000 7fb12c57a740 1 --2- 172.21.15.40:0/2584918995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 0x556e29199300 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb11c005d00 tx=0x7fb11c005f90 comp rx=0 tx=0).stop 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.213+0000 7fb12c57a740 1 -- 172.21.15.40:0/2584918995 shutdown_connections 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.213+0000 7fb12c57a740 1 --2- 172.21.15.40:0/2584918995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e291a4940 0x556e29199300 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.213+0000 7fb12c57a740 1 --2- 172.21.15.40:0/2584918995 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e2919f7d0 0x556e291b0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.213+0000 7fb12c57a740 1 --2- 172.21.15.40:0/2584918995 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e29198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.213+0000 7fb12c57a740 1 -- 172.21.15.40:0/2584918995 >> 172.21.15.40:0/2584918995 conn(0x556e2919a450 msgr2=0x556e291a90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.214+0000 7fb12c57a740 1 -- 172.21.15.40:0/2584918995 shutdown_connections 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.214+0000 7fb12c57a740 1 -- 172.21.15.40:0/2584918995 wait complete. 2024-04-27T00:08:38.219 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.214+0000 7fb12c57a740 1 -- start start 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.214+0000 7fb12c57a740 1 Processor -- start 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.214+0000 7fb12c57a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e291ad4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb12c57a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e2919f7d0 0x556e291ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb12c57a740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e291a4940 0x556e29218e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb12c57a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241abb50 con 0x556e2919f7d0 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb12c57a740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241acfc0 con 0x556e291a4940 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb12c57a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241ac220 con 0x556e2919ede0 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb12c57a740 10 client.0 _ll_register_callbacks cb 0x7fb11c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb11a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e2919f7d0 0x556e291ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb122a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e291ad4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb11a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e2919f7d0 0x556e291ad9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51924/0 (socket says 172.21.15.40:51924) 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb11a000640 1 -- 172.21.15.40:0/1617866811 learned_addr learned my addr 172.21.15.40:0/1617866811 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:38.220 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb123400640 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e291a4940 0x556e29218e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb11a000640 1 -- 172.21.15.40:0/1617866811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 msgr2=0x556e291ad4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb11a000640 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e291ad4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb11a000640 1 -- 172.21.15.40:0/1617866811 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e291a4940 msgr2=0x556e29218e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb11a000640 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e291a4940 0x556e29218e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb11a000640 1 -- 172.21.15.40:0/1617866811 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e290bdcb0 con 0x556e2919f7d0 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb123400640 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e291a4940 0x556e29218e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.215+0000 7fb122a00640 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e291ad4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.216+0000 7fb11a000640 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e2919f7d0 0x556e291ad9e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb10c09ca00 tx=0x7fb10c00efa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5529b90ff7c40c94 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.216+0000 7fb110c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.216+0000 7fb110c00640 1 -- 172.21.15.40:0/1617866811 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c09c660 con 0x556e2919f7d0 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.216+0000 7fb110c00640 1 -- 172.21.15.40:0/1617866811 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fb10c0d4020 con 0x556e2919f7d0 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.216+0000 7fb12c57a740 1 -- 172.21.15.40:0/1617866811 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb11c028070 con 0x556e2919f7d0 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.216+0000 7fb110c00640 1 -- 172.21.15.40:0/1617866811 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb10c018420 con 0x556e2919f7d0 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.216+0000 7fb12c57a740 10 client.15816 fetch_fsmap learned FSMap version 24 2024-04-27T00:08:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.216+0000 7fb12c57a740 1 -- 172.21.15.40:0/1617866811 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556e2921a560 con 0x556e2919f7d0 2024-04-27T00:08:38.222 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.216+0000 7fb110c00640 1 -- 172.21.15.40:0/1617866811 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb11c028070 con 0x556e2919f7d0 2024-04-27T00:08:38.222 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.216+0000 7fb110c00640 1 -- 172.21.15.40:0/1617866811 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb10c09c150 con 0x556e2919f7d0 2024-04-27T00:08:38.222 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.217+0000 7fb12c57a740 10 client.15816 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:08:38.222 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.217+0000 7fb12c57a740 10 client.15816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:08:38.222 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.217+0000 7fb12c57a740 20 client.15816 populate_metadata read hostname 'smithi040' 2024-04-27T00:08:38.222 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.217+0000 7fb12c57a740 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556e2921af10 0x556e2921d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:38.222 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.217+0000 7fb12c57a740 4 client.15816 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:08:38.222 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.217+0000 7fb12c57a740 1 -- 172.21.15.40:0/1617866811 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556e291a4670 con 0x556e2921af10 2024-04-27T00:08:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.217+0000 7fb122a00640 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556e2921af10 0x556e2921d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.217+0000 7fb122a00640 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556e2921af10 0x556e2921d330 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e806b6a5016afad5 server_cookie=d14076f27983e0ba in_seq=0 out_seq=0 2024-04-27T00:08:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.218+0000 7fb110c00640 10 client.15816 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb110c00640 1 -- 172.21.15.40:0/1617866811 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556e291a4670 con 0x556e2921af10 2024-04-27T00:08:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb110c00640 10 client.15816 handle_command_reply: tid=0 2024-04-27T00:08:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb12c57a740 1 client.15816 shutdown 2024-04-27T00:08:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb12c57a740 1 -- 172.21.15.40:0/1617866811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e2919f7d0 msgr2=0x556e291ad9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb12c57a740 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e2919f7d0 0x556e291ad9e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb10c09ca00 tx=0x7fb10c00efa0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb12c57a740 1 -- 172.21.15.40:0/1617866811 shutdown_connections 2024-04-27T00:08:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb12c57a740 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x556e2921af10 0x556e2921d330 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb12c57a740 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x556e291a4940 0x556e29218e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb110c00640 0 client.15816 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb12c57a740 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556e2919f7d0 0x556e291ad9e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb12c57a740 1 --2- 172.21.15.40:0/1617866811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556e2919ede0 0x556e291ad4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb12c57a740 1 -- 172.21.15.40:0/1617866811 >> 172.21.15.40:0/1617866811 conn(0x556e2919a450 msgr2=0x556e2919d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb12c57a740 1 -- 172.21.15.40:0/1617866811 shutdown_connections 2024-04-27T00:08:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb12c57a740 1 -- 172.21.15.40:0/1617866811 wait complete. 2024-04-27T00:08:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.233+0000 7fb12c57a740 20 client.15816 trim_cache size 0 max 16384 2024-04-27T00:08:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:08:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:08:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:08:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:08:38.240 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.234+0000 7fb123e00640 1 -- 172.21.15.40:0/2323256870 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb1080a3cb0 msgr2=0x7fb1080a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:38.240 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.234+0000 7fb123e00640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb1080a3cb0 0x7fb1080a6170 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb114009230 tx=0x7fb1140314a0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.240 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.234+0000 7fb123e00640 1 -- 172.21.15.40:0/2323256870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb124105be0 msgr2=0x7fb12419b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:38.240 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.234+0000 7fb123e00640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb124105be0 0x7fb12419b130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb10c00ebc0 tx=0x7fb10c00c6a0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.240 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.234+0000 7fb123e00640 1 -- 172.21.15.40:0/2323256870 shutdown_connections 2024-04-27T00:08:38.240 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.234+0000 7fb123e00640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fb1241065a0 0x7fb1241a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.234+0000 7fb123e00640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fb1080a3cb0 0x7fb1080a6170 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.234+0000 7fb123e00640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb124105be0 0x7fb12419b130 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.234+0000 7fb123e00640 1 --2- 172.21.15.40:0/2323256870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1241049f0 0x7fb12419abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:38.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.234+0000 7fb123e00640 1 -- 172.21.15.40:0/2323256870 >> 172.21.15.40:0/2323256870 conn(0x7fb1240fffa0 msgr2=0x7fb124108fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:38.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.235+0000 7fb123e00640 1 -- 172.21.15.40:0/2323256870 shutdown_connections 2024-04-27T00:08:38.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:38.235+0000 7fb123e00640 1 -- 172.21.15.40:0/2323256870 wait complete. 2024-04-27T00:08:38.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:38 smithi040 ceph-mon[28095]: pgmap v620: 129 pgs: 129 active+clean; 4.5 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-04-27T00:08:38.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:38 smithi080 ceph-mon[32362]: pgmap v620: 129 pgs: 129 active+clean; 4.5 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-04-27T00:08:38.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:38 smithi103 ceph-mon[32952]: pgmap v620: 129 pgs: 129 active+clean; 4.5 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-04-27T00:08:40.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:40 smithi040 ceph-mon[28095]: pgmap v621: 129 pgs: 129 active+clean; 4.5 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-27T00:08:40.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:40 smithi080 ceph-mon[32362]: pgmap v621: 129 pgs: 129 active+clean; 4.5 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-27T00:08:40.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:40 smithi103 ceph-mon[32952]: pgmap v621: 129 pgs: 129 active+clean; 4.5 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-27T00:08:42.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:42 smithi040 ceph-mon[28095]: pgmap v622: 129 pgs: 129 active+clean; 4.6 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-27T00:08:42.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:42 smithi080 ceph-mon[32362]: pgmap v622: 129 pgs: 129 active+clean; 4.6 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-27T00:08:42.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:42 smithi103 ceph-mon[32952]: pgmap v622: 129 pgs: 129 active+clean; 4.6 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-04-27T00:08:44.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:44 smithi040 ceph-mon[28095]: pgmap v623: 129 pgs: 129 active+clean; 4.6 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-04-27T00:08:44.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:44 smithi080 ceph-mon[32362]: pgmap v623: 129 pgs: 129 active+clean; 4.6 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-04-27T00:08:44.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:44 smithi103 ceph-mon[32952]: pgmap v623: 129 pgs: 129 active+clean; 4.6 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 15 op/s 2024-04-27T00:08:46.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:46 smithi040 ceph-mon[28095]: pgmap v624: 129 pgs: 129 active+clean; 4.6 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:08:46.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:46 smithi080 ceph-mon[32362]: pgmap v624: 129 pgs: 129 active+clean; 4.6 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:08:46.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:46 smithi103 ceph-mon[32952]: pgmap v624: 129 pgs: 129 active+clean; 4.6 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:08:48.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:48 smithi040 ceph-mon[28095]: pgmap v625: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:08:48.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:48 smithi080 ceph-mon[32362]: pgmap v625: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:08:48.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:48 smithi103 ceph-mon[32952]: pgmap v625: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:08:49.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:49 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:08:49.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:49 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:08:49.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:49 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:08:50.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:50 smithi040 ceph-mon[28095]: pgmap v626: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:08:50.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:50 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:08:50.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:50 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:08:50.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:50 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:08:50.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:50 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:08:50.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:50 smithi080 ceph-mon[32362]: pgmap v626: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:08:50.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:50 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:08:50.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:50 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:08:50.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:50 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:08:50.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:50 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:08:50.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:50 smithi103 ceph-mon[32952]: pgmap v626: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:08:50.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:50 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:08:50.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:50 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:08:50.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:50 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:08:50.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:50 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:08:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.383+0000 7fdbdea00640 1 -- 172.21.15.40:0/874414980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd81065a0 msgr2=0x7fdbd810ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.383+0000 7fdbdea00640 1 --2- 172.21.15.40:0/874414980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd81065a0 0x7fdbd810ce30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009ca0 tx=0x7fdbd002a8c0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.383+0000 7fdbdea00640 1 -- 172.21.15.40:0/874414980 shutdown_connections 2024-04-27T00:08:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.383+0000 7fdbdea00640 1 --2- 172.21.15.40:0/874414980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd81065a0 0x7fdbd810ce30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.383+0000 7fdbdea00640 1 --2- 172.21.15.40:0/874414980 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdbd8105be0 0x7fdbd8106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.383+0000 7fdbdea00640 1 --2- 172.21.15.40:0/874414980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd81049f0 0x7fdbd8104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.383+0000 7fdbdea00640 1 -- 172.21.15.40:0/874414980 >> 172.21.15.40:0/874414980 conn(0x7fdbd80fff60 msgr2=0x7fdbd81023a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.383+0000 7fdbdea00640 1 -- 172.21.15.40:0/874414980 shutdown_connections 2024-04-27T00:08:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.383+0000 7fdbdea00640 1 -- 172.21.15.40:0/874414980 wait complete. 2024-04-27T00:08:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.384+0000 7fdbdea00640 1 Processor -- start 2024-04-27T00:08:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.384+0000 7fdbdea00640 1 -- start start 2024-04-27T00:08:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.385+0000 7fdbdea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd81049f0 0x7fdbd819ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.385+0000 7fdbdea00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdbd8105be0 0x7fdbd819b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.385+0000 7fdbdea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd81065a0 0x7fdbd81a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.385+0000 7fdbdea00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd8110570 con 0x7fdbd81049f0 2024-04-27T00:08:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.385+0000 7fdbdea00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd81103f0 con 0x7fdbd8105be0 2024-04-27T00:08:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.385+0000 7fdbdea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd81106f0 con 0x7fdbd81065a0 2024-04-27T00:08:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.385+0000 7fdbdd600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd81049f0 0x7fdbd819ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.385+0000 7fdbde000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd81065a0 0x7fdbd81a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.385+0000 7fdbde000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd81065a0 0x7fdbd81a2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37238/0 (socket says 172.21.15.40:37238) 2024-04-27T00:08:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.385+0000 7fdbde000640 1 -- 172.21.15.40:0/2017471157 learned_addr learned my addr 172.21.15.40:0/2017471157 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.385+0000 7fdbdcc00640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdbd8105be0 0x7fdbd819b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.386+0000 7fdbdcc00640 1 -- 172.21.15.40:0/2017471157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd81065a0 msgr2=0x7fdbd81a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.386+0000 7fdbdcc00640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd81065a0 0x7fdbd81a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.386+0000 7fdbdcc00640 1 -- 172.21.15.40:0/2017471157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd81049f0 msgr2=0x7fdbd819ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.386+0000 7fdbdcc00640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd81049f0 0x7fdbd819ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.386+0000 7fdbdcc00640 1 -- 172.21.15.40:0/2017471157 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd81a27a0 con 0x7fdbd8105be0 2024-04-27T00:08:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.386+0000 7fdbdd600640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd81049f0 0x7fdbd819ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:08:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.386+0000 7fdbde000640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd81065a0 0x7fdbd81a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:08:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.386+0000 7fdbdcc00640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdbd8105be0 0x7fdbd819b0c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400e9f0 tx=0x7fdbc400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.386+0000 7fdbd6000640 1 -- 172.21.15.40:0/2017471157 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc400cd80 con 0x7fdbd8105be0 2024-04-27T00:08:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.386+0000 7fdbdea00640 1 -- 172.21.15.40:0/2017471157 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd81a2a90 con 0x7fdbd8105be0 2024-04-27T00:08:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.386+0000 7fdbd6000640 1 -- 172.21.15.40:0/2017471157 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbc40035f0 con 0x7fdbd8105be0 2024-04-27T00:08:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.386+0000 7fdbd6000640 1 -- 172.21.15.40:0/2017471157 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4010690 con 0x7fdbd8105be0 2024-04-27T00:08:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.387+0000 7fdbdea00640 1 -- 172.21.15.40:0/2017471157 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbd81a3030 con 0x7fdbd8105be0 2024-04-27T00:08:51.390 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbe2e60740 1 Processor -- start 2024-04-27T00:08:51.390 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbe2e60740 1 -- start start 2024-04-27T00:08:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbe2e60740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e01465f0 0x55d1e0146a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbe2e60740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e014bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbe2e60740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e014c150 0x55d1e014e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbe2e60740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d1e00a2380 con 0x55d1e01465f0 2024-04-27T00:08:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbe2e60740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55d1e003ec30 con 0x55d1e0146fe0 2024-04-27T00:08:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbe2e60740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d1e009e340 con 0x55d1e014c150 2024-04-27T00:08:51.392 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbd6000640 1 -- 172.21.15.40:0/2017471157 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdbc40037a0 con 0x7fdbd8105be0 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbde000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e01465f0 0x55d1e0146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbdd600640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e014bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbdd600640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e014bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49726/0 (socket says 172.21.15.40:49726) 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.390+0000 7fdbdd600640 1 -- 172.21.15.40:0/3732899844 learned_addr learned my addr 172.21.15.40:0/3732899844 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.391+0000 7fdbdcc00640 1 --2- 172.21.15.40:0/3732899844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e014c150 0x55d1e014e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.391+0000 7fdbdd600640 1 -- 172.21.15.40:0/3732899844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e014c150 msgr2=0x55d1e014e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.391+0000 7fdbdd600640 1 --2- 172.21.15.40:0/3732899844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e014c150 0x55d1e014e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.391+0000 7fdbdd600640 1 -- 172.21.15.40:0/3732899844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e01465f0 msgr2=0x55d1e0146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.391+0000 7fdbdd600640 1 --2- 172.21.15.40:0/3732899844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e01465f0 0x55d1e0146a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.391+0000 7fdbdd600640 1 -- 172.21.15.40:0/3732899844 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1e0060940 con 0x55d1e0146fe0 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.391+0000 7fdbdd600640 1 --2- 172.21.15.40:0/3732899844 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e014bae0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0114f0 tx=0x7fdbcc0119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca936a757e1a46bf server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.391+0000 7fdbca600640 1 -- 172.21.15.40:0/3732899844 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc01e070 con 0x55d1e0146fe0 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.392+0000 7fdbca600640 1 -- 172.21.15.40:0/3732899844 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbcc0035f0 con 0x55d1e0146fe0 2024-04-27T00:08:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.392+0000 7fdbca600640 1 -- 172.21.15.40:0/3732899844 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc00f7c0 con 0x55d1e0146fe0 2024-04-27T00:08:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.392+0000 7fdbd6000640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdbc00a3cb0 0x7fdbc00a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.392+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3732899844 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 msgr2=0x55d1e014bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.392+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/3732899844 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e014bae0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0114f0 tx=0x7fdbcc0119b0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.392+0000 7fdbd6000640 1 -- 172.21.15.40:0/2017471157 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fdbc40d0bc0 con 0x7fdbd8105be0 2024-04-27T00:08:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.393+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3732899844 shutdown_connections 2024-04-27T00:08:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.393+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/3732899844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e014c150 0x55d1e014e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.393+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/3732899844 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e014bae0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.393+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/3732899844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e01465f0 0x55d1e0146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.393+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3732899844 >> 172.21.15.40:0/3732899844 conn(0x55d1e0139450 msgr2=0x55d1e0139850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.393+0000 7fdbde000640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdbc00a3cb0 0x7fdbc00a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.393+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3732899844 shutdown_connections 2024-04-27T00:08:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.393+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3732899844 wait complete. 2024-04-27T00:08:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.393+0000 7fdbe2e60740 1 -- start start 2024-04-27T00:08:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.393+0000 7fdbde000640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdbc00a3cb0 0x7fdbc00a6170 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x55d1e0147840 tx=0x7fdbd0031af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.393+0000 7fdbe2e60740 1 Processor -- start 2024-04-27T00:08:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.394+0000 7fdbe2e60740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e01465f0 0x55d1e0150810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.394+0000 7fdbe2e60740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e0150d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.394+0000 7fdbe2e60740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e014c150 0x55d1e0157d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.394+0000 7fdbe2e60740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d1e014f100 con 0x55d1e014c150 2024-04-27T00:08:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.394+0000 7fdbe2e60740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55d1e0029c60 con 0x55d1e0146fe0 2024-04-27T00:08:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.394+0000 7fdbe2e60740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d1e014f280 con 0x55d1e01465f0 2024-04-27T00:08:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.394+0000 7fdbe2e60740 10 client.0 _ll_register_callbacks cb 0x55d1e0055e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:08:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.394+0000 7fdbde000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e014c150 0x55d1e0157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.394+0000 7fdbdd600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e01465f0 0x55d1e0150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.394+0000 7fdbdd600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e01465f0 0x55d1e0150810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37256/0 (socket says 172.21.15.40:37256) 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.394+0000 7fdbdd600640 1 -- 172.21.15.40:0/553458330 learned_addr learned my addr 172.21.15.40:0/553458330 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.394+0000 7fdbdcc00640 1 --2- 172.21.15.40:0/553458330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e0150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.395+0000 7fdbdd600640 1 -- 172.21.15.40:0/553458330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 msgr2=0x55d1e0150d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.395+0000 7fdbdd600640 1 --2- 172.21.15.40:0/553458330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e0150d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.395+0000 7fdbdd600640 1 -- 172.21.15.40:0/553458330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e014c150 msgr2=0x55d1e0157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.395+0000 7fdbdd600640 1 --2- 172.21.15.40:0/553458330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e014c150 0x55d1e0157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.395+0000 7fdbdd600640 1 -- 172.21.15.40:0/553458330 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1e0038510 con 0x55d1e01465f0 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.395+0000 7fdbdd600640 1 --2- 172.21.15.40:0/553458330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e01465f0 0x55d1e0150810 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0114c0 tx=0x7fdbcc0042b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d119924e87599c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.395+0000 7fdbbea00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.395+0000 7fdbbea00640 1 -- 172.21.15.40:0/553458330 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc026020 con 0x55d1e01465f0 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.396+0000 7fdbe2e60740 1 -- 172.21.15.40:0/553458330 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbcc0035f0 con 0x55d1e01465f0 2024-04-27T00:08:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.396+0000 7fdbbea00640 1 -- 172.21.15.40:0/553458330 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fdbcc005040 con 0x55d1e01465f0 2024-04-27T00:08:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.396+0000 7fdbbea00640 1 -- 172.21.15.40:0/553458330 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdbcc031050 con 0x55d1e01465f0 2024-04-27T00:08:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.396+0000 7fdbe2e60740 10 client.25969 fetch_fsmap learned FSMap version 24 2024-04-27T00:08:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.396+0000 7fdbe2e60740 1 -- 172.21.15.40:0/553458330 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d1dfdb4210 con 0x55d1e01465f0 2024-04-27T00:08:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.396+0000 7fdbbea00640 1 -- 172.21.15.40:0/553458330 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbcc0035f0 con 0x55d1e01465f0 2024-04-27T00:08:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.397+0000 7fdbbea00640 1 -- 172.21.15.40:0/553458330 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdbcc021030 con 0x55d1e01465f0 2024-04-27T00:08:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.397+0000 7fdbe2e60740 10 client.25969 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:08:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.397+0000 7fdbe2e60740 10 client.25969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:08:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.397+0000 7fdbe2e60740 20 client.25969 populate_metadata read hostname 'smithi040' 2024-04-27T00:08:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.397+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/553458330 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d1e015c060 0x55d1e015e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.397+0000 7fdbe2e60740 4 client.25969 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:08:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.397+0000 7fdbe2e60740 1 -- 172.21.15.40:0/553458330 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1e00aff90 con 0x55d1e015c060 2024-04-27T00:08:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.397+0000 7fdbdcc00640 1 --2- 172.21.15.40:0/553458330 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d1e015c060 0x55d1e015e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.398+0000 7fdbdcc00640 1 --2- 172.21.15.40:0/553458330 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d1e015c060 0x55d1e015e490 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35962c48d612e25e server_cookie=a89c3fbcb1ca9c21 in_seq=0 out_seq=0 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.398+0000 7fdbbea00640 10 client.25969 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbbea00640 1 -- 172.21.15.40:0/553458330 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d1e00aff90 con 0x55d1e015c060 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbbea00640 10 client.25969 handle_command_reply: tid=0 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbe2e60740 1 client.25969 shutdown 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbe2e60740 1 -- 172.21.15.40:0/553458330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e01465f0 msgr2=0x55d1e0150810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/553458330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e01465f0 0x55d1e0150810 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0114c0 tx=0x7fdbcc0042b0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbe2e60740 1 -- 172.21.15.40:0/553458330 shutdown_connections 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/553458330 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d1e015c060 0x55d1e015e490 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/553458330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e014c150 0x55d1e0157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/553458330 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e0150d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbbea00640 0 client.25969 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/553458330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e01465f0 0x55d1e0150810 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbe2e60740 1 -- 172.21.15.40:0/553458330 >> 172.21.15.40:0/553458330 conn(0x55d1e0139450 msgr2=0x55d1e0138650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbe2e60740 1 -- 172.21.15.40:0/553458330 shutdown_connections 2024-04-27T00:08:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.399+0000 7fdbe2e60740 1 -- 172.21.15.40:0/553458330 wait complete. 2024-04-27T00:08:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.400+0000 7fdbe2e60740 20 client.25969 trim_cache size 0 max 16384 2024-04-27T00:08:51.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.416+0000 7fdbe2e60740 1 Processor -- start 2024-04-27T00:08:51.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.416+0000 7fdbe2e60740 1 -- start start 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.416+0000 7fdbe2e60740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e01465f0 0x55d1e013cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.416+0000 7fdbe2e60740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e0146fe0 0x55d1e0157d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.416+0000 7fdbe2e60740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e014c150 0x55d1e014fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbe2e60740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd81ab9e0 con 0x55d1e014c150 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbe2e60740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd81a92b0 con 0x55d1e01465f0 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbe2e60740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd81a7100 con 0x55d1e0146fe0 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbde000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e014c150 0x55d1e014fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbde000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e014c150 0x55d1e014fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55476/0 (socket says 172.21.15.40:55476) 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbde000640 1 -- 172.21.15.40:0/2111295154 learned_addr learned my addr 172.21.15.40:0/2111295154 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbdcc00640 1 --2- 172.21.15.40:0/2111295154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e0146fe0 0x55d1e0157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbdd600640 1 --2- 172.21.15.40:0/2111295154 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e01465f0 0x55d1e013cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbde000640 1 -- 172.21.15.40:0/2111295154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e0146fe0 msgr2=0x55d1e0157d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:51.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbde000640 1 --2- 172.21.15.40:0/2111295154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e0146fe0 0x55d1e0157d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbde000640 1 -- 172.21.15.40:0/2111295154 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e01465f0 msgr2=0x55d1e013cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbde000640 1 --2- 172.21.15.40:0/2111295154 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e01465f0 0x55d1e013cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.417+0000 7fdbde000640 1 -- 172.21.15.40:0/2111295154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1dfe80620 con 0x55d1e014c150 2024-04-27T00:08:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.418+0000 7fdbde000640 1 --2- 172.21.15.40:0/2111295154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e014c150 0x55d1e014fe30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0031710 tx=0x7fdbd0042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e75cb1f94ff7ebe server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.418+0000 7fdbc9c00640 1 -- 172.21.15.40:0/2111295154 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0003af0 con 0x55d1e014c150 2024-04-27T00:08:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.418+0000 7fdbc9c00640 1 -- 172.21.15.40:0/2111295154 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbd0036040 con 0x55d1e014c150 2024-04-27T00:08:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.418+0000 7fdbc9c00640 1 -- 172.21.15.40:0/2111295154 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd002f610 con 0x55d1e014c150 2024-04-27T00:08:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.418+0000 7fdbe2e60740 1 -- 172.21.15.40:0/2111295154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e014c150 msgr2=0x55d1e014fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.419+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/2111295154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e014c150 0x55d1e014fe30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0031710 tx=0x7fdbd0042000 comp rx=0 tx=0).stop 2024-04-27T00:08:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.419+0000 7fdbe2e60740 1 -- 172.21.15.40:0/2111295154 shutdown_connections 2024-04-27T00:08:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.419+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/2111295154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e014c150 0x55d1e014fe30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.419+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/2111295154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e0146fe0 0x55d1e0157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.419+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/2111295154 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e01465f0 0x55d1e013cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.419+0000 7fdbe2e60740 1 -- 172.21.15.40:0/2111295154 >> 172.21.15.40:0/2111295154 conn(0x55d1e0139450 msgr2=0x55d1e013c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.419+0000 7fdbe2e60740 1 -- 172.21.15.40:0/2111295154 shutdown_connections 2024-04-27T00:08:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.419+0000 7fdbe2e60740 1 -- 172.21.15.40:0/2111295154 wait complete. 2024-04-27T00:08:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.419+0000 7fdbe2e60740 1 -- start start 2024-04-27T00:08:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.419+0000 7fdbe2e60740 1 Processor -- start 2024-04-27T00:08:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbe2e60740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e01465f0 0x55d1e0155620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbe2e60740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e0155b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbe2e60740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e014c150 0x55d1e0156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbe2e60740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd81a7770 con 0x55d1e01465f0 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbe2e60740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd81a8be0 con 0x55d1e0146fe0 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbe2e60740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd81a7e40 con 0x55d1e014c150 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbe2e60740 10 client.0 _ll_register_callbacks cb 0x7fdbcc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbdd600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e01465f0 0x55d1e0155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbde000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e014c150 0x55d1e0156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbdcc00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e0155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbdd600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e01465f0 0x55d1e0155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55478/0 (socket says 172.21.15.40:55478) 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbde000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e014c150 0x55d1e0156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37268/0 (socket says 172.21.15.40:37268) 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbdd600640 1 -- 172.21.15.40:0/3239495806 learned_addr learned my addr 172.21.15.40:0/3239495806 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.420+0000 7fdbdcc00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e0155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:49754/0 (socket says 172.21.15.40:49754) 2024-04-27T00:08:51.425 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.421+0000 7fdbdcc00640 1 -- 172.21.15.40:0/3239495806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e014c150 msgr2=0x55d1e0156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.421+0000 7fdbdcc00640 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e014c150 0x55d1e0156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.421+0000 7fdbdcc00640 1 -- 172.21.15.40:0/3239495806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e01465f0 msgr2=0x55d1e0155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.421+0000 7fdbdcc00640 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e01465f0 0x55d1e0155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.421+0000 7fdbdcc00640 1 -- 172.21.15.40:0/3239495806 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1e005ccb0 con 0x55d1e0146fe0 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.421+0000 7fdbde000640 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e014c150 0x55d1e0156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.421+0000 7fdbdd600640 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e01465f0 0x55d1e0155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.421+0000 7fdbdcc00640 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e0155b60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40ce170 tx=0x7fdbc40bd0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6313e1fe5065fb4b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.421+0000 7fdbbea00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.421+0000 7fdbbea00640 1 -- 172.21.15.40:0/3239495806 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4009310 con 0x55d1e0146fe0 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.421+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3239495806 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbd0036040 con 0x55d1e0146fe0 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.422+0000 7fdbbea00640 1 -- 172.21.15.40:0/3239495806 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fdbc4098fb0 con 0x55d1e0146fe0 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.422+0000 7fdbbea00640 1 -- 172.21.15.40:0/3239495806 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdbc40ce3c0 con 0x55d1e0146fe0 2024-04-27T00:08:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.422+0000 7fdbe2e60740 10 client.25946 fetch_fsmap learned FSMap version 24 2024-04-27T00:08:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.422+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3239495806 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d1e01c2680 con 0x55d1e0146fe0 2024-04-27T00:08:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.422+0000 7fdbbea00640 1 -- 172.21.15.40:0/3239495806 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbd0036040 con 0x55d1e0146fe0 2024-04-27T00:08:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.422+0000 7fdbbea00640 1 -- 172.21.15.40:0/3239495806 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdbc40bbb00 con 0x55d1e0146fe0 2024-04-27T00:08:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.422+0000 7fdbe2e60740 10 client.25946 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:08:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.422+0000 7fdbe2e60740 10 client.25946 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:08:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.422+0000 7fdbe2e60740 20 client.25946 populate_metadata read hostname 'smithi040' 2024-04-27T00:08:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.422+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d1e01c3030 0x55d1e01c5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:08:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.422+0000 7fdbe2e60740 4 client.25946 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:08:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.422+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3239495806 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d1e014bec0 con 0x55d1e01c3030 2024-04-27T00:08:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.423+0000 7fdbdd600640 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d1e01c3030 0x55d1e01c5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:08:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.423+0000 7fdbdd600640 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d1e01c3030 0x55d1e01c5450 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=847cfa99405f0eab server_cookie=294a030fcef67ea1 in_seq=0 out_seq=0 2024-04-27T00:08:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.423+0000 7fdbbea00640 10 client.25946 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.437+0000 7fdbbea00640 1 -- 172.21.15.40:0/3239495806 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d1e014bec0 con 0x55d1e01c3030 2024-04-27T00:08:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.437+0000 7fdbbea00640 10 client.25946 handle_command_reply: tid=0 2024-04-27T00:08:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.437+0000 7fdbe2e60740 1 client.25946 shutdown 2024-04-27T00:08:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.437+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3239495806 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 msgr2=0x55d1e0155b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.437+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e0155b60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40ce170 tx=0x7fdbc40bd0e0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.437+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3239495806 shutdown_connections 2024-04-27T00:08:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.437+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55d1e01c3030 0x55d1e01c5450 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.437+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1e014c150 0x55d1e0156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.437+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55d1e0146fe0 0x55d1e0155b60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.437+0000 7fdbe2e60740 1 --2- 172.21.15.40:0/3239495806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d1e01465f0 0x55d1e0155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.437+0000 7fdbbea00640 0 client.25946 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:08:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.437+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3239495806 >> 172.21.15.40:0/3239495806 conn(0x55d1e0139450 msgr2=0x55d1e0137dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.438+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3239495806 shutdown_connections 2024-04-27T00:08:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.438+0000 7fdbe2e60740 1 -- 172.21.15.40:0/3239495806 wait complete. 2024-04-27T00:08:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.438+0000 7fdbe2e60740 20 client.25946 trim_cache size 0 max 16384 2024-04-27T00:08:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:08:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:08:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:08:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:08:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.439+0000 7fdbdea00640 1 -- 172.21.15.40:0/2017471157 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdbc00a3cb0 msgr2=0x7fdbc00a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.439+0000 7fdbdea00640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdbc00a3cb0 0x7fdbc00a6170 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x55d1e0147840 tx=0x7fdbd0031af0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.439+0000 7fdbdea00640 1 -- 172.21.15.40:0/2017471157 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdbd8105be0 msgr2=0x7fdbd819b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:08:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.439+0000 7fdbdea00640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdbd8105be0 0x7fdbd819b0c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400e9f0 tx=0x7fdbc400eeb0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.439+0000 7fdbdea00640 1 -- 172.21.15.40:0/2017471157 shutdown_connections 2024-04-27T00:08:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.439+0000 7fdbdea00640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdbc00a3cb0 0x7fdbc00a6170 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.439+0000 7fdbdea00640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd81065a0 0x7fdbd81a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.439+0000 7fdbdea00640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdbd8105be0 0x7fdbd819b0c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.439+0000 7fdbdea00640 1 --2- 172.21.15.40:0/2017471157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd81049f0 0x7fdbd819ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:08:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.439+0000 7fdbdea00640 1 -- 172.21.15.40:0/2017471157 >> 172.21.15.40:0/2017471157 conn(0x7fdbd80fff60 msgr2=0x7fdbd8108ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:08:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.439+0000 7fdbdea00640 1 -- 172.21.15.40:0/2017471157 shutdown_connections 2024-04-27T00:08:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:08:51.439+0000 7fdbdea00640 1 -- 172.21.15.40:0/2017471157 wait complete. 2024-04-27T00:08:51.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:51 smithi040 ceph-mon[28095]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-04-27T00:08:51.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:51 smithi040 ceph-mon[28095]: Unable to set osd_memory_target on smithi103 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-27T00:08:51.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:51 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:08:51.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:51 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:08:51.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:51 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:08:51.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:51 smithi080 ceph-mon[32362]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-04-27T00:08:51.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:51 smithi080 ceph-mon[32362]: Unable to set osd_memory_target on smithi103 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-27T00:08:51.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:51 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:08:51.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:51 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:08:51.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:51 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:08:51.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:51 smithi103 ceph-mon[32952]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-04-27T00:08:51.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:51 smithi103 ceph-mon[32952]: Unable to set osd_memory_target on smithi103 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-27T00:08:51.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:51 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:08:51.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:51 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:08:51.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:51 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:08:52.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:52 smithi080 ceph-mon[32362]: pgmap v627: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:08:52.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:52 smithi103 ceph-mon[32952]: pgmap v627: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:08:52.717 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:52 smithi040 ceph-mon[28095]: pgmap v627: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:08:54.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:54 smithi080 ceph-mon[32362]: pgmap v628: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 14 op/s 2024-04-27T00:08:54.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:54 smithi103 ceph-mon[32952]: pgmap v628: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 14 op/s 2024-04-27T00:08:54.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:54 smithi040 ceph-mon[28095]: pgmap v628: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 14 op/s 2024-04-27T00:08:56.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:56 smithi080 ceph-mon[32362]: pgmap v629: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:08:56.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:56 smithi103 ceph-mon[32952]: pgmap v629: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:08:56.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:56 smithi040 ceph-mon[28095]: pgmap v629: 129 pgs: 129 active+clean; 4.7 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:08:58.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:08:58 smithi080 ceph-mon[32362]: pgmap v630: 129 pgs: 129 active+clean; 797 KiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:08:58.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:08:58 smithi103 ceph-mon[32952]: pgmap v630: 129 pgs: 129 active+clean; 797 KiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:08:58.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:08:58 smithi040 ceph-mon[28095]: pgmap v630: 129 pgs: 129 active+clean; 797 KiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:09:00.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:00 smithi080 ceph-mon[32362]: pgmap v631: 129 pgs: 129 active+clean; 797 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:00.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:00 smithi103 ceph-mon[32952]: pgmap v631: 129 pgs: 129 active+clean; 797 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:00.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:00 smithi040 ceph-mon[28095]: pgmap v631: 129 pgs: 129 active+clean; 797 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:02.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:02 smithi080 ceph-mon[32362]: pgmap v632: 129 pgs: 129 active+clean; 858 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:09:02.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:02 smithi103 ceph-mon[32952]: pgmap v632: 129 pgs: 129 active+clean; 858 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:09:02.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:02 smithi040 ceph-mon[28095]: pgmap v632: 129 pgs: 129 active+clean; 858 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:09:04.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:04 smithi080 ceph-mon[32362]: pgmap v633: 129 pgs: 129 active+clean; 858 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:09:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.588+0000 7f096ca00640 1 -- 172.21.15.40:0/170819928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09681039e0 msgr2=0x7f0968103e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.588+0000 7f096ca00640 1 --2- 172.21.15.40:0/170819928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09681039e0 0x7f0968103e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0950009860 tx=0x7f095002a340 comp rx=0 tx=0).stop 2024-04-27T00:09:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.588+0000 7f096ca00640 1 -- 172.21.15.40:0/170819928 shutdown_connections 2024-04-27T00:09:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.588+0000 7f096ca00640 1 --2- 172.21.15.40:0/170819928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09681043a0 0x7f096810ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.588+0000 7f096ca00640 1 --2- 172.21.15.40:0/170819928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09681039e0 0x7f0968103e60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.588+0000 7f096ca00640 1 --2- 172.21.15.40:0/170819928 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09681027f0 0x7f0968102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.589+0000 7f096ca00640 1 -- 172.21.15.40:0/170819928 >> 172.21.15.40:0/170819928 conn(0x7f09680fdd60 msgr2=0x7f09681001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.589+0000 7f096ca00640 1 -- 172.21.15.40:0/170819928 shutdown_connections 2024-04-27T00:09:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.589+0000 7f096ca00640 1 -- 172.21.15.40:0/170819928 wait complete. 2024-04-27T00:09:04.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.590+0000 7f096ca00640 1 Processor -- start 2024-04-27T00:09:04.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.590+0000 7f096ca00640 1 -- start start 2024-04-27T00:09:04.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.590+0000 7f096ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09681027f0 0x7f096819ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.591 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.590+0000 7f096ca00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09681039e0 0x7f096819b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.590+0000 7f096ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09681043a0 0x7f09681a21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.590+0000 7f096ca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f096810e300 con 0x7f09681027f0 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.590+0000 7f096ca00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f096810e180 con 0x7f09681039e0 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.590+0000 7f096ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f096810e480 con 0x7f09681043a0 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.590+0000 7f0967400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09681027f0 0x7f096819ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.590+0000 7f095fe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09681039e0 0x7f096819b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.590+0000 7f0967400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09681027f0 0x7f096819ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43564/0 (socket says 172.21.15.40:43564) 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.590+0000 7f0967400640 1 -- 172.21.15.40:0/429798333 learned_addr learned my addr 172.21.15.40:0/429798333 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.591+0000 7f0967e00640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09681043a0 0x7f09681a21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.591+0000 7f095fe00640 1 -- 172.21.15.40:0/429798333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09681043a0 msgr2=0x7f09681a21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.591+0000 7f095fe00640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09681043a0 0x7f09681a21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.591+0000 7f095fe00640 1 -- 172.21.15.40:0/429798333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09681027f0 msgr2=0x7f096819ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.591+0000 7f095fe00640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09681027f0 0x7f096819ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.591+0000 7f095fe00640 1 -- 172.21.15.40:0/429798333 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09681a2850 con 0x7f09681039e0 2024-04-27T00:09:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.591+0000 7f0967400640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09681027f0 0x7f096819ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:09:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.591+0000 7f095fe00640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09681039e0 0x7f096819b180 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f095002a860 tx=0x7f0950032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.592+0000 7f0964c00640 1 -- 172.21.15.40:0/429798333 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950040020 con 0x7f09681039e0 2024-04-27T00:09:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.592+0000 7f096ca00640 1 -- 172.21.15.40:0/429798333 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09681a2ae0 con 0x7f09681039e0 2024-04-27T00:09:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.592+0000 7f0964c00640 1 -- 172.21.15.40:0/429798333 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0950030d50 con 0x7f09681039e0 2024-04-27T00:09:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.592+0000 7f0964c00640 1 -- 172.21.15.40:0/429798333 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950037040 con 0x7f09681039e0 2024-04-27T00:09:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.592+0000 7f096ca00640 1 -- 172.21.15.40:0/429798333 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09681a3020 con 0x7f09681039e0 2024-04-27T00:09:04.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.595+0000 7f0970fb5740 1 Processor -- start 2024-04-27T00:09:04.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.595+0000 7f0970fb5740 1 -- start start 2024-04-27T00:09:04.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0970fb5740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2e0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0970fb5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2e52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0970fb5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e5940 0x55cbbf2e7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0970fb5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cbbf244380 con 0x55cbbf2e5940 2024-04-27T00:09:04.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0970fb5740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55cbbf1e0c30 con 0x55cbbf2dfde0 2024-04-27T00:09:04.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0970fb5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cbbf240340 con 0x55cbbf2e07d0 2024-04-27T00:09:04.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0967400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2e52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0967400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2e52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:40384/0 (socket says 172.21.15.40:40384) 2024-04-27T00:09:04.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0967400640 1 -- 172.21.15.40:0/1812635923 learned_addr learned my addr 172.21.15.40:0/1812635923 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:04.596 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0967e00640 1 --2- 172.21.15.40:0/1812635923 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2e0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0964c00640 1 -- 172.21.15.40:0/429798333 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0950043020 con 0x7f09681039e0 2024-04-27T00:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0967400640 1 -- 172.21.15.40:0/1812635923 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 msgr2=0x55cbbf2e0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0967400640 1 --2- 172.21.15.40:0/1812635923 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2e0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0967400640 1 -- 172.21.15.40:0/1812635923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e5940 msgr2=0x55cbbf2e7dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-27T00:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0967400640 1 --2- 172.21.15.40:0/1812635923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e5940 0x55cbbf2e7dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.596+0000 7f0967400640 1 -- 172.21.15.40:0/1812635923 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbbf202940 con 0x55cbbf2e07d0 2024-04-27T00:09:04.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:04 smithi103 ceph-mon[32952]: pgmap v633: 129 pgs: 129 active+clean; 858 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:09:04.601 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.597+0000 7f0967400640 1 --2- 172.21.15.40:0/1812635923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2e52d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f09580115f0 tx=0x7f0958011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d79dab1c82267965 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.597+0000 7f0957e00640 1 -- 172.21.15.40:0/1812635923 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095801e070 con 0x55cbbf2e07d0 2024-04-27T00:09:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.597+0000 7f0957e00640 1 -- 172.21.15.40:0/1812635923 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f095800fbb0 con 0x55cbbf2e07d0 2024-04-27T00:09:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.597+0000 7f0957e00640 1 -- 172.21.15.40:0/1812635923 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0958019b90 con 0x55cbbf2e07d0 2024-04-27T00:09:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0970fb5740 1 -- 172.21.15.40:0/1812635923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 msgr2=0x55cbbf2e52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0970fb5740 1 --2- 172.21.15.40:0/1812635923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2e52d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f09580115f0 tx=0x7f0958011ab0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0970fb5740 1 -- 172.21.15.40:0/1812635923 shutdown_connections 2024-04-27T00:09:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0970fb5740 1 --2- 172.21.15.40:0/1812635923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e5940 0x55cbbf2e7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0970fb5740 1 --2- 172.21.15.40:0/1812635923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2e52d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0970fb5740 1 --2- 172.21.15.40:0/1812635923 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2e0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0970fb5740 1 -- 172.21.15.40:0/1812635923 >> 172.21.15.40:0/1812635923 conn(0x55cbbf2db450 msgr2=0x55cbbf2db850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0970fb5740 1 -- 172.21.15.40:0/1812635923 shutdown_connections 2024-04-27T00:09:04.602 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0970fb5740 1 -- 172.21.15.40:0/1812635923 wait complete. 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0964c00640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f094c0a3b40 0x7f094c0a6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0964c00640 1 -- 172.21.15.40:0/429798333 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f09500f0540 con 0x7f09681039e0 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0967400640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f094c0a3b40 0x7f094c0a6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.598+0000 7f0970fb5740 1 -- start start 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.599+0000 7f0970fb5740 1 Processor -- start 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.599+0000 7f0970fb5740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ea060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.599+0000 7f0970fb5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2ea5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.599+0000 7f0967400640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f094c0a3b40 0x7f094c0a6000 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f09580115c0 tx=0x7f0958009840 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.599+0000 7f0970fb5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e5940 0x55cbbf2f1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.599+0000 7f0970fb5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cbbf2e88f0 con 0x55cbbf2e5940 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.599+0000 7f0970fb5740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55cbbf1cbc60 con 0x55cbbf2dfde0 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.599+0000 7f0970fb5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cbbf2e8a70 con 0x55cbbf2e07d0 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.599+0000 7f0970fb5740 10 client.0 _ll_register_callbacks cb 0x55cbbf1f7e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:09:04.603 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.599+0000 7f0967400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ea060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.600+0000 7f0967e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e5940 0x55cbbf2f1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.600+0000 7f095fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2ea5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.600+0000 7f0967400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ea060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:48634/0 (socket says 172.21.15.40:48634) 2024-04-27T00:09:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.600+0000 7f0967400640 1 -- 172.21.15.40:0/663675307 learned_addr learned my addr 172.21.15.40:0/663675307 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.600+0000 7f095fe00640 1 -- 172.21.15.40:0/663675307 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 msgr2=0x55cbbf2ea060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.600+0000 7f095fe00640 1 --2- 172.21.15.40:0/663675307 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ea060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.600+0000 7f095fe00640 1 -- 172.21.15.40:0/663675307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e5940 msgr2=0x55cbbf2f1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.600+0000 7f095fe00640 1 --2- 172.21.15.40:0/663675307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e5940 0x55cbbf2f1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.600+0000 7f095fe00640 1 -- 172.21.15.40:0/663675307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbbf1da510 con 0x55cbbf2e07d0 2024-04-27T00:09:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.600+0000 7f095fe00640 1 --2- 172.21.15.40:0/663675307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2ea5a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f09500edc60 tx=0x7f09500edfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9195e867cb7f807 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.600+0000 7f0954c00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:09:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.600+0000 7f0954c00640 1 -- 172.21.15.40:0/663675307 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09500d5060 con 0x55cbbf2e07d0 2024-04-27T00:09:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.601+0000 7f0970fb5740 1 -- 172.21.15.40:0/663675307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f095800fbb0 con 0x55cbbf2e07d0 2024-04-27T00:09:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.601+0000 7f0954c00640 1 -- 172.21.15.40:0/663675307 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f09500c4c10 con 0x55cbbf2e07d0 2024-04-27T00:09:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.601+0000 7f0954c00640 1 -- 172.21.15.40:0/663675307 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f09500ba570 con 0x55cbbf2e07d0 2024-04-27T00:09:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.601+0000 7f0970fb5740 10 client.25987 fetch_fsmap learned FSMap version 24 2024-04-27T00:09:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.601+0000 7f0970fb5740 1 -- 172.21.15.40:0/663675307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cbbef56210 con 0x55cbbf2e07d0 2024-04-27T00:09:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.601+0000 7f0954c00640 1 -- 172.21.15.40:0/663675307 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f095800fbb0 con 0x55cbbf2e07d0 2024-04-27T00:09:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.601+0000 7f0954c00640 1 -- 172.21.15.40:0/663675307 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f09500db170 con 0x55cbbf2e07d0 2024-04-27T00:09:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.602+0000 7f0970fb5740 10 client.25987 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:09:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.602+0000 7f0970fb5740 10 client.25987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:09:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.602+0000 7f0970fb5740 20 client.25987 populate_metadata read hostname 'smithi040' 2024-04-27T00:09:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.602+0000 7f0970fb5740 1 --2- 172.21.15.40:0/663675307 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55cbbf2f5b40 0x55cbbf2f7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.602+0000 7f0970fb5740 4 client.25987 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:09:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.602+0000 7f0970fb5740 1 -- 172.21.15.40:0/663675307 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbbf251f90 con 0x55cbbf2f5b40 2024-04-27T00:09:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.602+0000 7f0967400640 1 --2- 172.21.15.40:0/663675307 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55cbbf2f5b40 0x55cbbf2f7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.603+0000 7f0967400640 1 --2- 172.21.15.40:0/663675307 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55cbbf2f5b40 0x55cbbf2f7f50 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e97dd6b6e366162 server_cookie=39c9f0d98a9fa9f2 in_seq=0 out_seq=0 2024-04-27T00:09:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.603+0000 7f0954c00640 10 client.25987 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0954c00640 1 -- 172.21.15.40:0/663675307 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cbbf251f90 con 0x55cbbf2f5b40 2024-04-27T00:09:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0954c00640 10 client.25987 handle_command_reply: tid=0 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0970fb5740 1 client.25987 shutdown 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0970fb5740 1 -- 172.21.15.40:0/663675307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 msgr2=0x55cbbf2ea5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0970fb5740 1 --2- 172.21.15.40:0/663675307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2ea5a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f09500edc60 tx=0x7f09500edfd0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0970fb5740 1 -- 172.21.15.40:0/663675307 shutdown_connections 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0970fb5740 1 --2- 172.21.15.40:0/663675307 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55cbbf2f5b40 0x55cbbf2f7f50 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0970fb5740 1 --2- 172.21.15.40:0/663675307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e5940 0x55cbbf2f1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0970fb5740 1 --2- 172.21.15.40:0/663675307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2ea5a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0970fb5740 1 --2- 172.21.15.40:0/663675307 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ea060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0970fb5740 1 -- 172.21.15.40:0/663675307 >> 172.21.15.40:0/663675307 conn(0x55cbbf2db450 msgr2=0x55cbbf2da650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0954c00640 0 client.25987 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0970fb5740 1 -- 172.21.15.40:0/663675307 shutdown_connections 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0970fb5740 1 -- 172.21.15.40:0/663675307 wait complete. 2024-04-27T00:09:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.604+0000 7f0970fb5740 20 client.25987 trim_cache size 0 max 16384 2024-04-27T00:09:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f0970fb5740 1 Processor -- start 2024-04-27T00:09:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f0970fb5740 1 -- start start 2024-04-27T00:09:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f0970fb5740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ea060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f0970fb5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2f1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f0970fb5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e5940 0x55cbbf2ea5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f0970fb5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f09681afcf0 con 0x55cbbf2e07d0 2024-04-27T00:09:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f0970fb5740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f09681ad5c0 con 0x55cbbf2dfde0 2024-04-27T00:09:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f0970fb5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f09681ab410 con 0x55cbbf2e5940 2024-04-27T00:09:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f095fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f0967e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e5940 0x55cbbf2ea5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f095fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2f1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43590/0 (socket says 172.21.15.40:43590) 2024-04-27T00:09:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f095fe00640 1 -- 172.21.15.40:0/3007756966 learned_addr learned my addr 172.21.15.40:0/3007756966 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f0967e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e5940 0x55cbbf2ea5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:40404/0 (socket says 172.21.15.40:40404) 2024-04-27T00:09:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.621+0000 7f0967400640 1 --2- 172.21.15.40:0/3007756966 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ea060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.622+0000 7f0967e00640 1 -- 172.21.15.40:0/3007756966 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 msgr2=0x55cbbf2ea060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.622+0000 7f0967e00640 1 --2- 172.21.15.40:0/3007756966 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ea060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.622+0000 7f0967e00640 1 -- 172.21.15.40:0/3007756966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e07d0 msgr2=0x55cbbf2f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.622+0000 7f0967e00640 1 --2- 172.21.15.40:0/3007756966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2f1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.622+0000 7f0967e00640 1 -- 172.21.15.40:0/3007756966 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbbf022620 con 0x55cbbf2e5940 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.622+0000 7f0967e00640 1 --2- 172.21.15.40:0/3007756966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e5940 0x55cbbf2ea5a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55cbbf2e1030 tx=0x7f0960009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72d59031fa87027d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.622+0000 7f0957400640 1 -- 172.21.15.40:0/3007756966 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0960022070 con 0x55cbbf2e5940 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.622+0000 7f0957400640 1 -- 172.21.15.40:0/3007756966 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0960003e20 con 0x55cbbf2e5940 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.622+0000 7f0957400640 1 -- 172.21.15.40:0/3007756966 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0960004050 con 0x55cbbf2e5940 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.623+0000 7f0970fb5740 1 -- 172.21.15.40:0/3007756966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e5940 msgr2=0x55cbbf2ea5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.623+0000 7f0970fb5740 1 --2- 172.21.15.40:0/3007756966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e5940 0x55cbbf2ea5a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55cbbf2e1030 tx=0x7f0960009500 comp rx=0 tx=0).stop 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.623+0000 7f0970fb5740 1 -- 172.21.15.40:0/3007756966 shutdown_connections 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.623+0000 7f0970fb5740 1 --2- 172.21.15.40:0/3007756966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e5940 0x55cbbf2ea5a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.623+0000 7f0970fb5740 1 --2- 172.21.15.40:0/3007756966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.623+0000 7f0970fb5740 1 --2- 172.21.15.40:0/3007756966 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ea060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.623+0000 7f0970fb5740 1 -- 172.21.15.40:0/3007756966 >> 172.21.15.40:0/3007756966 conn(0x55cbbf2db450 msgr2=0x55cbbf2e9350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.623+0000 7f0970fb5740 1 -- 172.21.15.40:0/3007756966 shutdown_connections 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.623+0000 7f0970fb5740 1 -- 172.21.15.40:0/3007756966 wait complete. 2024-04-27T00:09:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.624+0000 7f0970fb5740 1 -- start start 2024-04-27T00:09:04.625 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.624+0000 7f0970fb5740 1 Processor -- start 2024-04-27T00:09:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.624+0000 7f0970fb5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ee6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.624+0000 7f0970fb5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2eec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.624+0000 7f0970fb5740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2e5940 0x55cbbf2efc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.624+0000 7f0970fb5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f09681aba80 con 0x55cbbf2dfde0 2024-04-27T00:09:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.624+0000 7f0970fb5740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f09681acef0 con 0x55cbbf2e5940 2024-04-27T00:09:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.624+0000 7f0970fb5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f09681ac150 con 0x55cbbf2e07d0 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.624+0000 7f0970fb5740 10 client.0 _ll_register_callbacks cb 0x7f09500057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f0967400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ee6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f0967e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2e5940 0x55cbbf2efc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f0967400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ee6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43602/0 (socket says 172.21.15.40:43602) 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f0967400640 1 -- 172.21.15.40:0/2387936610 learned_addr learned my addr 172.21.15.40:0/2387936610 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f095fe00640 1 --2- 172.21.15.40:0/2387936610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f0967400640 1 -- 172.21.15.40:0/2387936610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 msgr2=0x55cbbf2eec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f0967400640 1 --2- 172.21.15.40:0/2387936610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2eec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f0967400640 1 -- 172.21.15.40:0/2387936610 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2e5940 msgr2=0x55cbbf2efc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f0967400640 1 --2- 172.21.15.40:0/2387936610 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2e5940 0x55cbbf2efc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f0967400640 1 -- 172.21.15.40:0/2387936610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbbf1fecb0 con 0x55cbbf2dfde0 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f0967400640 1 --2- 172.21.15.40:0/2387936610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ee6d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f09580115f0 tx=0x7f0958009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea046c6be8d6020 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f0954c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:09:04.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.625+0000 7f0954c00640 1 -- 172.21.15.40:0/2387936610 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095801e040 con 0x55cbbf2dfde0 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0954c00640 1 -- 172.21.15.40:0/2387936610 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f0958019630 con 0x55cbbf2dfde0 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0970fb5740 1 -- 172.21.15.40:0/2387936610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0960003e20 con 0x55cbbf2dfde0 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0954c00640 1 -- 172.21.15.40:0/2387936610 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f095803f050 con 0x55cbbf2dfde0 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0970fb5740 10 client.15849 fetch_fsmap learned FSMap version 24 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0970fb5740 1 -- 172.21.15.40:0/2387936610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cbbf2f13c0 con 0x55cbbf2dfde0 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0954c00640 1 -- 172.21.15.40:0/2387936610 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0960003e20 con 0x55cbbf2dfde0 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0954c00640 1 -- 172.21.15.40:0/2387936610 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f09580051b0 con 0x55cbbf2dfde0 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0970fb5740 10 client.15849 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0970fb5740 10 client.15849 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0970fb5740 20 client.15849 populate_metadata read hostname 'smithi040' 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0970fb5740 1 --2- 172.21.15.40:0/2387936610 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55cbbf35c270 0x55cbbf35e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0970fb5740 4 client.15849 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.626+0000 7f0970fb5740 1 -- 172.21.15.40:0/2387936610 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cbbf24be00 con 0x55cbbf35c270 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.627+0000 7f095fe00640 1 --2- 172.21.15.40:0/2387936610 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55cbbf35c270 0x55cbbf35e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.627+0000 7f095fe00640 1 --2- 172.21.15.40:0/2387936610 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55cbbf35c270 0x55cbbf35e660 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a19c04c18d9cc93 server_cookie=f1dbd892e9639db7 in_seq=0 out_seq=0 2024-04-27T00:09:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.627+0000 7f0954c00640 10 client.15849 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:04.642 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0954c00640 1 -- 172.21.15.40:0/2387936610 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cbbf24be00 con 0x55cbbf35c270 2024-04-27T00:09:04.642 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0954c00640 10 client.15849 handle_command_reply: tid=0 2024-04-27T00:09:04.642 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0970fb5740 1 client.15849 shutdown 2024-04-27T00:09:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0970fb5740 1 -- 172.21.15.40:0/2387936610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2dfde0 msgr2=0x55cbbf2ee6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0970fb5740 1 --2- 172.21.15.40:0/2387936610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ee6d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f09580115f0 tx=0x7f0958009260 comp rx=0 tx=0).stop 2024-04-27T00:09:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0970fb5740 1 -- 172.21.15.40:0/2387936610 shutdown_connections 2024-04-27T00:09:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0970fb5740 1 --2- 172.21.15.40:0/2387936610 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55cbbf35c270 0x55cbbf35e660 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0970fb5740 1 --2- 172.21.15.40:0/2387936610 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55cbbf2e5940 0x55cbbf2efc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0970fb5740 1 --2- 172.21.15.40:0/2387936610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbbf2e07d0 0x55cbbf2eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0970fb5740 1 --2- 172.21.15.40:0/2387936610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbbf2dfde0 0x55cbbf2ee6d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0954c00640 0 client.15849 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:04.643 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0970fb5740 1 -- 172.21.15.40:0/2387936610 >> 172.21.15.40:0/2387936610 conn(0x55cbbf2db450 msgr2=0x55cbbf2e19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0970fb5740 1 -- 172.21.15.40:0/2387936610 shutdown_connections 2024-04-27T00:09:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0970fb5740 1 -- 172.21.15.40:0/2387936610 wait complete. 2024-04-27T00:09:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.642+0000 7f0970fb5740 20 client.15849 trim_cache size 0 max 16384 2024-04-27T00:09:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:09:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:09:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:09:04.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:09:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.643+0000 7f096ca00640 1 -- 172.21.15.40:0/429798333 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f094c0a3b40 msgr2=0x7f094c0a6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.643+0000 7f096ca00640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f094c0a3b40 0x7f094c0a6000 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f09580115c0 tx=0x7f0958009840 comp rx=0 tx=0).stop 2024-04-27T00:09:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.643+0000 7f096ca00640 1 -- 172.21.15.40:0/429798333 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09681039e0 msgr2=0x7f096819b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.643+0000 7f096ca00640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09681039e0 0x7f096819b180 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f095002a860 tx=0x7f0950032d50 comp rx=0 tx=0).stop 2024-04-27T00:09:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.643+0000 7f096ca00640 1 -- 172.21.15.40:0/429798333 shutdown_connections 2024-04-27T00:09:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.643+0000 7f096ca00640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f094c0a3b40 0x7f094c0a6000 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.643+0000 7f096ca00640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09681043a0 0x7f09681a21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.643+0000 7f096ca00640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f09681039e0 0x7f096819b180 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.643+0000 7f096ca00640 1 --2- 172.21.15.40:0/429798333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09681027f0 0x7f096819ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.643+0000 7f096ca00640 1 -- 172.21.15.40:0/429798333 >> 172.21.15.40:0/429798333 conn(0x7f09680fdd60 msgr2=0x7f0968074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.643+0000 7f096ca00640 1 -- 172.21.15.40:0/429798333 shutdown_connections 2024-04-27T00:09:04.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:04.643+0000 7f096ca00640 1 -- 172.21.15.40:0/429798333 wait complete. 2024-04-27T00:09:04.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:04 smithi040 ceph-mon[28095]: pgmap v633: 129 pgs: 129 active+clean; 858 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:09:06.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:06 smithi080 ceph-mon[32362]: pgmap v634: 129 pgs: 129 active+clean; 858 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:06.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:06 smithi103 ceph-mon[32952]: pgmap v634: 129 pgs: 129 active+clean; 858 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:06.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:06 smithi040 ceph-mon[28095]: pgmap v634: 129 pgs: 129 active+clean; 858 KiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:08.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:08 smithi080 ceph-mon[32362]: pgmap v635: 129 pgs: 129 active+clean; 930 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:09:08.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:08 smithi103 ceph-mon[32952]: pgmap v635: 129 pgs: 129 active+clean; 930 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:09:08.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:08 smithi040 ceph-mon[28095]: pgmap v635: 129 pgs: 129 active+clean; 930 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:09:10.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:10 smithi080 ceph-mon[32362]: pgmap v636: 129 pgs: 129 active+clean; 930 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:10.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:10 smithi103 ceph-mon[32952]: pgmap v636: 129 pgs: 129 active+clean; 930 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:10.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:10 smithi040 ceph-mon[28095]: pgmap v636: 129 pgs: 129 active+clean; 930 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:12.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:12 smithi080 ceph-mon[32362]: pgmap v637: 129 pgs: 129 active+clean; 991 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:09:12.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:12 smithi103 ceph-mon[32952]: pgmap v637: 129 pgs: 129 active+clean; 991 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:09:12.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:12 smithi040 ceph-mon[28095]: pgmap v637: 129 pgs: 129 active+clean; 991 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:09:14.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:14 smithi080 ceph-mon[32362]: pgmap v638: 129 pgs: 129 active+clean; 991 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:09:14.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:14 smithi103 ceph-mon[32952]: pgmap v638: 129 pgs: 129 active+clean; 991 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:09:14.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:14 smithi040 ceph-mon[28095]: pgmap v638: 129 pgs: 129 active+clean; 991 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:09:16.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:16 smithi080 ceph-mon[32362]: pgmap v639: 129 pgs: 129 active+clean; 991 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:16.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:16 smithi103 ceph-mon[32952]: pgmap v639: 129 pgs: 129 active+clean; 991 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:16.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:16 smithi040 ceph-mon[28095]: pgmap v639: 129 pgs: 129 active+clean; 991 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.789+0000 7f2adbc00640 1 -- 172.21.15.40:0/217010083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad410a1b0 msgr2=0x7f2ad410a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.789+0000 7f2adbc00640 1 --2- 172.21.15.40:0/217010083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad410a1b0 0x7f2ad410a630 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009900 tx=0x7f2ac002a400 comp rx=0 tx=0).stop 2024-04-27T00:09:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.790+0000 7f2adbc00640 1 -- 172.21.15.40:0/217010083 shutdown_connections 2024-04-27T00:09:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.790+0000 7f2adbc00640 1 --2- 172.21.15.40:0/217010083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ad410ab70 0x7f2ad4111400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.790+0000 7f2adbc00640 1 --2- 172.21.15.40:0/217010083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad410a1b0 0x7f2ad410a630 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.790+0000 7f2adbc00640 1 --2- 172.21.15.40:0/217010083 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2ad4108fc0 0x7f2ad41093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.790+0000 7f2adbc00640 1 -- 172.21.15.40:0/217010083 >> 172.21.15.40:0/217010083 conn(0x7f2ad4077f80 msgr2=0x7f2ad407a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.790+0000 7f2adbc00640 1 -- 172.21.15.40:0/217010083 shutdown_connections 2024-04-27T00:09:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.790+0000 7f2adbc00640 1 -- 172.21.15.40:0/217010083 wait complete. 2024-04-27T00:09:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.791+0000 7f2adbc00640 1 Processor -- start 2024-04-27T00:09:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.791+0000 7f2adbc00640 1 -- start start 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.791+0000 7f2adbc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4108fc0 0x7f2ad419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.791+0000 7f2adbc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ad410a1b0 0x7f2ad419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.791+0000 7f2adbc00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2ad410ab70 0x7f2ad41a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.791+0000 7f2adbc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad4114b50 con 0x7f2ad410a1b0 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.791+0000 7f2adbc00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41149d0 con 0x7f2ad410ab70 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.791+0000 7f2adbc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad4114cd0 con 0x7f2ad4108fc0 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.791+0000 7f2ad3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ad410a1b0 0x7f2ad419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.792+0000 7f2ad9c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2ad410ab70 0x7f2ad41a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.792+0000 7f2ad3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ad410a1b0 0x7f2ad419f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59360/0 (socket says 172.21.15.40:59360) 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.792+0000 7f2ad3e00640 1 -- 172.21.15.40:0/3837856262 learned_addr learned my addr 172.21.15.40:0/3837856262 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.792+0000 7f2ad9200640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4108fc0 0x7f2ad419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.792+0000 7f2ad9c00640 1 -- 172.21.15.40:0/3837856262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4108fc0 msgr2=0x7f2ad419f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.792+0000 7f2ad9c00640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4108fc0 0x7f2ad419f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.792+0000 7f2ad9c00640 1 -- 172.21.15.40:0/3837856262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ad410a1b0 msgr2=0x7f2ad419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.792+0000 7f2ad9c00640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ad410a1b0 0x7f2ad419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.792+0000 7f2ad9c00640 1 -- 172.21.15.40:0/3837856262 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad41a6d70 con 0x7f2ad410ab70 2024-04-27T00:09:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.792+0000 7f2ad3e00640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ad410a1b0 0x7f2ad419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:09:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.792+0000 7f2ad9c00640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2ad410ab70 0x7f2ad41a6700 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00da10 tx=0x7f2acc00ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.793+0000 7f2ad1600640 1 -- 172.21.15.40:0/3837856262 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc00bd70 con 0x7f2ad410ab70 2024-04-27T00:09:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.793+0000 7f2ad1600640 1 -- 172.21.15.40:0/3837856262 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2acc003e20 con 0x7f2ad410ab70 2024-04-27T00:09:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.793+0000 7f2ad1600640 1 -- 172.21.15.40:0/3837856262 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0106c0 con 0x7f2ad410ab70 2024-04-27T00:09:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.793+0000 7f2adbc00640 1 -- 172.21.15.40:0/3837856262 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad41a7060 con 0x7f2ad410ab70 2024-04-27T00:09:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.793+0000 7f2adbc00640 1 -- 172.21.15.40:0/3837856262 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad41a7560 con 0x7f2ad410ab70 2024-04-27T00:09:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.796+0000 7f2adea76740 1 Processor -- start 2024-04-27T00:09:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.796+0000 7f2ad1600640 1 -- 172.21.15.40:0/3837856262 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2acc004270 con 0x7f2ad410ab70 2024-04-27T00:09:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.797+0000 7f2adea76740 1 -- start start 2024-04-27T00:09:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.797+0000 7f2adea76740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e3900bd0 0x5599e3900ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.797+0000 7f2adea76740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e39015c0 0x5599e39060c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.797+0000 7f2adea76740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e3908bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.797+0000 7f2adea76740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5599e3865380 con 0x5599e39015c0 2024-04-27T00:09:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.797+0000 7f2adea76740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5599e3801c30 con 0x5599e3900bd0 2024-04-27T00:09:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.797+0000 7f2adea76740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5599e3861340 con 0x5599e3906730 2024-04-27T00:09:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.797+0000 7f2ad9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e39015c0 0x5599e39060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.797+0000 7f2ad3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e3900bd0 0x5599e3900ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.797+0000 7f2ad9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e39015c0 0x5599e39060c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59364/0 (socket says 172.21.15.40:59364) 2024-04-27T00:09:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.797+0000 7f2ad9200640 1 -- 172.21.15.40:0/1776729416 learned_addr learned my addr 172.21.15.40:0/1776729416 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.797+0000 7f2ad3e00640 1 --2- 172.21.15.40:0/1776729416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e3908bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.798+0000 7f2ad9200640 1 -- 172.21.15.40:0/1776729416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 msgr2=0x5599e3908bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.798+0000 7f2ad9200640 1 --2- 172.21.15.40:0/1776729416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e3908bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.798+0000 7f2ad9200640 1 -- 172.21.15.40:0/1776729416 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e3900bd0 msgr2=0x5599e3900ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.798+0000 7f2ad9200640 1 --2- 172.21.15.40:0/1776729416 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e3900bd0 0x5599e3900ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.798+0000 7f2ad9200640 1 -- 172.21.15.40:0/1776729416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599e3823940 con 0x5599e39015c0 2024-04-27T00:09:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.798+0000 7f2ad3e00640 1 --2- 172.21.15.40:0/1776729416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e3908bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:09:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.798+0000 7f2ad1600640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2abc0a3d70 0x7f2abc0a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.798+0000 7f2ad1600640 1 -- 172.21.15.40:0/3837856262 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f2acc0d0cd0 con 0x7f2ad410ab70 2024-04-27T00:09:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.798+0000 7f2ad3e00640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2abc0a3d70 0x7f2abc0a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.798+0000 7f2ad9200640 1 --2- 172.21.15.40:0/1776729416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e39015c0 0x5599e39060c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800bcb0 tx=0x7f2ac80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b07a92c1bf17dd server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.799+0000 7f2ac6000640 1 -- 172.21.15.40:0/1776729416 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8004020 con 0x5599e39015c0 2024-04-27T00:09:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.799+0000 7f2ac6000640 1 -- 172.21.15.40:0/1776729416 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ac80054d0 con 0x5599e39015c0 2024-04-27T00:09:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.799+0000 7f2ac6000640 1 -- 172.21.15.40:0/1776729416 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8004c80 con 0x5599e39015c0 2024-04-27T00:09:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.799+0000 7f2adea76740 1 -- 172.21.15.40:0/1776729416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e39015c0 msgr2=0x5599e39060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.799+0000 7f2ad3e00640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2abc0a3d70 0x7f2abc0a6230 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2ad41a0660 tx=0x7f2ac0005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.799+0000 7f2adea76740 1 --2- 172.21.15.40:0/1776729416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e39015c0 0x5599e39060c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800bcb0 tx=0x7f2ac80107a0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.800+0000 7f2adea76740 1 -- 172.21.15.40:0/1776729416 shutdown_connections 2024-04-27T00:09:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.800+0000 7f2adea76740 1 --2- 172.21.15.40:0/1776729416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e3908bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.800+0000 7f2adea76740 1 --2- 172.21.15.40:0/1776729416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e39015c0 0x5599e39060c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.800+0000 7f2adea76740 1 --2- 172.21.15.40:0/1776729416 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e3900bd0 0x5599e3900ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.800+0000 7f2adea76740 1 -- 172.21.15.40:0/1776729416 >> 172.21.15.40:0/1776729416 conn(0x5599e38fc240 msgr2=0x5599e38fc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.800+0000 7f2adea76740 1 -- 172.21.15.40:0/1776729416 shutdown_connections 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.800+0000 7f2adea76740 1 -- 172.21.15.40:0/1776729416 wait complete. 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.800+0000 7f2adea76740 1 -- start start 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.800+0000 7f2adea76740 1 Processor -- start 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.801+0000 7f2adea76740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e390ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.801+0000 7f2adea76740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e39015c0 0x5599e390b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.801+0000 7f2adea76740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e39122f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.801+0000 7f2adea76740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5599e39096e0 con 0x5599e3900bd0 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.801+0000 7f2adea76740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5599e37ecc60 con 0x5599e39015c0 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.801+0000 7f2adea76740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5599e3909860 con 0x5599e3906730 2024-04-27T00:09:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.801+0000 7f2adea76740 10 client.0 _ll_register_callbacks cb 0x5599e3818e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.801+0000 7f2ad9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e390ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.801+0000 7f2ad9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e390ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59372/0 (socket says 172.21.15.40:59372) 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.801+0000 7f2ad9200640 1 -- 172.21.15.40:0/588453482 learned_addr learned my addr 172.21.15.40:0/588453482 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.801+0000 7f2ad3e00640 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e39015c0 0x5599e390b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.801+0000 7f2ad9c00640 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e39122f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.802+0000 7f2ad9200640 1 -- 172.21.15.40:0/588453482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 msgr2=0x5599e39122f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.802+0000 7f2ad9200640 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e39122f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.802+0000 7f2ad9200640 1 -- 172.21.15.40:0/588453482 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e39015c0 msgr2=0x5599e390b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.802+0000 7f2ad9200640 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e39015c0 0x5599e390b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.802+0000 7f2ad9200640 1 -- 172.21.15.40:0/588453482 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599e37fb510 con 0x5599e3900bd0 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.802+0000 7f2ad9c00640 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e39122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.802+0000 7f2ad9200640 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e390ade0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2ac801a000 tx=0x7f2ac8004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81f8670faec0d6e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.802+0000 7f2abaa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.802+0000 7f2abaa00640 1 -- 172.21.15.40:0/588453482 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac802b060 con 0x5599e3900bd0 2024-04-27T00:09:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.802+0000 7f2abaa00640 1 -- 172.21.15.40:0/588453482 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f2ac8004bd0 con 0x5599e3900bd0 2024-04-27T00:09:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.802+0000 7f2adea76740 1 -- 172.21.15.40:0/588453482 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ac80054d0 con 0x5599e3900bd0 2024-04-27T00:09:17.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.803+0000 7f2abaa00640 1 -- 172.21.15.40:0/588453482 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2ac8032050 con 0x5599e3900bd0 2024-04-27T00:09:17.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.803+0000 7f2adea76740 10 client.15861 fetch_fsmap learned FSMap version 24 2024-04-27T00:09:17.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.803+0000 7f2adea76740 1 -- 172.21.15.40:0/588453482 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5599e3577210 con 0x5599e3900bd0 2024-04-27T00:09:17.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.803+0000 7f2abaa00640 1 -- 172.21.15.40:0/588453482 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ac80054d0 con 0x5599e3900bd0 2024-04-27T00:09:17.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.803+0000 7f2abaa00640 1 -- 172.21.15.40:0/588453482 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2ac8011690 con 0x5599e3900bd0 2024-04-27T00:09:17.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.803+0000 7f2adea76740 10 client.15861 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:09:17.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.803+0000 7f2adea76740 10 client.15861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:09:17.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.803+0000 7f2adea76740 20 client.15861 populate_metadata read hostname 'smithi040' 2024-04-27T00:09:17.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.803+0000 7f2adea76740 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5599e3916560 0x5599e3918950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.808 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.803+0000 7f2adea76740 4 client.15861 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:09:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.803+0000 7f2adea76740 1 -- 172.21.15.40:0/588453482 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599e3872f90 con 0x5599e3916560 2024-04-27T00:09:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.804+0000 7f2ad3e00640 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5599e3916560 0x5599e3918950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.804+0000 7f2ad3e00640 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5599e3916560 0x5599e3918950 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ed3d1372d4bfcf2 server_cookie=b1843ce47b654ad4 in_seq=0 out_seq=0 2024-04-27T00:09:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.804+0000 7f2abaa00640 10 client.15861 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.805+0000 7f2abaa00640 1 -- 172.21.15.40:0/588453482 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5599e3872f90 con 0x5599e3916560 2024-04-27T00:09:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.805+0000 7f2abaa00640 10 client.15861 handle_command_reply: tid=0 2024-04-27T00:09:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.805+0000 7f2adea76740 1 client.15861 shutdown 2024-04-27T00:09:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.806+0000 7f2adea76740 1 -- 172.21.15.40:0/588453482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 msgr2=0x5599e390ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.806+0000 7f2adea76740 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e390ade0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2ac801a000 tx=0x7f2ac8004710 comp rx=0 tx=0).stop 2024-04-27T00:09:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.806+0000 7f2adea76740 1 -- 172.21.15.40:0/588453482 shutdown_connections 2024-04-27T00:09:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.806+0000 7f2adea76740 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5599e3916560 0x5599e3918950 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.809 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.806+0000 7f2adea76740 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e39122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.806+0000 7f2adea76740 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e39015c0 0x5599e390b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.806+0000 7f2adea76740 1 --2- 172.21.15.40:0/588453482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e390ade0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.806+0000 7f2abaa00640 0 client.15861 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:17.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.806+0000 7f2adea76740 1 -- 172.21.15.40:0/588453482 >> 172.21.15.40:0/588453482 conn(0x5599e38fc240 msgr2=0x5599e38fb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:17.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.806+0000 7f2adea76740 1 -- 172.21.15.40:0/588453482 shutdown_connections 2024-04-27T00:09:17.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.806+0000 7f2adea76740 1 -- 172.21.15.40:0/588453482 wait complete. 2024-04-27T00:09:17.810 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.806+0000 7f2adea76740 20 client.15861 trim_cache size 0 max 16384 2024-04-27T00:09:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.822+0000 7f2adea76740 1 Processor -- start 2024-04-27T00:09:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.822+0000 7f2adea76740 1 -- start start 2024-04-27T00:09:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.823+0000 7f2adea76740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e38ff960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.823+0000 7f2adea76740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e39015c0 0x5599e39122b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.823+0000 7f2adea76740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e3906730 0x5599e390b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.823+0000 7f2adea76740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41b43d0 con 0x5599e3900bd0 2024-04-27T00:09:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.823+0000 7f2adea76740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41b1ca0 con 0x5599e3906730 2024-04-27T00:09:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.823+0000 7f2adea76740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41afaf0 con 0x5599e39015c0 2024-04-27T00:09:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.823+0000 7f2ad9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e38ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.823+0000 7f2ad9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e38ff960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59384/0 (socket says 172.21.15.40:59384) 2024-04-27T00:09:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.823+0000 7f2ad9200640 1 -- 172.21.15.40:0/2282670511 learned_addr learned my addr 172.21.15.40:0/2282670511 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.823+0000 7f2ad9c00640 1 --2- 172.21.15.40:0/2282670511 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e3906730 0x5599e390b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.823+0000 7f2ad3e00640 1 --2- 172.21.15.40:0/2282670511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e39015c0 0x5599e39122b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.827 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.824+0000 7f2ad9c00640 1 -- 172.21.15.40:0/2282670511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e39015c0 msgr2=0x5599e39122b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:17.827 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.824+0000 7f2ad9c00640 1 --2- 172.21.15.40:0/2282670511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e39015c0 0x5599e39122b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.824+0000 7f2ad9c00640 1 -- 172.21.15.40:0/2282670511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 msgr2=0x5599e38ff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.824+0000 7f2ad9c00640 1 --2- 172.21.15.40:0/2282670511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e38ff960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.824+0000 7f2ad9c00640 1 -- 172.21.15.40:0/2282670511 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599e38f7940 con 0x5599e3906730 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.824+0000 7f2ad9200640 1 --2- 172.21.15.40:0/2282670511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e38ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.824+0000 7f2ad9c00640 1 --2- 172.21.15.40:0/2282670511 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e3906730 0x5599e390b0c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0ce280 tx=0x7f2acc0bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74fa4c2ce556cc3e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.824+0000 7f2ac5600640 1 -- 172.21.15.40:0/2282670511 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc09ca50 con 0x5599e3906730 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.824+0000 7f2ac5600640 1 -- 172.21.15.40:0/2282670511 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2acc0a0e00 con 0x5599e3906730 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.824+0000 7f2ac5600640 1 -- 172.21.15.40:0/2282670511 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0990c0 con 0x5599e3906730 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.825+0000 7f2adea76740 1 -- 172.21.15.40:0/2282670511 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e3906730 msgr2=0x5599e390b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.825+0000 7f2adea76740 1 --2- 172.21.15.40:0/2282670511 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e3906730 0x5599e390b0c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0ce280 tx=0x7f2acc0bd040 comp rx=0 tx=0).stop 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.825+0000 7f2adea76740 1 -- 172.21.15.40:0/2282670511 shutdown_connections 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.825+0000 7f2adea76740 1 --2- 172.21.15.40:0/2282670511 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e3906730 0x5599e390b0c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.825+0000 7f2adea76740 1 --2- 172.21.15.40:0/2282670511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e39015c0 0x5599e39122b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.825+0000 7f2adea76740 1 --2- 172.21.15.40:0/2282670511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e38ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.825+0000 7f2adea76740 1 -- 172.21.15.40:0/2282670511 >> 172.21.15.40:0/2282670511 conn(0x5599e38fc240 msgr2=0x5599e38fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.825+0000 7f2adea76740 1 -- 172.21.15.40:0/2282670511 shutdown_connections 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.825+0000 7f2adea76740 1 -- 172.21.15.40:0/2282670511 wait complete. 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.825+0000 7f2adea76740 1 -- start start 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.825+0000 7f2adea76740 1 Processor -- start 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2adea76740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e390f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2adea76740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e39015c0 0x5599e390f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2adea76740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e397aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2adea76740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41b0160 con 0x5599e3900bd0 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2adea76740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41b15d0 con 0x5599e39015c0 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2adea76740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41b0830 con 0x5599e3906730 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2adea76740 10 client.0 _ll_register_callbacks cb 0x7f2ac0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2ad9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e390f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.829 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2ad9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e390f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59398/0 (socket says 172.21.15.40:59398) 2024-04-27T00:09:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2ad9200640 1 -- 172.21.15.40:0/1374430099 learned_addr learned my addr 172.21.15.40:0/1374430099 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2ad9c00640 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e397aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2ad3e00640 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e39015c0 0x5599e390f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2ad9200640 1 -- 172.21.15.40:0/1374430099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 msgr2=0x5599e397aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2ad9200640 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e397aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2ad9200640 1 -- 172.21.15.40:0/1374430099 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e39015c0 msgr2=0x5599e390f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2ad9200640 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e39015c0 0x5599e390f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.826+0000 7f2ad9200640 1 -- 172.21.15.40:0/1374430099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599e3643620 con 0x5599e3900bd0 2024-04-27T00:09:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.827+0000 7f2ad9200640 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e390f230 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2ac801b4d0 tx=0x7f2ac8010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e33ef270e91369 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.827+0000 7f2abaa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:09:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.827+0000 7f2abaa00640 1 -- 172.21.15.40:0/1374430099 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac802b070 con 0x5599e3900bd0 2024-04-27T00:09:17.830 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.827+0000 7f2abaa00640 1 -- 172.21.15.40:0/1374430099 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f2ac8011440 con 0x5599e3900bd0 2024-04-27T00:09:17.831 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.827+0000 7f2adea76740 1 -- 172.21.15.40:0/1374430099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2acc0a0e00 con 0x5599e3900bd0 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.827+0000 7f2abaa00640 1 -- 172.21.15.40:0/1374430099 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2ac8011640 con 0x5599e3900bd0 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.827+0000 7f2adea76740 10 client.15867 fetch_fsmap learned FSMap version 24 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.827+0000 7f2adea76740 1 -- 172.21.15.40:0/1374430099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5599e381fcb0 con 0x5599e3900bd0 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.828+0000 7f2abaa00640 1 -- 172.21.15.40:0/1374430099 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2acc0a0e00 con 0x5599e3900bd0 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.828+0000 7f2abaa00640 1 -- 172.21.15.40:0/1374430099 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2ac801b680 con 0x5599e3900bd0 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.828+0000 7f2adea76740 10 client.15867 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.828+0000 7f2adea76740 10 client.15867 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.828+0000 7f2adea76740 20 client.15867 populate_metadata read hostname 'smithi040' 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.828+0000 7f2adea76740 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5599e397c8f0 0x5599e397ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.828+0000 7f2adea76740 4 client.15867 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.828+0000 7f2adea76740 1 -- 172.21.15.40:0/1374430099 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5599e386ce00 con 0x5599e397c8f0 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.828+0000 7f2ad3e00640 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5599e397c8f0 0x5599e397ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:17.832 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.829+0000 7f2ad3e00640 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5599e397c8f0 0x5599e397ed10 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a766ab39ed4ee5ab server_cookie=92593f84f94fcd1b in_seq=0 out_seq=0 2024-04-27T00:09:17.833 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.829+0000 7f2abaa00640 10 client.15867 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:17.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.843+0000 7f2abaa00640 1 -- 172.21.15.40:0/1374430099 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5599e386ce00 con 0x5599e397c8f0 2024-04-27T00:09:17.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.843+0000 7f2abaa00640 10 client.15867 handle_command_reply: tid=0 2024-04-27T00:09:17.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.843+0000 7f2adea76740 1 client.15867 shutdown 2024-04-27T00:09:17.844 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.844+0000 7f2adea76740 1 -- 172.21.15.40:0/1374430099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 msgr2=0x5599e390f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:17.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.844+0000 7f2adea76740 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e390f230 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2ac801b4d0 tx=0x7f2ac8010dc0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.844+0000 7f2adea76740 1 -- 172.21.15.40:0/1374430099 shutdown_connections 2024-04-27T00:09:17.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.844+0000 7f2adea76740 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5599e397c8f0 0x5599e397ed10 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.844+0000 7f2adea76740 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5599e3906730 0x5599e397aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.844+0000 7f2adea76740 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5599e39015c0 0x5599e390f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.844+0000 7f2adea76740 1 --2- 172.21.15.40:0/1374430099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5599e3900bd0 0x5599e390f230 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.844+0000 7f2adea76740 1 -- 172.21.15.40:0/1374430099 >> 172.21.15.40:0/1374430099 conn(0x5599e38fc240 msgr2=0x5599e38fb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:17.845 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.844+0000 7f2abaa00640 0 client.15867 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:17.846 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.844+0000 7f2adea76740 1 -- 172.21.15.40:0/1374430099 shutdown_connections 2024-04-27T00:09:17.846 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.844+0000 7f2adea76740 1 -- 172.21.15.40:0/1374430099 wait complete. 2024-04-27T00:09:17.846 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.844+0000 7f2adea76740 20 client.15867 trim_cache size 0 max 16384 2024-04-27T00:09:17.846 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:09:17.846 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:09:17.846 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:09:17.846 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:09:17.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.845+0000 7f2adbc00640 1 -- 172.21.15.40:0/3837856262 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2abc0a3d70 msgr2=0x7f2abc0a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:17.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.845+0000 7f2adbc00640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2abc0a3d70 0x7f2abc0a6230 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2ad41a0660 tx=0x7f2ac0005e50 comp rx=0 tx=0).stop 2024-04-27T00:09:17.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.845+0000 7f2adbc00640 1 -- 172.21.15.40:0/3837856262 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2ad410ab70 msgr2=0x7f2ad41a6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:17.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.845+0000 7f2adbc00640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2ad410ab70 0x7f2ad41a6700 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00da10 tx=0x7f2acc00ded0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.845+0000 7f2adbc00640 1 -- 172.21.15.40:0/3837856262 shutdown_connections 2024-04-27T00:09:17.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.845+0000 7f2adbc00640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2ad410ab70 0x7f2ad41a6700 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.845+0000 7f2adbc00640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ad410a1b0 0x7f2ad419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.845+0000 7f2adbc00640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2abc0a3d70 0x7f2abc0a6230 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.845+0000 7f2adbc00640 1 --2- 172.21.15.40:0/3837856262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4108fc0 0x7f2ad419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:17.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.845+0000 7f2adbc00640 1 -- 172.21.15.40:0/3837856262 >> 172.21.15.40:0/3837856262 conn(0x7f2ad4077f80 msgr2=0x7f2ad410d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:17.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.845+0000 7f2adbc00640 1 -- 172.21.15.40:0/3837856262 shutdown_connections 2024-04-27T00:09:17.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:17.845+0000 7f2adbc00640 1 -- 172.21.15.40:0/3837856262 wait complete. 2024-04-27T00:09:18.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:18 smithi080 ceph-mon[32362]: pgmap v640: 129 pgs: 129 active+clean; 1.0 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:09:18.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:18 smithi103 ceph-mon[32952]: pgmap v640: 129 pgs: 129 active+clean; 1.0 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:09:18.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:18 smithi040 ceph-mon[28095]: pgmap v640: 129 pgs: 129 active+clean; 1.0 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:09:20.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:20 smithi080 ceph-mon[32362]: pgmap v641: 129 pgs: 129 active+clean; 1.0 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:09:20.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:20 smithi103 ceph-mon[32952]: pgmap v641: 129 pgs: 129 active+clean; 1.0 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:09:20.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:20 smithi040 ceph-mon[28095]: pgmap v641: 129 pgs: 129 active+clean; 1.0 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:09:22.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:22 smithi103 ceph-mon[32952]: pgmap v642: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:09:22.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:22 smithi040 ceph-mon[28095]: pgmap v642: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:09:22.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:22 smithi080 ceph-mon[32362]: pgmap v642: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:09:24.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:24 smithi103 ceph-mon[32952]: pgmap v643: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:09:24.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:24 smithi040 ceph-mon[28095]: pgmap v643: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:09:24.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:24 smithi080 ceph-mon[32362]: pgmap v643: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:09:26.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:26 smithi103 ceph-mon[32952]: pgmap v644: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:09:26.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:26 smithi040 ceph-mon[28095]: pgmap v644: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:09:26.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:26 smithi080 ceph-mon[32362]: pgmap v644: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:09:28.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:27 smithi040 ceph-mon[28095]: pgmap v645: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:09:28.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:27 smithi080 ceph-mon[32362]: pgmap v645: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:09:28.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:27 smithi103 ceph-mon[32952]: pgmap v645: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:09:30.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:30 smithi103 ceph-mon[32952]: pgmap v646: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:30.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:30 smithi040 ceph-mon[28095]: pgmap v646: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:30.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:30 smithi080 ceph-mon[32362]: pgmap v646: 129 pgs: 129 active+clean; 1.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:30.996 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.995+0000 7f6031800640 1 -- 172.21.15.40:0/3476751217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f602c109fd0 msgr2=0x7f602c10a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.995+0000 7f6031800640 1 --2- 172.21.15.40:0/3476751217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f602c109fd0 0x7f602c10a450 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6014009900 tx=0x7f601402a410 comp rx=0 tx=0).stop 2024-04-27T00:09:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.995+0000 7f6031800640 1 -- 172.21.15.40:0/3476751217 shutdown_connections 2024-04-27T00:09:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.995+0000 7f6031800640 1 --2- 172.21.15.40:0/3476751217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602c10a990 0x7f602c111220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.995+0000 7f6031800640 1 --2- 172.21.15.40:0/3476751217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f602c109fd0 0x7f602c10a450 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.995+0000 7f6031800640 1 --2- 172.21.15.40:0/3476751217 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602c108de0 0x7f602c109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.996+0000 7f6031800640 1 -- 172.21.15.40:0/3476751217 >> 172.21.15.40:0/3476751217 conn(0x7f602c077dc0 msgr2=0x7f602c07a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.996+0000 7f6031800640 1 -- 172.21.15.40:0/3476751217 shutdown_connections 2024-04-27T00:09:30.997 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.996+0000 7f6031800640 1 -- 172.21.15.40:0/3476751217 wait complete. 2024-04-27T00:09:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.997+0000 7f6031800640 1 Processor -- start 2024-04-27T00:09:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.997+0000 7f6031800640 1 -- start start 2024-04-27T00:09:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.997+0000 7f6031800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602c108de0 0x7f602c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.997+0000 7f6031800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602c109fd0 0x7f602c19f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.997+0000 7f6031800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f602c10a990 0x7f602c1a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.997+0000 7f6031800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f602c114940 con 0x7f602c10a990 2024-04-27T00:09:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.997+0000 7f6031800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f602c1147c0 con 0x7f602c109fd0 2024-04-27T00:09:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.997+0000 7f6031800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f602c114ac0 con 0x7f602c108de0 2024-04-27T00:09:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.998+0000 7f602be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602c108de0 0x7f602c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.998+0000 7f602be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602c108de0 0x7f602c19f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43226/0 (socket says 172.21.15.40:43226) 2024-04-27T00:09:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.998+0000 7f6030e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f602c10a990 0x7f602c1a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:30.999 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.998+0000 7f602be00640 1 -- 172.21.15.40:0/2628566433 learned_addr learned my addr 172.21.15.40:0/2628566433 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.998+0000 7f602b400640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602c109fd0 0x7f602c19f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.998+0000 7f602b400640 1 -- 172.21.15.40:0/2628566433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602c108de0 msgr2=0x7f602c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.998+0000 7f602b400640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602c108de0 0x7f602c19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.998+0000 7f602b400640 1 -- 172.21.15.40:0/2628566433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f602c10a990 msgr2=0x7f602c1a65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.998+0000 7f602b400640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f602c10a990 0x7f602c1a65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.998+0000 7f602b400640 1 -- 172.21.15.40:0/2628566433 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f602c1a6c30 con 0x7f602c109fd0 2024-04-27T00:09:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.998+0000 7f6030e00640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f602c10a990 0x7f602c1a65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:09:31.000 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.999+0000 7f602b400640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602c109fd0 0x7f602c19f550 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f601402aa00 tx=0x7f6014003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.999+0000 7f6028c00640 1 -- 172.21.15.40:0/2628566433 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601403f070 con 0x7f602c109fd0 2024-04-27T00:09:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.999+0000 7f6031800640 1 -- 172.21.15.40:0/2628566433 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f602c1a6ec0 con 0x7f602c109fd0 2024-04-27T00:09:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.999+0000 7f6028c00640 1 -- 172.21.15.40:0/2628566433 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6014031c50 con 0x7f602c109fd0 2024-04-27T00:09:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.999+0000 7f6028c00640 1 -- 172.21.15.40:0/2628566433 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6014035030 con 0x7f602c109fd0 2024-04-27T00:09:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:30.999+0000 7f6031800640 1 -- 172.21.15.40:0/2628566433 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f602c1a7320 con 0x7f602c109fd0 2024-04-27T00:09:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.002+0000 7f6035c26740 1 Processor -- start 2024-04-27T00:09:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.002+0000 7f6035c26740 1 -- start start 2024-04-27T00:09:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.002+0000 7f6028c00640 1 -- 172.21.15.40:0/2628566433 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6014042070 con 0x7f602c109fd0 2024-04-27T00:09:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6035c26740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af07de0 0x55961af08200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6035c26740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af087d0 0x55961af0d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6035c26740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af0d940 0x55961af0fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6035c26740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55961ae6c380 con 0x55961af087d0 2024-04-27T00:09:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6035c26740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55961ae08c30 con 0x55961af07de0 2024-04-27T00:09:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6035c26740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55961ae68340 con 0x55961af0d940 2024-04-27T00:09:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6030e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af07de0 0x55961af08200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6030e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af07de0 0x55961af08200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:51840/0 (socket says 172.21.15.40:51840) 2024-04-27T00:09:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6030e00640 1 -- 172.21.15.40:0/1290929965 learned_addr learned my addr 172.21.15.40:0/1290929965 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f602be00640 1 --2- 172.21.15.40:0/1290929965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af087d0 0x55961af0d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f602b400640 1 --2- 172.21.15.40:0/1290929965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af0d940 0x55961af0fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6030e00640 1 -- 172.21.15.40:0/1290929965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af0d940 msgr2=0x55961af0fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6030e00640 1 --2- 172.21.15.40:0/1290929965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af0d940 0x55961af0fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6030e00640 1 -- 172.21.15.40:0/1290929965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af087d0 msgr2=0x55961af0d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6030e00640 1 --2- 172.21.15.40:0/1290929965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af087d0 0x55961af0d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f6030e00640 1 -- 172.21.15.40:0/1290929965 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55961ae2a940 con 0x55961af07de0 2024-04-27T00:09:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.003+0000 7f602b400640 1 --2- 172.21.15.40:0/1290929965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af0d940 0x55961af0fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:09:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.004+0000 7f6030e00640 1 --2- 172.21.15.40:0/1290929965 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af07de0 0x55961af08200 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6024011570 tx=0x7f6024011970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a61d1614e6bd7cf server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.004+0000 7f6021600640 1 -- 172.21.15.40:0/1290929965 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602401e070 con 0x55961af07de0 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.004+0000 7f6021600640 1 -- 172.21.15.40:0/1290929965 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f602400fa10 con 0x55961af07de0 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.004+0000 7f6021600640 1 -- 172.21.15.40:0/1290929965 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024019a90 con 0x55961af07de0 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.004+0000 7f6028c00640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f60100a3c30 0x7f60100a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.004+0000 7f6028c00640 1 -- 172.21.15.40:0/2628566433 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f60140f0f10 con 0x7f602c109fd0 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.005+0000 7f602be00640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f60100a3c30 0x7f60100a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.005+0000 7f6035c26740 1 -- 172.21.15.40:0/1290929965 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af07de0 msgr2=0x55961af08200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.005+0000 7f6035c26740 1 --2- 172.21.15.40:0/1290929965 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af07de0 0x55961af08200 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6024011570 tx=0x7f6024011970 comp rx=0 tx=0).stop 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.005+0000 7f6035c26740 1 -- 172.21.15.40:0/1290929965 shutdown_connections 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.005+0000 7f6035c26740 1 --2- 172.21.15.40:0/1290929965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af0d940 0x55961af0fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.005+0000 7f6035c26740 1 --2- 172.21.15.40:0/1290929965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af087d0 0x55961af0d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.005+0000 7f6035c26740 1 --2- 172.21.15.40:0/1290929965 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af07de0 0x55961af08200 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.005+0000 7f6035c26740 1 -- 172.21.15.40:0/1290929965 >> 172.21.15.40:0/1290929965 conn(0x55961af03450 msgr2=0x55961af03850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.005+0000 7f6035c26740 1 -- 172.21.15.40:0/1290929965 shutdown_connections 2024-04-27T00:09:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.005+0000 7f6035c26740 1 -- 172.21.15.40:0/1290929965 wait complete. 2024-04-27T00:09:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.005+0000 7f602be00640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f60100a3c30 0x7f60100a60f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55961af043c0 tx=0x7f601c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.006+0000 7f6035c26740 1 -- start start 2024-04-27T00:09:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.006+0000 7f6035c26740 1 Processor -- start 2024-04-27T00:09:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.006+0000 7f6035c26740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 0x55961af12050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.006+0000 7f6035c26740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af087d0 0x55961af12590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.006+0000 7f6035c26740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af0d940 0x55961af195d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.006+0000 7f6035c26740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55961af108f0 con 0x55961af07de0 2024-04-27T00:09:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.006+0000 7f6035c26740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55961adf3c60 con 0x55961af0d940 2024-04-27T00:09:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.006+0000 7f6035c26740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55961af10a70 con 0x55961af087d0 2024-04-27T00:09:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.006+0000 7f6035c26740 10 client.0 _ll_register_callbacks cb 0x55961ae1fe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:09:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.007+0000 7f602be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 0x55961af12050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.007+0000 7f602be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 0x55961af12050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44180/0 (socket says 172.21.15.40:44180) 2024-04-27T00:09:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.007+0000 7f602be00640 1 -- 172.21.15.40:0/1339680460 learned_addr learned my addr 172.21.15.40:0/1339680460 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.007+0000 7f6030e00640 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af0d940 0x55961af195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.007+0000 7f602b400640 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af087d0 0x55961af12590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.007+0000 7f602be00640 1 -- 172.21.15.40:0/1339680460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af087d0 msgr2=0x55961af12590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.007+0000 7f602be00640 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af087d0 0x55961af12590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.007+0000 7f602be00640 1 -- 172.21.15.40:0/1339680460 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af0d940 msgr2=0x55961af195d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.007+0000 7f602be00640 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af0d940 0x55961af195d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.007+0000 7f602be00640 1 -- 172.21.15.40:0/1339680460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55961ae02510 con 0x55961af07de0 2024-04-27T00:09:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.007+0000 7f602be00640 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 0x55961af12050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f601c00fad0 tx=0x7f601c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ba3f22a227de5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.007+0000 7f601a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:09:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.008+0000 7f601a000640 1 -- 172.21.15.40:0/1339680460 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c02e060 con 0x55961af07de0 2024-04-27T00:09:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.008+0000 7f601a000640 1 -- 172.21.15.40:0/1339680460 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f601c018030 con 0x55961af07de0 2024-04-27T00:09:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.008+0000 7f6035c26740 1 -- 172.21.15.40:0/1339680460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f602400fa10 con 0x55961af07de0 2024-04-27T00:09:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.008+0000 7f601a000640 1 -- 172.21.15.40:0/1339680460 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f601c035050 con 0x55961af07de0 2024-04-27T00:09:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.008+0000 7f6035c26740 10 client.15879 fetch_fsmap learned FSMap version 24 2024-04-27T00:09:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.008+0000 7f6035c26740 1 -- 172.21.15.40:0/1339680460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55961ab7e210 con 0x55961af07de0 2024-04-27T00:09:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.008+0000 7f601a000640 1 -- 172.21.15.40:0/1339680460 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f602400fa10 con 0x55961af07de0 2024-04-27T00:09:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.009+0000 7f601a000640 1 -- 172.21.15.40:0/1339680460 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f601c01c740 con 0x55961af07de0 2024-04-27T00:09:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.009+0000 7f6035c26740 10 client.15879 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:09:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.009+0000 7f6035c26740 10 client.15879 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:09:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.009+0000 7f6035c26740 20 client.15879 populate_metadata read hostname 'smithi040' 2024-04-27T00:09:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.009+0000 7f6035c26740 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55961af1db80 0x55961af20010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.009+0000 7f6035c26740 4 client.15879 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:09:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.009+0000 7f6035c26740 1 -- 172.21.15.40:0/1339680460 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55961ae79f90 con 0x55961af1db80 2024-04-27T00:09:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.009+0000 7f602b400640 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55961af1db80 0x55961af20010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.010+0000 7f602b400640 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55961af1db80 0x55961af20010 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2953da4d5af37098 server_cookie=56d59889e6ce158f in_seq=0 out_seq=0 2024-04-27T00:09:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.010+0000 7f601a000640 10 client.15879 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f601a000640 1 -- 172.21.15.40:0/1339680460 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55961ae79f90 con 0x55961af1db80 2024-04-27T00:09:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f601a000640 10 client.15879 handle_command_reply: tid=0 2024-04-27T00:09:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f6035c26740 1 client.15879 shutdown 2024-04-27T00:09:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f6035c26740 1 -- 172.21.15.40:0/1339680460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 msgr2=0x55961af12050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f6035c26740 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 0x55961af12050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f601c00fad0 tx=0x7f601c004a40 comp rx=0 tx=0).stop 2024-04-27T00:09:31.016 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f6035c26740 1 -- 172.21.15.40:0/1339680460 shutdown_connections 2024-04-27T00:09:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f6035c26740 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55961af1db80 0x55961af20010 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f6035c26740 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af0d940 0x55961af195d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f6035c26740 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af087d0 0x55961af12590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f6035c26740 1 --2- 172.21.15.40:0/1339680460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 0x55961af12050 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f6035c26740 1 -- 172.21.15.40:0/1339680460 >> 172.21.15.40:0/1339680460 conn(0x55961af03450 msgr2=0x55961af02650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f601a000640 0 client.15879 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f6035c26740 1 -- 172.21.15.40:0/1339680460 shutdown_connections 2024-04-27T00:09:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f6035c26740 1 -- 172.21.15.40:0/1339680460 wait complete. 2024-04-27T00:09:31.017 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.011+0000 7f6035c26740 20 client.15879 trim_cache size 0 max 16384 2024-04-27T00:09:31.029 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.028+0000 7f6035c26740 1 Processor -- start 2024-04-27T00:09:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.028+0000 7f6035c26740 1 -- start start 2024-04-27T00:09:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.028+0000 7f6035c26740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af07de0 0x55961af124f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.028+0000 7f6035c26740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af087d0 0x55961af19590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.028+0000 7f6035c26740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af0d940 0x55961af06b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.028+0000 7f6035c26740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f602c1b41d0 con 0x55961af087d0 2024-04-27T00:09:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.028+0000 7f6035c26740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f602c1b1aa0 con 0x55961af0d940 2024-04-27T00:09:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.028+0000 7f6035c26740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f602c1af8f0 con 0x55961af07de0 2024-04-27T00:09:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.028+0000 7f602b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af087d0 0x55961af19590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.028+0000 7f602b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af087d0 0x55961af19590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44192/0 (socket says 172.21.15.40:44192) 2024-04-27T00:09:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.028+0000 7f602b400640 1 -- 172.21.15.40:0/655050764 learned_addr learned my addr 172.21.15.40:0/655050764 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.028+0000 7f602be00640 1 --2- 172.21.15.40:0/655050764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af07de0 0x55961af124f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.029+0000 7f6030e00640 1 --2- 172.21.15.40:0/655050764 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af0d940 0x55961af06b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.029+0000 7f602b400640 1 -- 172.21.15.40:0/655050764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af07de0 msgr2=0x55961af124f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:31.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.029+0000 7f602b400640 1 --2- 172.21.15.40:0/655050764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af07de0 0x55961af124f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.029+0000 7f602b400640 1 -- 172.21.15.40:0/655050764 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af0d940 msgr2=0x55961af06b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:31.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.029+0000 7f602b400640 1 --2- 172.21.15.40:0/655050764 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af0d940 0x55961af06b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.029+0000 7f602b400640 1 -- 172.21.15.40:0/655050764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55961ac4a620 con 0x55961af087d0 2024-04-27T00:09:31.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.029+0000 7f6030e00640 1 --2- 172.21.15.40:0/655050764 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af0d940 0x55961af06b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:09:31.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.029+0000 7f602b400640 1 --2- 172.21.15.40:0/655050764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af087d0 0x55961af19590 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6014000c00 tx=0x7f60140dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4cf49fc3701b2d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.029+0000 7f6020c00640 1 -- 172.21.15.40:0/655050764 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6014049040 con 0x55961af087d0 2024-04-27T00:09:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.029+0000 7f6020c00640 1 -- 172.21.15.40:0/655050764 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6014031e00 con 0x55961af087d0 2024-04-27T00:09:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.029+0000 7f6020c00640 1 -- 172.21.15.40:0/655050764 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60140c5960 con 0x55961af087d0 2024-04-27T00:09:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.030+0000 7f6035c26740 1 -- 172.21.15.40:0/655050764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af087d0 msgr2=0x55961af19590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.030+0000 7f6035c26740 1 --2- 172.21.15.40:0/655050764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af087d0 0x55961af19590 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6014000c00 tx=0x7f60140dd040 comp rx=0 tx=0).stop 2024-04-27T00:09:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.030+0000 7f6035c26740 1 -- 172.21.15.40:0/655050764 shutdown_connections 2024-04-27T00:09:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.030+0000 7f6035c26740 1 --2- 172.21.15.40:0/655050764 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af0d940 0x55961af06b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.030+0000 7f6035c26740 1 --2- 172.21.15.40:0/655050764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af087d0 0x55961af19590 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.030+0000 7f6035c26740 1 --2- 172.21.15.40:0/655050764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af07de0 0x55961af124f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.030+0000 7f6035c26740 1 -- 172.21.15.40:0/655050764 >> 172.21.15.40:0/655050764 conn(0x55961af03450 msgr2=0x55961af06400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.030+0000 7f6035c26740 1 -- 172.21.15.40:0/655050764 shutdown_connections 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.030+0000 7f6035c26740 1 -- 172.21.15.40:0/655050764 wait complete. 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.030+0000 7f6035c26740 1 -- start start 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.031+0000 7f6035c26740 1 Processor -- start 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.031+0000 7f6035c26740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 0x55961af171c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.031+0000 7f6035c26740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af087d0 0x55961af17700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.031+0000 7f6035c26740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af0d940 0x55961af18650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.031+0000 7f6035c26740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f602c1aff60 con 0x55961af07de0 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.031+0000 7f6035c26740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f602c1b13d0 con 0x55961af087d0 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.031+0000 7f6035c26740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f602c1b0630 con 0x55961af0d940 2024-04-27T00:09:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.031+0000 7f6035c26740 10 client.0 _ll_register_callbacks cb 0x7f601c009e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.031+0000 7f602be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 0x55961af171c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.031+0000 7f602be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 0x55961af171c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44208/0 (socket says 172.21.15.40:44208) 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.031+0000 7f602be00640 1 -- 172.21.15.40:0/3245101692 learned_addr learned my addr 172.21.15.40:0/3245101692 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.032+0000 7f602b400640 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af087d0 0x55961af17700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.032+0000 7f6030e00640 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af0d940 0x55961af18650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.032+0000 7f602be00640 1 -- 172.21.15.40:0/3245101692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af0d940 msgr2=0x55961af18650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.032+0000 7f602be00640 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af0d940 0x55961af18650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.032+0000 7f602be00640 1 -- 172.21.15.40:0/3245101692 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af087d0 msgr2=0x55961af17700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.032+0000 7f602be00640 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af087d0 0x55961af17700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.032+0000 7f602be00640 1 -- 172.21.15.40:0/3245101692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55961ae26cb0 con 0x55961af07de0 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.032+0000 7f602b400640 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af087d0 0x55961af17700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.032+0000 7f602be00640 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 0x55961af171c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f601c009fd0 tx=0x7f601c00f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0693e23b93bd916 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.032+0000 7f601a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:09:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.032+0000 7f601a000640 1 -- 172.21.15.40:0/3245101692 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c03d070 con 0x55961af07de0 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f601a000640 1 -- 172.21.15.40:0/3245101692 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f601c004e00 con 0x55961af07de0 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f6035c26740 1 -- 172.21.15.40:0/3245101692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55961ae73e00 con 0x55961af07de0 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f601a000640 1 -- 172.21.15.40:0/3245101692 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f601c049050 con 0x55961af07de0 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f6035c26740 10 client.15885 fetch_fsmap learned FSMap version 24 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f6035c26740 1 -- 172.21.15.40:0/3245101692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55961af843d0 con 0x55961af07de0 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f601a000640 1 -- 172.21.15.40:0/3245101692 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55961ae73e00 con 0x55961af07de0 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f601a000640 1 -- 172.21.15.40:0/3245101692 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f601c01ea50 con 0x55961af07de0 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f6035c26740 10 client.15885 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f6035c26740 10 client.15885 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f6035c26740 20 client.15885 populate_metadata read hostname 'smithi040' 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f6035c26740 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55961af84da0 0x55961af871c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f6035c26740 4 client.15885 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:09:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.033+0000 7f6035c26740 1 -- 172.21.15.40:0/3245101692 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55961adef4b0 con 0x55961af84da0 2024-04-27T00:09:31.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.034+0000 7f602b400640 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55961af84da0 0x55961af871c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:31.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.034+0000 7f602b400640 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55961af84da0 0x55961af871c0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c690fae19654326 server_cookie=c5a8267fae51ba97 in_seq=0 out_seq=0 2024-04-27T00:09:31.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.034+0000 7f601a000640 10 client.15885 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:31.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.048+0000 7f601a000640 1 -- 172.21.15.40:0/3245101692 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55961adef4b0 con 0x55961af84da0 2024-04-27T00:09:31.050 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.048+0000 7f601a000640 10 client.15885 handle_command_reply: tid=0 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.048+0000 7f6035c26740 1 client.15885 shutdown 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.049+0000 7f6035c26740 1 -- 172.21.15.40:0/3245101692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 msgr2=0x55961af171c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.049+0000 7f6035c26740 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 0x55961af171c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f601c009fd0 tx=0x7f601c00f9c0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.049+0000 7f6035c26740 1 -- 172.21.15.40:0/3245101692 shutdown_connections 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.049+0000 7f6035c26740 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55961af84da0 0x55961af871c0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.049+0000 7f6035c26740 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55961af0d940 0x55961af18650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.049+0000 7f6035c26740 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55961af087d0 0x55961af17700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.049+0000 7f601a000640 0 client.15885 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.049+0000 7f6035c26740 1 --2- 172.21.15.40:0/3245101692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55961af07de0 0x55961af171c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.049+0000 7f6035c26740 1 -- 172.21.15.40:0/3245101692 >> 172.21.15.40:0/3245101692 conn(0x55961af03450 msgr2=0x55961af12350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.049+0000 7f6035c26740 1 -- 172.21.15.40:0/3245101692 shutdown_connections 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.049+0000 7f6035c26740 1 -- 172.21.15.40:0/3245101692 wait complete. 2024-04-27T00:09:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.049+0000 7f6035c26740 20 client.15885 trim_cache size 0 max 16384 2024-04-27T00:09:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:09:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:09:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:09:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:09:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.050+0000 7f6031800640 1 -- 172.21.15.40:0/2628566433 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f60100a3c30 msgr2=0x7f60100a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.050+0000 7f6031800640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f60100a3c30 0x7f60100a60f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55961af043c0 tx=0x7f601c009500 comp rx=0 tx=0).stop 2024-04-27T00:09:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.050+0000 7f6031800640 1 -- 172.21.15.40:0/2628566433 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602c109fd0 msgr2=0x7f602c19f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.050+0000 7f6031800640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602c109fd0 0x7f602c19f550 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f601402aa00 tx=0x7f6014003c10 comp rx=0 tx=0).stop 2024-04-27T00:09:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.050+0000 7f6031800640 1 -- 172.21.15.40:0/2628566433 shutdown_connections 2024-04-27T00:09:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.050+0000 7f6031800640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f602c10a990 0x7f602c1a65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.050+0000 7f6031800640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602c109fd0 0x7f602c19f550 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.050+0000 7f6031800640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f60100a3c30 0x7f60100a60f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.050+0000 7f6031800640 1 --2- 172.21.15.40:0/2628566433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602c108de0 0x7f602c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.050+0000 7f6031800640 1 -- 172.21.15.40:0/2628566433 >> 172.21.15.40:0/2628566433 conn(0x7f602c077dc0 msgr2=0x7f602c10d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.050+0000 7f6031800640 1 -- 172.21.15.40:0/2628566433 shutdown_connections 2024-04-27T00:09:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:31.050+0000 7f6031800640 1 -- 172.21.15.40:0/2628566433 wait complete. 2024-04-27T00:09:32.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:32 smithi103 ceph-mon[32952]: pgmap v647: 129 pgs: 129 active+clean; 1.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-04-27T00:09:32.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:32 smithi040 ceph-mon[28095]: pgmap v647: 129 pgs: 129 active+clean; 1.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-04-27T00:09:32.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:32 smithi080 ceph-mon[32362]: pgmap v647: 129 pgs: 129 active+clean; 1.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-04-27T00:09:34.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:34 smithi040 ceph-mon[28095]: pgmap v648: 129 pgs: 129 active+clean; 1.2 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:09:34.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:34 smithi080 ceph-mon[32362]: pgmap v648: 129 pgs: 129 active+clean; 1.2 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:09:34.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:34 smithi103 ceph-mon[32952]: pgmap v648: 129 pgs: 129 active+clean; 1.2 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:09:36.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:36 smithi040 ceph-mon[28095]: pgmap v649: 129 pgs: 129 active+clean; 1.2 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:09:36.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:36 smithi080 ceph-mon[32362]: pgmap v649: 129 pgs: 129 active+clean; 1.2 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:09:36.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:36 smithi103 ceph-mon[32952]: pgmap v649: 129 pgs: 129 active+clean; 1.2 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:09:37.614 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.614 INFO:tasks.workunit.client.0.smithi040.stdout:313/600: open fd = 3 2024-04-27T00:09:37.614 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.614 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.614 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.614 INFO:tasks.workunit.client.0.smithi040.stdout:314/600: open fd = 3 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:315/600: open fd = 3 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:316/600: open fd = 3 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:317/600: open fd = 3 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.615 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.616 INFO:tasks.workunit.client.0.smithi040.stdout:318/600: open fd = 3 2024-04-27T00:09:37.616 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.616 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.616 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.616 INFO:tasks.workunit.client.0.smithi040.stdout:319/600: open fd = 3 2024-04-27T00:09:37.616 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.616 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.616 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.616 INFO:tasks.workunit.client.0.smithi040.stdout:320/600: open fd = 3 2024-04-27T00:09:37.616 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.616 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.617 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.617 INFO:tasks.workunit.client.0.smithi040.stdout:321/600: open fd = 3 2024-04-27T00:09:37.617 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.617 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.617 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.617 INFO:tasks.workunit.client.0.smithi040.stdout:322/600: open fd = 3 2024-04-27T00:09:37.618 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.618 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.618 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.618 INFO:tasks.workunit.client.0.smithi040.stdout:323/600: open fd = 3 2024-04-27T00:09:37.618 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.618 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.618 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.618 INFO:tasks.workunit.client.0.smithi040.stdout:324/600: open fd = 3 2024-04-27T00:09:37.618 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.618 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.618 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:325/600: open fd = 3 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:326/600: open fd = 3 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:327/600: open fd = 3 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:328/600: open fd = 3 2024-04-27T00:09:37.619 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:329/600: open fd = 3 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:330/600: open fd = 3 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:331/600: open fd = 3 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.620 INFO:tasks.workunit.client.0.smithi040.stdout:332/600: open fd = 3 2024-04-27T00:09:37.621 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.621 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.621 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.621 INFO:tasks.workunit.client.0.smithi040.stdout:333/600: open fd = 3 2024-04-27T00:09:37.621 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.621 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.621 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.621 INFO:tasks.workunit.client.0.smithi040.stdout:334/600: open fd = 3 2024-04-27T00:09:37.621 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.621 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.622 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.622 INFO:tasks.workunit.client.0.smithi040.stdout:335/600: open fd = 3 2024-04-27T00:09:37.622 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.622 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.622 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.622 INFO:tasks.workunit.client.0.smithi040.stdout:336/600: open fd = 3 2024-04-27T00:09:37.622 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.622 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.622 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.623 INFO:tasks.workunit.client.0.smithi040.stdout:337/600: open fd = 3 2024-04-27T00:09:37.623 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.623 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.623 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.623 INFO:tasks.workunit.client.0.smithi040.stdout:338/600: open fd = 3 2024-04-27T00:09:37.623 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.623 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.623 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.623 INFO:tasks.workunit.client.0.smithi040.stdout:339/600: open fd = 3 2024-04-27T00:09:37.623 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.623 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.624 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.624 INFO:tasks.workunit.client.0.smithi040.stdout:340/600: open fd = 3 2024-04-27T00:09:37.625 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.626 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.626 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.626 INFO:tasks.workunit.client.0.smithi040.stdout:341/600: open fd = 3 2024-04-27T00:09:37.626 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.626 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.626 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.626 INFO:tasks.workunit.client.0.smithi040.stdout:342/600: open fd = 3 2024-04-27T00:09:37.626 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.626 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.626 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.626 INFO:tasks.workunit.client.0.smithi040.stdout:343/600: open fd = 3 2024-04-27T00:09:37.627 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.627 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.627 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.627 INFO:tasks.workunit.client.0.smithi040.stdout:344/600: open fd = 3 2024-04-27T00:09:37.627 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.627 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.627 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.628 INFO:tasks.workunit.client.0.smithi040.stdout:345/600: open fd = 3 2024-04-27T00:09:37.628 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.628 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.628 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.628 INFO:tasks.workunit.client.0.smithi040.stdout:346/600: open fd = 3 2024-04-27T00:09:37.628 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.629 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.629 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.629 INFO:tasks.workunit.client.0.smithi040.stdout:347/600: open fd = 3 2024-04-27T00:09:37.629 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.629 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.629 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:348/600: open fd = 3 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:349/600: open fd = 3 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:350/600: open fd = 3 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.630 INFO:tasks.workunit.client.0.smithi040.stdout:351/600: open fd = 3 2024-04-27T00:09:37.631 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.631 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.631 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.631 INFO:tasks.workunit.client.0.smithi040.stdout:352/600: open fd = 3 2024-04-27T00:09:37.631 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.631 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.631 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.631 INFO:tasks.workunit.client.0.smithi040.stdout:353/600: open fd = 3 2024-04-27T00:09:37.631 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.631 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.631 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.631 INFO:tasks.workunit.client.0.smithi040.stdout:354/600: open fd = 3 2024-04-27T00:09:37.632 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.632 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.632 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.632 INFO:tasks.workunit.client.0.smithi040.stdout:355/600: open fd = 3 2024-04-27T00:09:37.632 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.632 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.632 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.632 INFO:tasks.workunit.client.0.smithi040.stdout:356/600: open fd = 3 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:357/600: open fd = 3 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:358/600: open fd = 3 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:359/600: open fd = 3 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.633 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.634 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.634 INFO:tasks.workunit.client.0.smithi040.stdout:360/600: open fd = 3 2024-04-27T00:09:37.634 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.634 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.634 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.634 INFO:tasks.workunit.client.0.smithi040.stdout:361/600: open fd = 3 2024-04-27T00:09:37.634 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.634 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.634 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.635 INFO:tasks.workunit.client.0.smithi040.stdout:362/600: open fd = 3 2024-04-27T00:09:37.635 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.635 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.635 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.635 INFO:tasks.workunit.client.0.smithi040.stdout:363/600: open fd = 3 2024-04-27T00:09:37.635 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.635 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.636 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.636 INFO:tasks.workunit.client.0.smithi040.stdout:364/600: open fd = 3 2024-04-27T00:09:37.636 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.636 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.636 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.636 INFO:tasks.workunit.client.0.smithi040.stdout:365/600: open fd = 3 2024-04-27T00:09:37.637 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.637 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.637 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.637 INFO:tasks.workunit.client.0.smithi040.stdout:366/600: open fd = 3 2024-04-27T00:09:37.637 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.637 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.637 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.638 INFO:tasks.workunit.client.0.smithi040.stdout:367/600: open fd = 3 2024-04-27T00:09:37.638 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.638 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.638 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.638 INFO:tasks.workunit.client.0.smithi040.stdout:368/600: open fd = 3 2024-04-27T00:09:37.638 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:369/600: open fd = 3 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:370/600: open fd = 3 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:371/600: open fd = 3 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.639 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:372/600: open fd = 3 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:373/600: open fd = 3 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:374/600: open fd = 3 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:09:37.640 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:09:38.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:38 smithi040 ceph-mon[28095]: pgmap v650: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:09:38.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:38 smithi080 ceph-mon[32362]: pgmap v650: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:09:38.598 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:38 smithi103 ceph-mon[32952]: pgmap v650: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:09:40.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:40 smithi040 ceph-mon[28095]: pgmap v651: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:09:40.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:40 smithi080 ceph-mon[32362]: pgmap v651: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:09:40.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:40 smithi103 ceph-mon[32952]: pgmap v651: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:09:42.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:42 smithi040 ceph-mon[28095]: pgmap v652: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:09:42.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:42 smithi080 ceph-mon[32362]: pgmap v652: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:09:42.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:42 smithi103 ceph-mon[32952]: pgmap v652: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:09:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.195+0000 7f13b8800640 1 -- 172.21.15.40:0/3518707139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b0104540 msgr2=0x7f13b010add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.195+0000 7f13b8800640 1 --2- 172.21.15.40:0/3518707139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b0104540 0x7f13b010add0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f13ac009ca0 tx=0x7f13ac02a8b0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.195+0000 7f13b8800640 1 -- 172.21.15.40:0/3518707139 shutdown_connections 2024-04-27T00:09:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.195+0000 7f13b8800640 1 --2- 172.21.15.40:0/3518707139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b0104540 0x7f13b010add0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.195+0000 7f13b8800640 1 --2- 172.21.15.40:0/3518707139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b0103b80 0x7f13b0104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.195+0000 7f13b8800640 1 --2- 172.21.15.40:0/3518707139 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13b0102990 0x7f13b0102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.195+0000 7f13b8800640 1 -- 172.21.15.40:0/3518707139 >> 172.21.15.40:0/3518707139 conn(0x7f13b00fdf20 msgr2=0x7f13b0100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.195+0000 7f13b8800640 1 -- 172.21.15.40:0/3518707139 shutdown_connections 2024-04-27T00:09:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.196+0000 7f13b8800640 1 -- 172.21.15.40:0/3518707139 wait complete. 2024-04-27T00:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.197+0000 7f13b8800640 1 Processor -- start 2024-04-27T00:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.197+0000 7f13b8800640 1 -- start start 2024-04-27T00:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.197+0000 7f13b8800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b0102990 0x7f13b019acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.197+0000 7f13b8800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b0103b80 0x7f13b019b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.197+0000 7f13b8800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13b0104540 0x7f13b01a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.197+0000 7f13b8800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f13b010e4f0 con 0x7f13b0102990 2024-04-27T00:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.197+0000 7f13b8800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f13b010e370 con 0x7f13b0104540 2024-04-27T00:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.197+0000 7f13b8800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13b010e670 con 0x7f13b0103b80 2024-04-27T00:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.198+0000 7f13b5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b0102990 0x7f13b019acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.198+0000 7f13b5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b0102990 0x7f13b019acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38312/0 (socket says 172.21.15.40:38312) 2024-04-27T00:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.198+0000 7f13b5e00640 1 -- 172.21.15.40:0/2592533754 learned_addr learned my addr 172.21.15.40:0/2592533754 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.198+0000 7f13b5400640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b0103b80 0x7f13b019b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.198+0000 7f13b6800640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13b0104540 0x7f13b01a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.198+0000 7f13b5e00640 1 -- 172.21.15.40:0/2592533754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b0103b80 msgr2=0x7f13b019b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.198+0000 7f13b5e00640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b0103b80 0x7f13b019b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.198+0000 7f13b5e00640 1 -- 172.21.15.40:0/2592533754 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13b0104540 msgr2=0x7f13b01a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.198+0000 7f13b5e00640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13b0104540 0x7f13b01a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.198+0000 7f13b5e00640 1 -- 172.21.15.40:0/2592533754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b01a2900 con 0x7f13b0102990 2024-04-27T00:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.198+0000 7f13b6800640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13b0104540 0x7f13b01a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:09:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.198+0000 7f13b5e00640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b0102990 0x7f13b019acc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f13a400e9c0 tx=0x7f13a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.199+0000 7f13aaa00640 1 -- 172.21.15.40:0/2592533754 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a400cd80 con 0x7f13b0102990 2024-04-27T00:09:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.199+0000 7f13aaa00640 1 -- 172.21.15.40:0/2592533754 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a40035f0 con 0x7f13b0102990 2024-04-27T00:09:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.199+0000 7f13aaa00640 1 -- 172.21.15.40:0/2592533754 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4010690 con 0x7f13b0102990 2024-04-27T00:09:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.199+0000 7f13b8800640 1 -- 172.21.15.40:0/2592533754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b01a2bf0 con 0x7f13b0102990 2024-04-27T00:09:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.199+0000 7f13b8800640 1 -- 172.21.15.40:0/2592533754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b01a3140 con 0x7f13b0102990 2024-04-27T00:09:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.202+0000 7f13bb5eb740 1 Processor -- start 2024-04-27T00:09:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.202+0000 7f13bb5eb740 1 -- start start 2024-04-27T00:09:44.203 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.202+0000 7f13aaa00640 1 -- 172.21.15.40:0/2592533754 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f13a40037a0 con 0x7f13b0102990 2024-04-27T00:09:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.203+0000 7f13bb5eb740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e85557bd0 0x559e85557ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.203+0000 7f13bb5eb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e855585c0 0x559e8555d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.203+0000 7f13bb5eb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e8555fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.204 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.203+0000 7f13bb5eb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559e854bc380 con 0x559e8555d730 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.203+0000 7f13bb5eb740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x559e85458c30 con 0x559e85557bd0 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.203+0000 7f13bb5eb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559e854b8340 con 0x559e855585c0 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.203+0000 7f13b5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e8555fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.203+0000 7f13b5e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e855585c0 0x559e8555d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.203+0000 7f13b5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e8555fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38328/0 (socket says 172.21.15.40:38328) 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.203+0000 7f13b5400640 1 -- 172.21.15.40:0/135251982 learned_addr learned my addr 172.21.15.40:0/135251982 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.203+0000 7f13b6800640 1 --2- 172.21.15.40:0/135251982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e85557bd0 0x559e85557ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.204+0000 7f13b5400640 1 -- 172.21.15.40:0/135251982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e855585c0 msgr2=0x559e8555d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.204+0000 7f13b5400640 1 --2- 172.21.15.40:0/135251982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e855585c0 0x559e8555d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.204+0000 7f13b5400640 1 -- 172.21.15.40:0/135251982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e85557bd0 msgr2=0x559e85557ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.204+0000 7f13b5400640 1 --2- 172.21.15.40:0/135251982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e85557bd0 0x559e85557ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.204+0000 7f13b5400640 1 -- 172.21.15.40:0/135251982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e8547a940 con 0x559e8555d730 2024-04-27T00:09:44.205 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.204+0000 7f13b5400640 1 --2- 172.21.15.40:0/135251982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e8555fbc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009c90 tx=0x7f13a0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60509241122334d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.204+0000 7f139f000640 1 -- 172.21.15.40:0/135251982 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a001e070 con 0x559e8555d730 2024-04-27T00:09:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.204+0000 7f139f000640 1 -- 172.21.15.40:0/135251982 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a0019ae0 con 0x559e8555d730 2024-04-27T00:09:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.204+0000 7f13aaa00640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f13980a3cf0 0x7f13980a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.204+0000 7f139f000640 1 -- 172.21.15.40:0/135251982 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a000ea40 con 0x559e8555d730 2024-04-27T00:09:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.205+0000 7f13aaa00640 1 -- 172.21.15.40:0/2592533754 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f13a40d0da0 con 0x7f13b0102990 2024-04-27T00:09:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.205+0000 7f13b6800640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f13980a3cf0 0x7f13980a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.205+0000 7f13bb5eb740 1 -- 172.21.15.40:0/135251982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 msgr2=0x559e8555fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.205+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/135251982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e8555fbc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009c90 tx=0x7f13a0011810 comp rx=0 tx=0).stop 2024-04-27T00:09:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.205+0000 7f13bb5eb740 1 -- 172.21.15.40:0/135251982 shutdown_connections 2024-04-27T00:09:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.205+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/135251982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e8555fbc0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.205+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/135251982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e855585c0 0x559e8555d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.205+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/135251982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e85557bd0 0x559e85557ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.205+0000 7f13bb5eb740 1 -- 172.21.15.40:0/135251982 >> 172.21.15.40:0/135251982 conn(0x559e85553240 msgr2=0x559e85553640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.205+0000 7f13bb5eb740 1 -- 172.21.15.40:0/135251982 shutdown_connections 2024-04-27T00:09:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.205+0000 7f13bb5eb740 1 -- 172.21.15.40:0/135251982 wait complete. 2024-04-27T00:09:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.205+0000 7f13b6800640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f13980a3cf0 0x7f13980a61b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f13ac004af0 tx=0x7f13ac005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.206+0000 7f13bb5eb740 1 -- start start 2024-04-27T00:09:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.206+0000 7f13bb5eb740 1 Processor -- start 2024-04-27T00:09:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.206+0000 7f13bb5eb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e85557bd0 0x559e85561e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.206+0000 7f13bb5eb740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 0x559e85562350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.206+0000 7f13bb5eb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e85569320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13bb5eb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559e855606e0 con 0x559e8555d730 2024-04-27T00:09:44.210 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13bb5eb740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x559e85443c60 con 0x559e855585c0 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13bb5eb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559e85560860 con 0x559e85557bd0 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13bb5eb740 10 client.0 _ll_register_callbacks cb 0x559e8546fe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13b6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e85569320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13b5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 0x559e85562350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13b5e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e85557bd0 0x559e85561e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13b5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 0x559e85562350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:47142/0 (socket says 172.21.15.40:47142) 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13b5400640 1 -- 172.21.15.40:0/4137801470 learned_addr learned my addr 172.21.15.40:0/4137801470 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13b5400640 1 -- 172.21.15.40:0/4137801470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e85557bd0 msgr2=0x559e85561e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13b5400640 1 --2- 172.21.15.40:0/4137801470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e85557bd0 0x559e85561e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13b5400640 1 -- 172.21.15.40:0/4137801470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 msgr2=0x559e85569320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13b5400640 1 --2- 172.21.15.40:0/4137801470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e85569320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.207+0000 7f13b5400640 1 -- 172.21.15.40:0/4137801470 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e85452510 con 0x559e855585c0 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.208+0000 7f13b5400640 1 --2- 172.21.15.40:0/4137801470 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 0x559e85562350 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f13a0019660 tx=0x7f13a0019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28495ebfb0b20e7a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.208+0000 7f1393400640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:09:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.208+0000 7f1393400640 1 -- 172.21.15.40:0/4137801470 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0029030 con 0x559e855585c0 2024-04-27T00:09:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.208+0000 7f1393400640 1 -- 172.21.15.40:0/4137801470 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f13a00198b0 con 0x559e855585c0 2024-04-27T00:09:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.208+0000 7f13bb5eb740 1 -- 172.21.15.40:0/4137801470 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a0019ae0 con 0x559e855585c0 2024-04-27T00:09:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.208+0000 7f1393400640 1 -- 172.21.15.40:0/4137801470 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f13a0033050 con 0x559e855585c0 2024-04-27T00:09:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.208+0000 7f13bb5eb740 10 client.26012 fetch_fsmap learned FSMap version 24 2024-04-27T00:09:44.212 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.208+0000 7f13bb5eb740 1 -- 172.21.15.40:0/4137801470 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559e851ce210 con 0x559e855585c0 2024-04-27T00:09:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.209+0000 7f1393400640 1 -- 172.21.15.40:0/4137801470 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a0019ae0 con 0x559e855585c0 2024-04-27T00:09:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.209+0000 7f1393400640 1 -- 172.21.15.40:0/4137801470 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f13a00054a0 con 0x559e855585c0 2024-04-27T00:09:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.209+0000 7f13bb5eb740 10 client.26012 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:09:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.209+0000 7f13bb5eb740 10 client.26012 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:09:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.209+0000 7f13bb5eb740 20 client.26012 populate_metadata read hostname 'smithi040' 2024-04-27T00:09:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.209+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/4137801470 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559e8556d700 0x559e8556faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.209+0000 7f13bb5eb740 4 client.26012 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:09:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.209+0000 7f13bb5eb740 1 -- 172.21.15.40:0/4137801470 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e854c9f90 con 0x559e8556d700 2024-04-27T00:09:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.210+0000 7f13b5e00640 1 --2- 172.21.15.40:0/4137801470 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559e8556d700 0x559e8556faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.210+0000 7f13b5e00640 1 --2- 172.21.15.40:0/4137801470 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559e8556d700 0x559e8556faf0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17dee86cf75f50a5 server_cookie=137aea848789ed6f in_seq=0 out_seq=0 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.210+0000 7f1393400640 10 client.26012 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f1393400640 1 -- 172.21.15.40:0/4137801470 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e854c9f90 con 0x559e8556d700 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f1393400640 10 client.26012 handle_command_reply: tid=0 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f13bb5eb740 1 client.26012 shutdown 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f13bb5eb740 1 -- 172.21.15.40:0/4137801470 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 msgr2=0x559e85562350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/4137801470 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 0x559e85562350 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f13a0019660 tx=0x7f13a0019690 comp rx=0 tx=0).stop 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f13bb5eb740 1 -- 172.21.15.40:0/4137801470 shutdown_connections 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/4137801470 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559e8556d700 0x559e8556faf0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/4137801470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e85569320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f1393400640 0 client.26012 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/4137801470 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 0x559e85562350 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/4137801470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e85557bd0 0x559e85561e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f13bb5eb740 1 -- 172.21.15.40:0/4137801470 >> 172.21.15.40:0/4137801470 conn(0x559e85553240 msgr2=0x559e85552440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f13bb5eb740 1 -- 172.21.15.40:0/4137801470 shutdown_connections 2024-04-27T00:09:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f13bb5eb740 1 -- 172.21.15.40:0/4137801470 wait complete. 2024-04-27T00:09:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.211+0000 7f13bb5eb740 20 client.26012 trim_cache size 0 max 16384 2024-04-27T00:09:44.236 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.228+0000 7f13bb5eb740 1 Processor -- start 2024-04-27T00:09:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.228+0000 7f13bb5eb740 1 -- start start 2024-04-27T00:09:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.228+0000 7f13bb5eb740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e85557bd0 0x559e85561e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.228+0000 7f13bb5eb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e855585c0 0x559e85562350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.228+0000 7f13bb5eb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e855692e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.228+0000 7f13bb5eb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01affb0 con 0x559e8555d730 2024-04-27T00:09:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.228+0000 7f13bb5eb740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01ad880 con 0x559e85557bd0 2024-04-27T00:09:44.238 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.228+0000 7f13bb5eb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01ab6d0 con 0x559e855585c0 2024-04-27T00:09:44.238 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.229+0000 7f13b6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e855692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.238 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.229+0000 7f13b5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e855585c0 0x559e85562350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.238 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.229+0000 7f13b5e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e85557bd0 0x559e85561e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.238 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.229+0000 7f13b5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e855585c0 0x559e85562350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46290/0 (socket says 172.21.15.40:46290) 2024-04-27T00:09:44.238 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.229+0000 7f13b5400640 1 -- 172.21.15.40:0/1399335034 learned_addr learned my addr 172.21.15.40:0/1399335034 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.229+0000 7f13b5400640 1 -- 172.21.15.40:0/1399335034 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e85557bd0 msgr2=0x559e85561e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.229+0000 7f13b5400640 1 --2- 172.21.15.40:0/1399335034 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e85557bd0 0x559e85561e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.229+0000 7f13b5400640 1 -- 172.21.15.40:0/1399335034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 msgr2=0x559e855692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.229+0000 7f13b5400640 1 --2- 172.21.15.40:0/1399335034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e855692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.229+0000 7f13b5400640 1 -- 172.21.15.40:0/1399335034 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e8529a620 con 0x559e855585c0 2024-04-27T00:09:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.229+0000 7f13b5400640 1 --2- 172.21.15.40:0/1399335034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e855585c0 0x559e85562350 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009be0 tx=0x7f13a0012f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=357286a10e031b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.230+0000 7f139e600640 1 -- 172.21.15.40:0/1399335034 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0029070 con 0x559e855585c0 2024-04-27T00:09:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.230+0000 7f139e600640 1 -- 172.21.15.40:0/1399335034 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a00097a0 con 0x559e855585c0 2024-04-27T00:09:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.230+0000 7f139e600640 1 -- 172.21.15.40:0/1399335034 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a001e040 con 0x559e855585c0 2024-04-27T00:09:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.230+0000 7f13bb5eb740 1 -- 172.21.15.40:0/1399335034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e855585c0 msgr2=0x559e85562350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.230+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/1399335034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e855585c0 0x559e85562350 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009be0 tx=0x7f13a0012f90 comp rx=0 tx=0).stop 2024-04-27T00:09:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.230+0000 7f13bb5eb740 1 -- 172.21.15.40:0/1399335034 shutdown_connections 2024-04-27T00:09:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.230+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/1399335034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e855692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.230+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/1399335034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e855585c0 0x559e85562350 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.230+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/1399335034 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e85557bd0 0x559e85561e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.230+0000 7f13bb5eb740 1 -- 172.21.15.40:0/1399335034 >> 172.21.15.40:0/1399335034 conn(0x559e85553240 msgr2=0x559e8555ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.231+0000 7f13bb5eb740 1 -- 172.21.15.40:0/1399335034 shutdown_connections 2024-04-27T00:09:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.231+0000 7f13bb5eb740 1 -- 172.21.15.40:0/1399335034 wait complete. 2024-04-27T00:09:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.231+0000 7f13bb5eb740 1 -- start start 2024-04-27T00:09:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.231+0000 7f13bb5eb740 1 Processor -- start 2024-04-27T00:09:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13bb5eb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e85557bd0 0x559e85566310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13bb5eb740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 0x559e85566850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13bb5eb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e855d1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13bb5eb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01abd40 con 0x559e8555d730 2024-04-27T00:09:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13bb5eb740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01ad1b0 con 0x559e855585c0 2024-04-27T00:09:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13bb5eb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01ac410 con 0x559e85557bd0 2024-04-27T00:09:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13bb5eb740 10 client.0 _ll_register_callbacks cb 0x7f13a001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:09:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13b6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e855d1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13b5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 0x559e85566850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13b5e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e85557bd0 0x559e85566310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13b5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 0x559e85566850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:47164/0 (socket says 172.21.15.40:47164) 2024-04-27T00:09:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13b5400640 1 -- 172.21.15.40:0/862211655 learned_addr learned my addr 172.21.15.40:0/862211655 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13b5400640 1 -- 172.21.15.40:0/862211655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e85557bd0 msgr2=0x559e85566310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13b5400640 1 --2- 172.21.15.40:0/862211655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e85557bd0 0x559e85566310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13b5400640 1 -- 172.21.15.40:0/862211655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 msgr2=0x559e855d1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13b5400640 1 --2- 172.21.15.40:0/862211655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e855d1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.232+0000 7f13b5400640 1 -- 172.21.15.40:0/862211655 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e85476cb0 con 0x559e855585c0 2024-04-27T00:09:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.233+0000 7f13b5400640 1 --2- 172.21.15.40:0/862211655 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 0x559e85566850 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f13a0005890 tx=0x7f13a00096f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dbc82d5af71c46d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.233+0000 7f1393400640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:09:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.233+0000 7f1393400640 1 -- 172.21.15.40:0/862211655 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a001e070 con 0x559e855585c0 2024-04-27T00:09:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.233+0000 7f13bb5eb740 1 -- 172.21.15.40:0/862211655 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a00097a0 con 0x559e855585c0 2024-04-27T00:09:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.233+0000 7f1393400640 1 -- 172.21.15.40:0/862211655 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f13a0019aa0 con 0x559e855585c0 2024-04-27T00:09:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.233+0000 7f1393400640 1 -- 172.21.15.40:0/862211655 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f13a0047050 con 0x559e855585c0 2024-04-27T00:09:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.233+0000 7f13bb5eb740 10 client.26018 fetch_fsmap learned FSMap version 24 2024-04-27T00:09:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.233+0000 7f13bb5eb740 1 -- 172.21.15.40:0/862211655 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559e855d3290 con 0x559e855585c0 2024-04-27T00:09:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.234+0000 7f1393400640 1 -- 172.21.15.40:0/862211655 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a00097a0 con 0x559e855585c0 2024-04-27T00:09:44.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.234+0000 7f1393400640 1 -- 172.21.15.40:0/862211655 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f13a00035f0 con 0x559e855585c0 2024-04-27T00:09:44.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.234+0000 7f13bb5eb740 10 client.26018 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:09:44.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.234+0000 7f13bb5eb740 10 client.26018 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:09:44.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.234+0000 7f13bb5eb740 20 client.26018 populate_metadata read hostname 'smithi040' 2024-04-27T00:09:44.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.234+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/862211655 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559e855d3c00 0x559e855d6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:44.247 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.234+0000 7f13bb5eb740 4 client.26018 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:09:44.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.234+0000 7f13bb5eb740 1 -- 172.21.15.40:0/862211655 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559e854c3e00 con 0x559e855d3c00 2024-04-27T00:09:44.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.234+0000 7f13b5e00640 1 --2- 172.21.15.40:0/862211655 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559e855d3c00 0x559e855d6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:44.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.235+0000 7f13b5e00640 1 --2- 172.21.15.40:0/862211655 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559e855d3c00 0x559e855d6020 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=252a8d4e849cad9d server_cookie=6c8d4aa72e0cc0dd in_seq=0 out_seq=0 2024-04-27T00:09:44.248 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.235+0000 7f1393400640 10 client.26018 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:44.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f1393400640 1 -- 172.21.15.40:0/862211655 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559e854c3e00 con 0x559e855d3c00 2024-04-27T00:09:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f1393400640 10 client.26018 handle_command_reply: tid=0 2024-04-27T00:09:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f13bb5eb740 1 client.26018 shutdown 2024-04-27T00:09:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f13bb5eb740 1 -- 172.21.15.40:0/862211655 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 msgr2=0x559e85566850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/862211655 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 0x559e85566850 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f13a0005890 tx=0x7f13a00096f0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f13bb5eb740 1 -- 172.21.15.40:0/862211655 shutdown_connections 2024-04-27T00:09:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/862211655 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559e855d3c00 0x559e855d6020 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/862211655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e8555d730 0x559e855d1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/862211655 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559e855585c0 0x559e85566850 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f13bb5eb740 1 --2- 172.21.15.40:0/862211655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e85557bd0 0x559e85566310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f13bb5eb740 1 -- 172.21.15.40:0/862211655 >> 172.21.15.40:0/862211655 conn(0x559e85553240 msgr2=0x559e855597a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f1393400640 0 client.26018 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f13bb5eb740 1 -- 172.21.15.40:0/862211655 shutdown_connections 2024-04-27T00:09:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f13bb5eb740 1 -- 172.21.15.40:0/862211655 wait complete. 2024-04-27T00:09:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.250+0000 7f13bb5eb740 20 client.26018 trim_cache size 0 max 16384 2024-04-27T00:09:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:09:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:09:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:09:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:09:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.251+0000 7f13b8800640 1 -- 172.21.15.40:0/2592533754 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f13980a3cf0 msgr2=0x7f13980a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.251+0000 7f13b8800640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f13980a3cf0 0x7f13980a61b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f13ac004af0 tx=0x7f13ac005da0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.252+0000 7f13b8800640 1 -- 172.21.15.40:0/2592533754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b0102990 msgr2=0x7f13b019acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.252+0000 7f13b8800640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b0102990 0x7f13b019acc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f13a400e9c0 tx=0x7f13a400ee80 comp rx=0 tx=0).stop 2024-04-27T00:09:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.252+0000 7f13b8800640 1 -- 172.21.15.40:0/2592533754 shutdown_connections 2024-04-27T00:09:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.252+0000 7f13b8800640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f13b0104540 0x7f13b01a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.252+0000 7f13b8800640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f13980a3cf0 0x7f13980a61b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.257 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.252+0000 7f13b8800640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f13b0103b80 0x7f13b019b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.252+0000 7f13b8800640 1 --2- 172.21.15.40:0/2592533754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13b0102990 0x7f13b019acc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.252+0000 7f13b8800640 1 -- 172.21.15.40:0/2592533754 >> 172.21.15.40:0/2592533754 conn(0x7f13b00fdf20 msgr2=0x7f13b0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.252+0000 7f13b8800640 1 -- 172.21.15.40:0/2592533754 shutdown_connections 2024-04-27T00:09:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:44.252+0000 7f13b8800640 1 -- 172.21.15.40:0/2592533754 wait complete. 2024-04-27T00:09:44.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:44 smithi040 ceph-mon[28095]: pgmap v653: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:09:44.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:44 smithi080 ceph-mon[32362]: pgmap v653: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:09:44.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:44 smithi103 ceph-mon[32952]: pgmap v653: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:09:46.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:46 smithi040 ceph-mon[28095]: pgmap v654: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:09:46.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:46 smithi080 ceph-mon[32362]: pgmap v654: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:09:46.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:46 smithi103 ceph-mon[32952]: pgmap v654: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:09:48.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:48 smithi040 ceph-mon[28095]: pgmap v655: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:09:48.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:48 smithi080 ceph-mon[32362]: pgmap v655: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:09:48.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:48 smithi103 ceph-mon[32952]: pgmap v655: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:09:50.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:50 smithi040 ceph-mon[28095]: pgmap v656: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:50.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:50 smithi080 ceph-mon[32362]: pgmap v656: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:50.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:50 smithi103 ceph-mon[32952]: pgmap v656: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:51.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:51 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:09:51.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:51 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:09:51.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:51 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:09:52.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:52 smithi040 ceph-mon[28095]: pgmap v657: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:09:52.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:09:52.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:09:52.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:09:52.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:52 smithi080 ceph-mon[32362]: pgmap v657: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:09:52.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:09:52.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:09:52.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:09:52.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:52 smithi103 ceph-mon[32952]: pgmap v657: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:09:52.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:09:52.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:09:52.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:09:54.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:54 smithi040 ceph-mon[28095]: pgmap v658: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:09:54.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:54 smithi080 ceph-mon[32362]: pgmap v658: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:09:54.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:54 smithi103 ceph-mon[32952]: pgmap v658: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:09:56.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:56 smithi040 ceph-mon[28095]: pgmap v659: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:56.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:56 smithi080 ceph-mon[32362]: pgmap v659: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:56.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:56 smithi103 ceph-mon[32952]: pgmap v659: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:09:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.401+0000 7fbfc3200640 1 -- 172.21.15.40:0/1774135631 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfbc1049f0 msgr2=0x7fbfbc104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.401+0000 7fbfc3200640 1 --2- 172.21.15.40:0/1774135631 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfbc1049f0 0x7fbfbc104e10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009860 tx=0x7fbfb002a340 comp rx=0 tx=0).stop 2024-04-27T00:09:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.401+0000 7fbfc3200640 1 -- 172.21.15.40:0/1774135631 shutdown_connections 2024-04-27T00:09:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.401+0000 7fbfc3200640 1 --2- 172.21.15.40:0/1774135631 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfbc1065a0 0x7fbfbc10ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.401+0000 7fbfc3200640 1 --2- 172.21.15.40:0/1774135631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfbc105be0 0x7fbfbc106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.401+0000 7fbfc3200640 1 --2- 172.21.15.40:0/1774135631 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfbc1049f0 0x7fbfbc104e10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.401+0000 7fbfc3200640 1 -- 172.21.15.40:0/1774135631 >> 172.21.15.40:0/1774135631 conn(0x7fbfbc0fffa0 msgr2=0x7fbfbc1023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.401+0000 7fbfc3200640 1 -- 172.21.15.40:0/1774135631 shutdown_connections 2024-04-27T00:09:57.403 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.401+0000 7fbfc3200640 1 -- 172.21.15.40:0/1774135631 wait complete. 2024-04-27T00:09:57.404 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.402+0000 7fbfc3200640 1 Processor -- start 2024-04-27T00:09:57.404 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc3200640 1 -- start start 2024-04-27T00:09:57.404 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc3200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfbc1049f0 0x7fbfbc19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfbc105be0 0x7fbfbc19b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfbc1065a0 0x7fbfbc1a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc110520 con 0x7fbfbc105be0 2024-04-27T00:09:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc3200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc1103a0 con 0x7fbfbc1049f0 2024-04-27T00:09:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc1106a0 con 0x7fbfbc1065a0 2024-04-27T00:09:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfbc105be0 0x7fbfbc19b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc1e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfbc1049f0 0x7fbfbc19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc1e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfbc1049f0 0x7fbfbc19abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:60494/0 (socket says 172.21.15.40:60494) 2024-04-27T00:09:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfbc105be0 0x7fbfbc19b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50508/0 (socket says 172.21.15.40:50508) 2024-04-27T00:09:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc1e00640 1 -- 172.21.15.40:0/132985159 learned_addr learned my addr 172.21.15.40:0/132985159 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.403+0000 7fbfc2800640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfbc1065a0 0x7fbfbc1a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.404+0000 7fbfc1e00640 1 -- 172.21.15.40:0/132985159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfbc1065a0 msgr2=0x7fbfbc1a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.404+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfbc1065a0 0x7fbfbc1a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.404+0000 7fbfc1e00640 1 -- 172.21.15.40:0/132985159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfbc105be0 msgr2=0x7fbfbc19b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.404+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfbc105be0 0x7fbfbc19b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.404+0000 7fbfc1e00640 1 -- 172.21.15.40:0/132985159 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfbc1a2810 con 0x7fbfbc1049f0 2024-04-27T00:09:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.404+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfbc1049f0 0x7fbfbc19abf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00057c0 tx=0x7fbfb0032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.404+0000 7fbfb6a00640 1 -- 172.21.15.40:0/132985159 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0037040 con 0x7fbfbc1049f0 2024-04-27T00:09:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.404+0000 7fbfb6a00640 1 -- 172.21.15.40:0/132985159 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb0030bd0 con 0x7fbfbc1049f0 2024-04-27T00:09:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.404+0000 7fbfb6a00640 1 -- 172.21.15.40:0/132985159 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0005280 con 0x7fbfbc1049f0 2024-04-27T00:09:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.405+0000 7fbfc3200640 1 -- 172.21.15.40:0/132985159 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfbc1a2aa0 con 0x7fbfbc1049f0 2024-04-27T00:09:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.405+0000 7fbfc3200640 1 -- 172.21.15.40:0/132985159 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfbc1a2ee0 con 0x7fbfbc1049f0 2024-04-27T00:09:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.408+0000 7fbfc776f740 1 Processor -- start 2024-04-27T00:09:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.408+0000 7fbfc776f740 1 -- start start 2024-04-27T00:09:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.408+0000 7fbfc776f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd6de0 0x55dafafd7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.408+0000 7fbfc776f740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd77d0 0x55dafafdc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.408+0000 7fbfc776f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafdc940 0x55dafafdedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.408+0000 7fbfc776f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dafaf3b380 con 0x55dafafd6de0 2024-04-27T00:09:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.408+0000 7fbfc776f740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55dafaed7c30 con 0x55dafafd77d0 2024-04-27T00:09:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.408+0000 7fbfc776f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dafaf37340 con 0x55dafafdc940 2024-04-27T00:09:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.409+0000 7fbfc2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd6de0 0x55dafafd7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.409+0000 7fbfc1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafdc940 0x55dafafdedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.409+0000 7fbfc1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafdc940 0x55dafafdedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:59776/0 (socket says 172.21.15.40:59776) 2024-04-27T00:09:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.409+0000 7fbfc1400640 1 -- 172.21.15.40:0/230439832 learned_addr learned my addr 172.21.15.40:0/230439832 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.409+0000 7fbfb6a00640 1 -- 172.21.15.40:0/132985159 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbfb0043020 con 0x7fbfbc1049f0 2024-04-27T00:09:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.409+0000 7fbfc1400640 1 -- 172.21.15.40:0/230439832 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd77d0 msgr2=0x55dafafdc2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-27T00:09:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.409+0000 7fbfc1400640 1 --2- 172.21.15.40:0/230439832 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd77d0 0x55dafafdc2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.409+0000 7fbfc1400640 1 -- 172.21.15.40:0/230439832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd6de0 msgr2=0x55dafafd7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.409+0000 7fbfc1400640 1 --2- 172.21.15.40:0/230439832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd6de0 0x55dafafd7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.409+0000 7fbfc1400640 1 -- 172.21.15.40:0/230439832 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dafaef9940 con 0x55dafafdc940 2024-04-27T00:09:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.409+0000 7fbfc1400640 1 --2- 172.21.15.40:0/230439832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafdc940 0x55dafafdedd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009c80 tx=0x7fbfac011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f325e958af7e414 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.410+0000 7fbfab000640 1 -- 172.21.15.40:0/230439832 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac004220 con 0x55dafafdc940 2024-04-27T00:09:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.410+0000 7fbfab000640 1 -- 172.21.15.40:0/230439832 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfac009e30 con 0x55dafafdc940 2024-04-27T00:09:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.410+0000 7fbfab000640 1 -- 172.21.15.40:0/230439832 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac0197b0 con 0x55dafafdc940 2024-04-27T00:09:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.410+0000 7fbfc776f740 1 -- 172.21.15.40:0/230439832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafdc940 msgr2=0x55dafafdedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.410+0000 7fbfc776f740 1 --2- 172.21.15.40:0/230439832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafdc940 0x55dafafdedd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009c80 tx=0x7fbfac011810 comp rx=0 tx=0).stop 2024-04-27T00:09:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.410+0000 7fbfc776f740 1 -- 172.21.15.40:0/230439832 shutdown_connections 2024-04-27T00:09:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.410+0000 7fbfc776f740 1 --2- 172.21.15.40:0/230439832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafdc940 0x55dafafdedd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.410+0000 7fbfc776f740 1 --2- 172.21.15.40:0/230439832 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd77d0 0x55dafafdc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.410+0000 7fbfc776f740 1 --2- 172.21.15.40:0/230439832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd6de0 0x55dafafd7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.410+0000 7fbfc776f740 1 -- 172.21.15.40:0/230439832 >> 172.21.15.40:0/230439832 conn(0x55dafafd2450 msgr2=0x55dafafd2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.411+0000 7fbfb6a00640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbfa40a3cb0 0x7fbfa40a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.411+0000 7fbfc776f740 1 -- 172.21.15.40:0/230439832 shutdown_connections 2024-04-27T00:09:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.411+0000 7fbfc776f740 1 -- 172.21.15.40:0/230439832 wait complete. 2024-04-27T00:09:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.411+0000 7fbfb6a00640 1 -- 172.21.15.40:0/132985159 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fbfb00efde0 con 0x7fbfbc1049f0 2024-04-27T00:09:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.411+0000 7fbfc2800640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbfa40a3cb0 0x7fbfa40a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.411+0000 7fbfc776f740 1 -- start start 2024-04-27T00:09:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.411+0000 7fbfc776f740 1 Processor -- start 2024-04-27T00:09:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc2800640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbfa40a3cb0 0x7fbfa40a6170 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55dafafd8030 tx=0x7fbfb8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc776f740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd6de0 0x55dafafe10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc776f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd77d0 0x55dafafe1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc776f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafdc940 0x55dafafe85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc776f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dafafdf8f0 con 0x55dafafd77d0 2024-04-27T00:09:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc776f740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55dafaec2c60 con 0x55dafafd6de0 2024-04-27T00:09:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc776f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dafafdfa70 con 0x55dafafdc940 2024-04-27T00:09:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc776f740 10 client.0 _ll_register_callbacks cb 0x55dafaeeee50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:09:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc1e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd6de0 0x55dafafe10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc1e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd6de0 0x55dafafe10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:60518/0 (socket says 172.21.15.40:60518) 2024-04-27T00:09:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc1e00640 1 -- 172.21.15.40:0/1328927128 learned_addr learned my addr 172.21.15.40:0/1328927128 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc1400640 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd77d0 0x55dafafe1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.412+0000 7fbfc2800640 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafdc940 0x55dafafe85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.413+0000 7fbfc1e00640 1 -- 172.21.15.40:0/1328927128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafdc940 msgr2=0x55dafafe85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.413+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafdc940 0x55dafafe85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.413+0000 7fbfc1e00640 1 -- 172.21.15.40:0/1328927128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd77d0 msgr2=0x55dafafe1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.413+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd77d0 0x55dafafe1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.413+0000 7fbfc1e00640 1 -- 172.21.15.40:0/1328927128 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dafaed1510 con 0x55dafafd6de0 2024-04-27T00:09:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.413+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd6de0 0x55dafafe10e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00ed4d0 tx=0x7fbfb00bb5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25a4f271c9bed338 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.413+0000 7fbf9f400640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:09:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.413+0000 7fbf9f400640 1 -- 172.21.15.40:0/1328927128 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00c0020 con 0x55dafafd6de0 2024-04-27T00:09:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.413+0000 7fbfc776f740 1 -- 172.21.15.40:0/1328927128 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfac009e30 con 0x55dafafd6de0 2024-04-27T00:09:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.413+0000 7fbf9f400640 1 -- 172.21.15.40:0/1328927128 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fbfb0005420 con 0x55dafafd6de0 2024-04-27T00:09:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.414+0000 7fbf9f400640 1 -- 172.21.15.40:0/1328927128 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbfb00c48e0 con 0x55dafafd6de0 2024-04-27T00:09:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.414+0000 7fbfc776f740 10 client.26027 fetch_fsmap learned FSMap version 24 2024-04-27T00:09:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.414+0000 7fbfc776f740 1 -- 172.21.15.40:0/1328927128 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dafac4d210 con 0x55dafafd6de0 2024-04-27T00:09:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.414+0000 7fbf9f400640 1 -- 172.21.15.40:0/1328927128 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfac009e30 con 0x55dafafd6de0 2024-04-27T00:09:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.414+0000 7fbf9f400640 1 -- 172.21.15.40:0/1328927128 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fbfb00dc100 con 0x55dafafd6de0 2024-04-27T00:09:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.414+0000 7fbfc776f740 10 client.26027 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:09:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.414+0000 7fbfc776f740 10 client.26027 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:09:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.414+0000 7fbfc776f740 20 client.26027 populate_metadata read hostname 'smithi040' 2024-04-27T00:09:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.414+0000 7fbfc776f740 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55dafafec8a0 0x55dafafeecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.420 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.414+0000 7fbfc776f740 4 client.26027 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:09:57.420 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.414+0000 7fbfc776f740 1 -- 172.21.15.40:0/1328927128 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dafaf48f90 con 0x55dafafec8a0 2024-04-27T00:09:57.420 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.415+0000 7fbfc1400640 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55dafafec8a0 0x55dafafeecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.415+0000 7fbfc1400640 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55dafafec8a0 0x55dafafeecf0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfb6a69ae2104ba5 server_cookie=331e68425624e275 in_seq=0 out_seq=0 2024-04-27T00:09:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.415+0000 7fbf9f400640 10 client.26027 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.416+0000 7fbf9f400640 1 -- 172.21.15.40:0/1328927128 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dafaf48f90 con 0x55dafafec8a0 2024-04-27T00:09:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.416+0000 7fbf9f400640 10 client.26027 handle_command_reply: tid=0 2024-04-27T00:09:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.416+0000 7fbfc776f740 1 client.26027 shutdown 2024-04-27T00:09:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.416+0000 7fbfc776f740 1 -- 172.21.15.40:0/1328927128 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd6de0 msgr2=0x55dafafe10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.416+0000 7fbfc776f740 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd6de0 0x55dafafe10e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00ed4d0 tx=0x7fbfb00bb5d0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.416+0000 7fbfc776f740 1 -- 172.21.15.40:0/1328927128 shutdown_connections 2024-04-27T00:09:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.416+0000 7fbfc776f740 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55dafafec8a0 0x55dafafeecf0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.416+0000 7fbfc776f740 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafdc940 0x55dafafe85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.416+0000 7fbfc776f740 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd77d0 0x55dafafe1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.416+0000 7fbfc776f740 1 --2- 172.21.15.40:0/1328927128 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd6de0 0x55dafafe10e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.416+0000 7fbfc776f740 1 -- 172.21.15.40:0/1328927128 >> 172.21.15.40:0/1328927128 conn(0x55dafafd2450 msgr2=0x55dafafd1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.416+0000 7fbf9f400640 0 client.26027 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.417+0000 7fbfc776f740 1 -- 172.21.15.40:0/1328927128 shutdown_connections 2024-04-27T00:09:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.417+0000 7fbfc776f740 1 -- 172.21.15.40:0/1328927128 wait complete. 2024-04-27T00:09:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.417+0000 7fbfc776f740 20 client.26027 trim_cache size 0 max 16384 2024-04-27T00:09:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.433+0000 7fbfc776f740 1 Processor -- start 2024-04-27T00:09:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.433+0000 7fbfc776f740 1 -- start start 2024-04-27T00:09:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.433+0000 7fbfc776f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd2450 0x55dafafd5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.433+0000 7fbfc776f740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd6de0 0x55dafafe8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.433+0000 7fbfc776f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafd77d0 0x55dafafe0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.433+0000 7fbfc776f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc1afd40 con 0x55dafafd2450 2024-04-27T00:09:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.433+0000 7fbfc776f740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc1ad610 con 0x55dafafd6de0 2024-04-27T00:09:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.433+0000 7fbfc776f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc1ab460 con 0x55dafafd77d0 2024-04-27T00:09:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.434+0000 7fbfc1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd2450 0x55dafafd5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.434+0000 7fbfc1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd2450 0x55dafafd5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50554/0 (socket says 172.21.15.40:50554) 2024-04-27T00:09:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.434+0000 7fbfc1e00640 1 -- 172.21.15.40:0/2036101300 learned_addr learned my addr 172.21.15.40:0/2036101300 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.434+0000 7fbfc2800640 1 --2- 172.21.15.40:0/2036101300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafd77d0 0x55dafafe0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.434+0000 7fbfc1400640 1 --2- 172.21.15.40:0/2036101300 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd6de0 0x55dafafe8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.434+0000 7fbfc1e00640 1 -- 172.21.15.40:0/2036101300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafd77d0 msgr2=0x55dafafe0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:57.436 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.434+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/2036101300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafd77d0 0x55dafafe0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.434+0000 7fbfc1e00640 1 -- 172.21.15.40:0/2036101300 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd6de0 msgr2=0x55dafafe8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.434+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/2036101300 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd6de0 0x55dafafe8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.434+0000 7fbfc1e00640 1 -- 172.21.15.40:0/2036101300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dafad19620 con 0x55dafafd2450 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.434+0000 7fbfc2800640 1 --2- 172.21.15.40:0/2036101300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafd77d0 0x55dafafe0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.434+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/2036101300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd2450 0x55dafafd5bf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00bbc00 tx=0x7fbfb00bbdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc4003896c53d4a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.435+0000 7fbfaa600640 1 -- 172.21.15.40:0/2036101300 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00c0020 con 0x55dafafd2450 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.435+0000 7fbfaa600640 1 -- 172.21.15.40:0/2036101300 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb0005420 con 0x55dafafd2450 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.435+0000 7fbfaa600640 1 -- 172.21.15.40:0/2036101300 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00d5080 con 0x55dafafd2450 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.435+0000 7fbfc776f740 1 -- 172.21.15.40:0/2036101300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd2450 msgr2=0x55dafafd5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.435+0000 7fbfc776f740 1 --2- 172.21.15.40:0/2036101300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd2450 0x55dafafd5bf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00bbc00 tx=0x7fbfb00bbdf0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.435+0000 7fbfc776f740 1 -- 172.21.15.40:0/2036101300 shutdown_connections 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.435+0000 7fbfc776f740 1 --2- 172.21.15.40:0/2036101300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafd77d0 0x55dafafe0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.437 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.435+0000 7fbfc776f740 1 --2- 172.21.15.40:0/2036101300 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd6de0 0x55dafafe8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.435+0000 7fbfc776f740 1 --2- 172.21.15.40:0/2036101300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd2450 0x55dafafd5bf0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.435+0000 7fbfc776f740 1 -- 172.21.15.40:0/2036101300 >> 172.21.15.40:0/2036101300 conn(0x55dafafdc440 msgr2=0x55dafafd54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:57.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.435+0000 7fbfc776f740 1 -- 172.21.15.40:0/2036101300 shutdown_connections 2024-04-27T00:09:57.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.435+0000 7fbfc776f740 1 -- 172.21.15.40:0/2036101300 wait complete. 2024-04-27T00:09:57.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.436+0000 7fbfc776f740 1 -- start start 2024-04-27T00:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.436+0000 7fbfc776f740 1 Processor -- start 2024-04-27T00:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.436+0000 7fbfc776f740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd2450 0x55dafafe5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.436+0000 7fbfc776f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafd6de0 0x55dafafe5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.436+0000 7fbfc776f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd77d0 0x55dafb050eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.436+0000 7fbfc776f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc1abad0 con 0x55dafafd77d0 2024-04-27T00:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.436+0000 7fbfc776f740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc1acf40 con 0x55dafafd2450 2024-04-27T00:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.436+0000 7fbfc776f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc1ac1a0 con 0x55dafafd6de0 2024-04-27T00:09:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.437+0000 7fbfc776f740 10 client.0 _ll_register_callbacks cb 0x7fbfb00097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.437+0000 7fbfc2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd77d0 0x55dafb050eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.437+0000 7fbfc2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd77d0 0x55dafb050eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50558/0 (socket says 172.21.15.40:50558) 2024-04-27T00:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.437+0000 7fbfc2800640 1 -- 172.21.15.40:0/3310152864 learned_addr learned my addr 172.21.15.40:0/3310152864 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.437+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd2450 0x55dafafe5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.437+0000 7fbfc1400640 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafd6de0 0x55dafafe5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.437+0000 7fbfc2800640 1 -- 172.21.15.40:0/3310152864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafd6de0 msgr2=0x55dafafe5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.437+0000 7fbfc2800640 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafd6de0 0x55dafafe5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.437+0000 7fbfc2800640 1 -- 172.21.15.40:0/3310152864 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd2450 msgr2=0x55dafafe5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.437+0000 7fbfc2800640 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd2450 0x55dafafe5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.437+0000 7fbfc2800640 1 -- 172.21.15.40:0/3310152864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dafaef5cb0 con 0x55dafafd77d0 2024-04-27T00:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.437+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd2450 0x55dafafe5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:09:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.438+0000 7fbfc2800640 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd77d0 0x55dafb050eb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80042e0 tx=0x7fbfb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366b9da14cc5f04e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.438+0000 7fbf9f400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.438+0000 7fbf9f400640 1 -- 172.21.15.40:0/3310152864 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb802d060 con 0x55dafafd77d0 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.438+0000 7fbfc776f740 1 -- 172.21.15.40:0/3310152864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfb0005420 con 0x55dafafd77d0 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.438+0000 7fbf9f400640 1 -- 172.21.15.40:0/3310152864 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fbfb800fc10 con 0x55dafafd77d0 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.438+0000 7fbf9f400640 1 -- 172.21.15.40:0/3310152864 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbfb8034050 con 0x55dafafd77d0 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.438+0000 7fbfc776f740 10 client.15921 fetch_fsmap learned FSMap version 24 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.438+0000 7fbfc776f740 1 -- 172.21.15.40:0/3310152864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dafb052700 con 0x55dafafd77d0 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.438+0000 7fbf9f400640 1 -- 172.21.15.40:0/3310152864 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb0005420 con 0x55dafafd77d0 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.439+0000 7fbf9f400640 1 -- 172.21.15.40:0/3310152864 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fbfb801e400 con 0x55dafafd77d0 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.439+0000 7fbfc776f740 10 client.15921 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.439+0000 7fbfc776f740 10 client.15921 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.439+0000 7fbfc776f740 20 client.15921 populate_metadata read hostname 'smithi040' 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.439+0000 7fbfc776f740 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55dafafdc940 0x55dafb0550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:09:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.439+0000 7fbfc776f740 4 client.15921 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:09:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.439+0000 7fbfc776f740 1 -- 172.21.15.40:0/3310152864 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dafaf42e00 con 0x55dafafdc940 2024-04-27T00:09:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.439+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55dafafdc940 0x55dafb0550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:09:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.439+0000 7fbfc1e00640 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55dafafdc940 0x55dafb0550c0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=709aec8d53d68527 server_cookie=c1b463e4cd4624bb in_seq=0 out_seq=0 2024-04-27T00:09:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.440+0000 7fbf9f400640 10 client.15921 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbf9f400640 1 -- 172.21.15.40:0/3310152864 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dafaf42e00 con 0x55dafafdc940 2024-04-27T00:09:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbf9f400640 10 client.15921 handle_command_reply: tid=0 2024-04-27T00:09:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbfc776f740 1 client.15921 shutdown 2024-04-27T00:09:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbfc776f740 1 -- 172.21.15.40:0/3310152864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd77d0 msgr2=0x55dafb050eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbfc776f740 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd77d0 0x55dafb050eb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80042e0 tx=0x7fbfb8004a40 comp rx=0 tx=0).stop 2024-04-27T00:09:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbfc776f740 1 -- 172.21.15.40:0/3310152864 shutdown_connections 2024-04-27T00:09:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbfc776f740 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55dafafdc940 0x55dafb0550c0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbfc776f740 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dafafd77d0 0x55dafb050eb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbfc776f740 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dafafd6de0 0x55dafafe5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbfc776f740 1 --2- 172.21.15.40:0/3310152864 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55dafafd2450 0x55dafafe5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbfc776f740 1 -- 172.21.15.40:0/3310152864 >> 172.21.15.40:0/3310152864 conn(0x55dafafdc440 msgr2=0x55dafafe0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbf9f400640 0 client.15921 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:09:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbfc776f740 1 -- 172.21.15.40:0/3310152864 shutdown_connections 2024-04-27T00:09:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbfc776f740 1 -- 172.21.15.40:0/3310152864 wait complete. 2024-04-27T00:09:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.455+0000 7fbfc776f740 20 client.15921 trim_cache size 0 max 16384 2024-04-27T00:09:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:09:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:09:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:09:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:09:57.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.456+0000 7fbfc3200640 1 -- 172.21.15.40:0/132985159 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbfa40a3cb0 msgr2=0x7fbfa40a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.456+0000 7fbfc3200640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbfa40a3cb0 0x7fbfa40a6170 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55dafafd8030 tx=0x7fbfb8009500 comp rx=0 tx=0).stop 2024-04-27T00:09:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.456+0000 7fbfc3200640 1 -- 172.21.15.40:0/132985159 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfbc1049f0 msgr2=0x7fbfbc19abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:09:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.456+0000 7fbfc3200640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfbc1049f0 0x7fbfbc19abf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00057c0 tx=0x7fbfb0032da0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.457+0000 7fbfc3200640 1 -- 172.21.15.40:0/132985159 shutdown_connections 2024-04-27T00:09:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.457+0000 7fbfc3200640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fbfa40a3cb0 0x7fbfa40a6170 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.457+0000 7fbfc3200640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfbc1065a0 0x7fbfbc1a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.457+0000 7fbfc3200640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfbc105be0 0x7fbfbc19b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.457+0000 7fbfc3200640 1 --2- 172.21.15.40:0/132985159 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbfbc1049f0 0x7fbfbc19abf0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:09:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.457+0000 7fbfc3200640 1 -- 172.21.15.40:0/132985159 >> 172.21.15.40:0/132985159 conn(0x7fbfbc0fffa0 msgr2=0x7fbfbc108fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:09:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.457+0000 7fbfc3200640 1 -- 172.21.15.40:0/132985159 shutdown_connections 2024-04-27T00:09:57.461 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:09:57.457+0000 7fbfc3200640 1 -- 172.21.15.40:0/132985159 wait complete. 2024-04-27T00:09:58.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:09:57 smithi040 ceph-mon[28095]: pgmap v660: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:09:58.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:09:57 smithi080 ceph-mon[32362]: pgmap v660: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:09:58.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:09:57 smithi103 ceph-mon[32952]: pgmap v660: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:10:00.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:00 smithi103 ceph-mon[32952]: pgmap v661: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:10:00.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:00 smithi103 ceph-mon[32952]: overall HEALTH_OK 2024-04-27T00:10:00.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:00 smithi040 ceph-mon[28095]: pgmap v661: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:10:00.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:00 smithi040 ceph-mon[28095]: overall HEALTH_OK 2024-04-27T00:10:00.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:00 smithi080 ceph-mon[32362]: pgmap v661: 129 pgs: 129 active+clean; 1.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:10:00.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:00 smithi080 ceph-mon[32362]: overall HEALTH_OK 2024-04-27T00:10:02.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:02 smithi040 ceph-mon[28095]: pgmap v662: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:10:02.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:02 smithi080 ceph-mon[32362]: pgmap v662: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:10:02.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:02 smithi103 ceph-mon[32952]: pgmap v662: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:10:04.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:04 smithi040 ceph-mon[28095]: pgmap v663: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:10:04.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:04 smithi080 ceph-mon[32362]: pgmap v663: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:10:04.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:04 smithi103 ceph-mon[32952]: pgmap v663: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:10:06.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:06 smithi040 ceph-mon[28095]: pgmap v664: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:10:06.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:06 smithi080 ceph-mon[32362]: pgmap v664: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:10:06.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:06 smithi103 ceph-mon[32952]: pgmap v664: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:10:08.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:08 smithi040 ceph-mon[28095]: pgmap v665: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:10:08.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:08 smithi080 ceph-mon[32362]: pgmap v665: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:10:08.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:08 smithi103 ceph-mon[32952]: pgmap v665: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:10:10.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:10 smithi040 ceph-mon[28095]: pgmap v666: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:10.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:10 smithi080 ceph-mon[32362]: pgmap v666: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:10.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:10 smithi103 ceph-mon[32952]: pgmap v666: 129 pgs: 129 active+clean; 1.6 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.608+0000 7fa916e00640 1 -- 172.21.15.40:0/2114170824 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa910075310 msgr2=0x7fa910075770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.608+0000 7fa916e00640 1 --2- 172.21.15.40:0/2114170824 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa910075310 0x7fa910075770 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009900 tx=0x7fa8fc02a480 comp rx=0 tx=0).stop 2024-04-27T00:10:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.608+0000 7fa916e00640 1 -- 172.21.15.40:0/2114170824 shutdown_connections 2024-04-27T00:10:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.608+0000 7fa916e00640 1 --2- 172.21.15.40:0/2114170824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa91010a6a0 0x7fa91010cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.608+0000 7fa916e00640 1 --2- 172.21.15.40:0/2114170824 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa910075310 0x7fa910075770 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.608+0000 7fa916e00640 1 --2- 172.21.15.40:0/2114170824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9100768f0 0x7fa910074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.608+0000 7fa916e00640 1 -- 172.21.15.40:0/2114170824 >> 172.21.15.40:0/2114170824 conn(0x7fa9100fdd60 msgr2=0x7fa9101001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.609+0000 7fa916e00640 1 -- 172.21.15.40:0/2114170824 shutdown_connections 2024-04-27T00:10:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.609+0000 7fa916e00640 1 -- 172.21.15.40:0/2114170824 wait complete. 2024-04-27T00:10:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa916e00640 1 Processor -- start 2024-04-27T00:10:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa916e00640 1 -- start start 2024-04-27T00:10:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa916e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa910075310 0x7fa91019ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa916e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9100768f0 0x7fa91019b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa916e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa91010a6a0 0x7fa9101a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa916e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa910110550 con 0x7fa9100768f0 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa916e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa9101103d0 con 0x7fa91010a6a0 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa916e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9101106d0 con 0x7fa910075310 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa915000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9100768f0 0x7fa91019b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa916400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa91010a6a0 0x7fa9101a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa916400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa91010a6a0 0x7fa9101a21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:38678/0 (socket says 172.21.15.40:38678) 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa916400640 1 -- 172.21.15.40:0/284859934 learned_addr learned my addr 172.21.15.40:0/284859934 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.610+0000 7fa915a00640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa910075310 0x7fa91019ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.611+0000 7fa916400640 1 -- 172.21.15.40:0/284859934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa910075310 msgr2=0x7fa91019ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.611+0000 7fa916400640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa910075310 0x7fa91019ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.611+0000 7fa916400640 1 -- 172.21.15.40:0/284859934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9100768f0 msgr2=0x7fa91019b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.611+0000 7fa916400640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9100768f0 0x7fa91019b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.611+0000 7fa916400640 1 -- 172.21.15.40:0/284859934 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9101a2880 con 0x7fa91010a6a0 2024-04-27T00:10:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.611+0000 7fa916400640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa91010a6a0 0x7fa9101a21e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa90800efd0 tx=0x7fa90800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.611+0000 7fa90e000640 1 -- 172.21.15.40:0/284859934 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa908003b70 con 0x7fa91010a6a0 2024-04-27T00:10:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.611+0000 7fa90e000640 1 -- 172.21.15.40:0/284859934 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa908003d10 con 0x7fa91010a6a0 2024-04-27T00:10:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.611+0000 7fa916e00640 1 -- 172.21.15.40:0/284859934 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9101a2b70 con 0x7fa91010a6a0 2024-04-27T00:10:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.611+0000 7fa90e000640 1 -- 172.21.15.40:0/284859934 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9080108a0 con 0x7fa91010a6a0 2024-04-27T00:10:10.615 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.612+0000 7fa916e00640 1 -- 172.21.15.40:0/284859934 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9101a3070 con 0x7fa91010a6a0 2024-04-27T00:10:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.615+0000 7fa91b215740 1 Processor -- start 2024-04-27T00:10:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.615+0000 7fa90e000640 1 -- 172.21.15.40:0/284859934 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa908004020 con 0x7fa91010a6a0 2024-04-27T00:10:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.615+0000 7fa91b215740 1 -- start start 2024-04-27T00:10:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa91b215740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa91b215740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecc52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa91b215740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecc5940 0x55ee4ecc7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa91b215740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ee4ec24380 con 0x55ee4ecc07d0 2024-04-27T00:10:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa91b215740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55ee4ebc0c30 con 0x55ee4ecc5940 2024-04-27T00:10:10.618 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa91b215740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ee4ec20340 con 0x55ee4ecbfde0 2024-04-27T00:10:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa915000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa915a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecc52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa915000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56996/0 (socket says 172.21.15.40:56996) 2024-04-27T00:10:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa915000640 1 -- 172.21.15.40:0/1969711686 learned_addr learned my addr 172.21.15.40:0/1969711686 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa915000640 1 --2- 172.21.15.40:0/1969711686 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecc5940 0x55ee4ecc7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa915000640 1 -- 172.21.15.40:0/1969711686 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecc5940 msgr2=0x55ee4ecc7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.616+0000 7fa915000640 1 --2- 172.21.15.40:0/1969711686 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecc5940 0x55ee4ecc7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.617+0000 7fa915000640 1 -- 172.21.15.40:0/1969711686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 msgr2=0x55ee4ecc52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.617+0000 7fa915000640 1 --2- 172.21.15.40:0/1969711686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecc52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.617+0000 7fa915000640 1 -- 172.21.15.40:0/1969711686 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee4ebe2940 con 0x55ee4ecbfde0 2024-04-27T00:10:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.617+0000 7fa915a00640 1 --2- 172.21.15.40:0/1969711686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecc52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:10:10.619 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.617+0000 7fa915000640 1 --2- 172.21.15.40:0/1969711686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc0200 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc006010 tx=0x7fa8fc005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac5803655b30a9d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.617+0000 7fa90e000640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa8f80a3c30 0x7fa8f80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.617+0000 7fa902a00640 1 -- 172.21.15.40:0/1969711686 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc035040 con 0x55ee4ecbfde0 2024-04-27T00:10:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.617+0000 7fa90e000640 1 -- 172.21.15.40:0/284859934 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fa9080d0df0 con 0x7fa91010a6a0 2024-04-27T00:10:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.617+0000 7fa902a00640 1 -- 172.21.15.40:0/1969711686 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8fc003aa0 con 0x55ee4ecbfde0 2024-04-27T00:10:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.617+0000 7fa902a00640 1 -- 172.21.15.40:0/1969711686 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc0319c0 con 0x55ee4ecbfde0 2024-04-27T00:10:10.620 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.617+0000 7fa915a00640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa8f80a3c30 0x7fa8f80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.618+0000 7fa91b215740 1 -- 172.21.15.40:0/1969711686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecbfde0 msgr2=0x55ee4ecc0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.618+0000 7fa91b215740 1 --2- 172.21.15.40:0/1969711686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc0200 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc006010 tx=0x7fa8fc005d50 comp rx=0 tx=0).stop 2024-04-27T00:10:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.618+0000 7fa91b215740 1 -- 172.21.15.40:0/1969711686 shutdown_connections 2024-04-27T00:10:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.618+0000 7fa91b215740 1 --2- 172.21.15.40:0/1969711686 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecc5940 0x55ee4ecc7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.618+0000 7fa91b215740 1 --2- 172.21.15.40:0/1969711686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecc52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.618+0000 7fa91b215740 1 --2- 172.21.15.40:0/1969711686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc0200 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.618+0000 7fa91b215740 1 -- 172.21.15.40:0/1969711686 >> 172.21.15.40:0/1969711686 conn(0x55ee4ecbb450 msgr2=0x55ee4ecbb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.618+0000 7fa915a00640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa8f80a3c30 0x7fa8f80a60f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x55ee4ecc0bd0 tx=0x7fa904005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.618+0000 7fa91b215740 1 -- 172.21.15.40:0/1969711686 shutdown_connections 2024-04-27T00:10:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.618+0000 7fa91b215740 1 -- 172.21.15.40:0/1969711686 wait complete. 2024-04-27T00:10:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.619+0000 7fa91b215740 1 -- start start 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.619+0000 7fa91b215740 1 Processor -- start 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.619+0000 7fa91b215740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.619+0000 7fa91b215740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecca530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.619+0000 7fa91b215740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ecd1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.619+0000 7fa91b215740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ee4ecc8890 con 0x55ee4ecbfde0 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.619+0000 7fa91b215740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55ee4ebabc60 con 0x55ee4ecc07d0 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.619+0000 7fa91b215740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ee4ecc8a10 con 0x55ee4ecc5940 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.619+0000 7fa91b215740 10 client.0 _ll_register_callbacks cb 0x55ee4ebd7e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.620+0000 7fa915a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.620+0000 7fa915a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36474/0 (socket says 172.21.15.40:36474) 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.620+0000 7fa915a00640 1 -- 172.21.15.40:0/3431077983 learned_addr learned my addr 172.21.15.40:0/3431077983 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.620+0000 7fa916400640 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ecd1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.620+0000 7fa915000640 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.620+0000 7fa915a00640 1 -- 172.21.15.40:0/3431077983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 msgr2=0x55ee4ecd1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.620+0000 7fa915a00640 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ecd1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.620+0000 7fa915a00640 1 -- 172.21.15.40:0/3431077983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecc07d0 msgr2=0x55ee4ecca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.620+0000 7fa915a00640 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecca530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.620+0000 7fa915a00640 1 -- 172.21.15.40:0/3431077983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee4ebba510 con 0x55ee4ecbfde0 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.620+0000 7fa915000640 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.620+0000 7fa916400640 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ecd1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.621+0000 7fa915a00640 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc9ff0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa9040045f0 tx=0x7fa904004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a555b342a584548 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.621+0000 7fa8f7400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.621+0000 7fa8f7400640 1 -- 172.21.15.40:0/3431077983 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90401f640 con 0x55ee4ecbfde0 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.621+0000 7fa8f7400640 1 -- 172.21.15.40:0/3431077983 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fa90401f7e0 con 0x55ee4ecbfde0 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.621+0000 7fa91b215740 1 -- 172.21.15.40:0/3431077983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8fc003aa0 con 0x55ee4ecbfde0 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.621+0000 7fa8f7400640 1 -- 172.21.15.40:0/3431077983 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa90401f9e0 con 0x55ee4ecbfde0 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.621+0000 7fa91b215740 10 client.15933 fetch_fsmap learned FSMap version 24 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.621+0000 7fa91b215740 1 -- 172.21.15.40:0/3431077983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ee4e936210 con 0x55ee4ecbfde0 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.622+0000 7fa8f7400640 1 -- 172.21.15.40:0/3431077983 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8fc003aa0 con 0x55ee4ecbfde0 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.622+0000 7fa8f7400640 1 -- 172.21.15.40:0/3431077983 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa904027070 con 0x55ee4ecbfde0 2024-04-27T00:10:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.622+0000 7fa91b215740 10 client.15933 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.622+0000 7fa91b215740 10 client.15933 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.622+0000 7fa91b215740 20 client.15933 populate_metadata read hostname 'smithi040' 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.622+0000 7fa91b215740 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ee4ecd5b90 0x55ee4ecd7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.622+0000 7fa91b215740 4 client.15933 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.622+0000 7fa91b215740 1 -- 172.21.15.40:0/3431077983 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee4ec31f90 con 0x55ee4ecd5b90 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.622+0000 7fa915000640 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ee4ecd5b90 0x55ee4ecd7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.623+0000 7fa915000640 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ee4ecd5b90 0x55ee4ecd7fc0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ad11d60a27c32bc server_cookie=f351534ff327d0d8 in_seq=0 out_seq=0 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.623+0000 7fa8f7400640 10 client.15933 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa8f7400640 1 -- 172.21.15.40:0/3431077983 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee4ec31f90 con 0x55ee4ecd5b90 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa8f7400640 10 client.15933 handle_command_reply: tid=0 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa91b215740 1 client.15933 shutdown 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa91b215740 1 -- 172.21.15.40:0/3431077983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecbfde0 msgr2=0x55ee4ecc9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa91b215740 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc9ff0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa9040045f0 tx=0x7fa904004730 comp rx=0 tx=0).stop 2024-04-27T00:10:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa91b215740 1 -- 172.21.15.40:0/3431077983 shutdown_connections 2024-04-27T00:10:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa91b215740 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ee4ecd5b90 0x55ee4ecd7fc0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa91b215740 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ecd1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa91b215740 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa91b215740 1 --2- 172.21.15.40:0/3431077983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc9ff0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa8f7400640 0 client.15933 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa91b215740 1 -- 172.21.15.40:0/3431077983 >> 172.21.15.40:0/3431077983 conn(0x55ee4ecbb450 msgr2=0x55ee4ecba650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa91b215740 1 -- 172.21.15.40:0/3431077983 shutdown_connections 2024-04-27T00:10:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa91b215740 1 -- 172.21.15.40:0/3431077983 wait complete. 2024-04-27T00:10:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.624+0000 7fa91b215740 20 client.15933 trim_cache size 0 max 16384 2024-04-27T00:10:10.641 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.641+0000 7fa91b215740 1 Processor -- start 2024-04-27T00:10:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.641+0000 7fa91b215740 1 -- start start 2024-04-27T00:10:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.641+0000 7fa91b215740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.641+0000 7fa91b215740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecd1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.641+0000 7fa91b215740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ecc9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.641+0000 7fa91b215740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa9101afdf0 con 0x55ee4ecc07d0 2024-04-27T00:10:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.641+0000 7fa91b215740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa9101ad6c0 con 0x55ee4ecbfde0 2024-04-27T00:10:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.641+0000 7fa91b215740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9101ab510 con 0x55ee4ecc5940 2024-04-27T00:10:10.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.642+0000 7fa915000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecd1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.642+0000 7fa915000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecd1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36486/0 (socket says 172.21.15.40:36486) 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.642+0000 7fa915000640 1 -- 172.21.15.40:0/3766119116 learned_addr learned my addr 172.21.15.40:0/3766119116 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.642+0000 7fa915a00640 1 --2- 172.21.15.40:0/3766119116 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.642+0000 7fa916400640 1 --2- 172.21.15.40:0/3766119116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ecc9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.642+0000 7fa915000640 1 -- 172.21.15.40:0/3766119116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 msgr2=0x55ee4ecc9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.642+0000 7fa915000640 1 --2- 172.21.15.40:0/3766119116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ecc9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.642+0000 7fa915000640 1 -- 172.21.15.40:0/3766119116 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecbfde0 msgr2=0x55ee4ecc94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.642+0000 7fa915000640 1 --2- 172.21.15.40:0/3766119116 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.642+0000 7fa915000640 1 -- 172.21.15.40:0/3766119116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee4ea02620 con 0x55ee4ecc07d0 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.642+0000 7fa915000640 1 --2- 172.21.15.40:0/3766119116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecd1530 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009230 tx=0x7fa8fc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd15960954079e1a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.643+0000 7fa902000640 1 -- 172.21.15.40:0/3766119116 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc053070 con 0x55ee4ecc07d0 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.643+0000 7fa902000640 1 -- 172.21.15.40:0/3766119116 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8fc04e160 con 0x55ee4ecc07d0 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.643+0000 7fa902000640 1 -- 172.21.15.40:0/3766119116 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc035040 con 0x55ee4ecc07d0 2024-04-27T00:10:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.643+0000 7fa91b215740 1 -- 172.21.15.40:0/3766119116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 msgr2=0x55ee4ecd1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:10.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.643+0000 7fa91b215740 1 --2- 172.21.15.40:0/3766119116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecd1530 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009230 tx=0x7fa8fc009260 comp rx=0 tx=0).stop 2024-04-27T00:10:10.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.643+0000 7fa91b215740 1 -- 172.21.15.40:0/3766119116 shutdown_connections 2024-04-27T00:10:10.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.643+0000 7fa91b215740 1 --2- 172.21.15.40:0/3766119116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ecc9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.643+0000 7fa91b215740 1 --2- 172.21.15.40:0/3766119116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecd1530 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.643+0000 7fa91b215740 1 --2- 172.21.15.40:0/3766119116 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecc94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.643+0000 7fa91b215740 1 -- 172.21.15.40:0/3766119116 >> 172.21.15.40:0/3766119116 conn(0x55ee4ecbb450 msgr2=0x55ee4ecc8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:10.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.643+0000 7fa91b215740 1 -- 172.21.15.40:0/3766119116 shutdown_connections 2024-04-27T00:10:10.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.644+0000 7fa91b215740 1 -- 172.21.15.40:0/3766119116 wait complete. 2024-04-27T00:10:10.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.644+0000 7fa91b215740 1 -- start start 2024-04-27T00:10:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.644+0000 7fa91b215740 1 Processor -- start 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa91b215740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa91b215740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecceac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa91b215740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ed3a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa91b215740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa9101abb80 con 0x55ee4ecc07d0 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa91b215740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa9101acff0 con 0x55ee4ecbfde0 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa91b215740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9101ac250 con 0x55ee4ecc5940 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa91b215740 10 client.0 _ll_register_callbacks cb 0x7fa904009c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa915000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa915000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecceac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36498/0 (socket says 172.21.15.40:36498) 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa915000640 1 -- 172.21.15.40:0/1628641234 learned_addr learned my addr 172.21.15.40:0/1628641234 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa916400640 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ed3a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa915a00640 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa915000640 1 -- 172.21.15.40:0/1628641234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 msgr2=0x55ee4ed3a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa915000640 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ed3a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa915000640 1 -- 172.21.15.40:0/1628641234 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecbfde0 msgr2=0x55ee4ecce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa915000640 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.645+0000 7fa915000640 1 -- 172.21.15.40:0/1628641234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee4ebdecb0 con 0x55ee4ecc07d0 2024-04-27T00:10:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.646+0000 7fa915000640 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecceac0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc051040 tx=0x7fa8fc005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e448b09d31012d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.646+0000 7fa8f7400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:10:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.646+0000 7fa8f7400640 1 -- 172.21.15.40:0/1628641234 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc05c060 con 0x55ee4ecc07d0 2024-04-27T00:10:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.646+0000 7fa8f7400640 1 -- 172.21.15.40:0/1628641234 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fa8fc04e720 con 0x55ee4ecc07d0 2024-04-27T00:10:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.646+0000 7fa91b215740 1 -- 172.21.15.40:0/1628641234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8fc04e160 con 0x55ee4ecc07d0 2024-04-27T00:10:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.646+0000 7fa8f7400640 1 -- 172.21.15.40:0/1628641234 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa8fc063050 con 0x55ee4ecc07d0 2024-04-27T00:10:10.651 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.646+0000 7fa91b215740 10 client.15939 fetch_fsmap learned FSMap version 24 2024-04-27T00:10:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.646+0000 7fa91b215740 1 -- 172.21.15.40:0/1628641234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ee4ed3b800 con 0x55ee4ecc07d0 2024-04-27T00:10:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.647+0000 7fa8f7400640 1 -- 172.21.15.40:0/1628641234 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8fc04e160 con 0x55ee4ecc07d0 2024-04-27T00:10:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.647+0000 7fa8f7400640 1 -- 172.21.15.40:0/1628641234 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa8fc031850 con 0x55ee4ecc07d0 2024-04-27T00:10:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.647+0000 7fa91b215740 10 client.15939 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:10:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.647+0000 7fa91b215740 10 client.15939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:10:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.647+0000 7fa91b215740 20 client.15939 populate_metadata read hostname 'smithi040' 2024-04-27T00:10:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.647+0000 7fa91b215740 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ee4ed3bcb0 0x55ee4ed3e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.647+0000 7fa91b215740 4 client.15939 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:10:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.647+0000 7fa91b215740 1 -- 172.21.15.40:0/1628641234 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ee4ec2be00 con 0x55ee4ed3bcb0 2024-04-27T00:10:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.647+0000 7fa915a00640 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ee4ed3bcb0 0x55ee4ed3e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.648+0000 7fa915a00640 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ee4ed3bcb0 0x55ee4ed3e0a0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ceffd5fac07bd16 server_cookie=bde5bacc73017497 in_seq=0 out_seq=0 2024-04-27T00:10:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.648+0000 7fa8f7400640 10 client.15939 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.662+0000 7fa8f7400640 1 -- 172.21.15.40:0/1628641234 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ee4ec2be00 con 0x55ee4ed3bcb0 2024-04-27T00:10:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.662+0000 7fa8f7400640 10 client.15939 handle_command_reply: tid=0 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.662+0000 7fa91b215740 1 client.15939 shutdown 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.663+0000 7fa91b215740 1 -- 172.21.15.40:0/1628641234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 msgr2=0x55ee4ecceac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.663+0000 7fa91b215740 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecceac0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc051040 tx=0x7fa8fc005e80 comp rx=0 tx=0).stop 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.663+0000 7fa91b215740 1 -- 172.21.15.40:0/1628641234 shutdown_connections 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.663+0000 7fa91b215740 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ee4ed3bcb0 0x55ee4ed3e0a0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.663+0000 7fa91b215740 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4ecc5940 0x55ee4ed3a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.663+0000 7fa91b215740 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee4ecc07d0 0x55ee4ecceac0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.663+0000 7fa91b215740 1 --2- 172.21.15.40:0/1628641234 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ee4ecbfde0 0x55ee4ecce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.663+0000 7fa8f7400640 0 client.15939 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.663+0000 7fa91b215740 1 -- 172.21.15.40:0/1628641234 >> 172.21.15.40:0/1628641234 conn(0x55ee4ecbb450 msgr2=0x55ee4ecc8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.663+0000 7fa91b215740 1 -- 172.21.15.40:0/1628641234 shutdown_connections 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.663+0000 7fa91b215740 1 -- 172.21.15.40:0/1628641234 wait complete. 2024-04-27T00:10:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.663+0000 7fa91b215740 20 client.15939 trim_cache size 0 max 16384 2024-04-27T00:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:10:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:10:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.664+0000 7fa916e00640 1 -- 172.21.15.40:0/284859934 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa8f80a3c30 msgr2=0x7fa8f80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.664+0000 7fa916e00640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa8f80a3c30 0x7fa8f80a60f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x55ee4ecc0bd0 tx=0x7fa904005f90 comp rx=0 tx=0).stop 2024-04-27T00:10:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.664+0000 7fa916e00640 1 -- 172.21.15.40:0/284859934 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa91010a6a0 msgr2=0x7fa9101a21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.664+0000 7fa916e00640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa91010a6a0 0x7fa9101a21e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa90800efd0 tx=0x7fa90800c6a0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.664+0000 7fa916e00640 1 -- 172.21.15.40:0/284859934 shutdown_connections 2024-04-27T00:10:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.664+0000 7fa916e00640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa91010a6a0 0x7fa9101a21e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.664+0000 7fa916e00640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9100768f0 0x7fa91019b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.664+0000 7fa916e00640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa8f80a3c30 0x7fa8f80a60f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.664+0000 7fa916e00640 1 --2- 172.21.15.40:0/284859934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa910075310 0x7fa91019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.664+0000 7fa916e00640 1 -- 172.21.15.40:0/284859934 >> 172.21.15.40:0/284859934 conn(0x7fa9100fdd60 msgr2=0x7fa910106630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.664+0000 7fa916e00640 1 -- 172.21.15.40:0/284859934 shutdown_connections 2024-04-27T00:10:10.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:10.664+0000 7fa916e00640 1 -- 172.21.15.40:0/284859934 wait complete. 2024-04-27T00:10:12.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:12 smithi040 ceph-mon[28095]: pgmap v667: 129 pgs: 129 active+clean; 1.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-04-27T00:10:12.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:12 smithi080 ceph-mon[32362]: pgmap v667: 129 pgs: 129 active+clean; 1.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-04-27T00:10:12.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:12 smithi103 ceph-mon[32952]: pgmap v667: 129 pgs: 129 active+clean; 1.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-04-27T00:10:14.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:14 smithi040 ceph-mon[28095]: pgmap v668: 129 pgs: 129 active+clean; 1.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:10:14.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:14 smithi080 ceph-mon[32362]: pgmap v668: 129 pgs: 129 active+clean; 1.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:10:14.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:14 smithi103 ceph-mon[32952]: pgmap v668: 129 pgs: 129 active+clean; 1.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:10:16.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:16 smithi040 ceph-mon[28095]: pgmap v669: 129 pgs: 129 active+clean; 1.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:10:16.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:16 smithi080 ceph-mon[32362]: pgmap v669: 129 pgs: 129 active+clean; 1.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:10:16.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:16 smithi103 ceph-mon[32952]: pgmap v669: 129 pgs: 129 active+clean; 1.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:10:18.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:18 smithi040 ceph-mon[28095]: pgmap v670: 129 pgs: 129 active+clean; 1.8 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:10:18.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:18 smithi080 ceph-mon[32362]: pgmap v670: 129 pgs: 129 active+clean; 1.8 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:10:18.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:18 smithi103 ceph-mon[32952]: pgmap v670: 129 pgs: 129 active+clean; 1.8 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:10:20.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:20 smithi040 ceph-mon[28095]: pgmap v671: 129 pgs: 129 active+clean; 1.8 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:20.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:20 smithi080 ceph-mon[32362]: pgmap v671: 129 pgs: 129 active+clean; 1.8 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:20.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:20 smithi103 ceph-mon[32952]: pgmap v671: 129 pgs: 129 active+clean; 1.8 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:22.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:22 smithi040 ceph-mon[28095]: pgmap v672: 129 pgs: 129 active+clean; 1.8 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:10:22.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:22 smithi080 ceph-mon[32362]: pgmap v672: 129 pgs: 129 active+clean; 1.8 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:10:22.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:22 smithi103 ceph-mon[32952]: pgmap v672: 129 pgs: 129 active+clean; 1.8 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:10:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.815+0000 7f2669c00640 1 -- 172.21.15.40:0/2209532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2664105be0 msgr2=0x7f2664106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.815+0000 7f2669c00640 1 --2- 172.21.15.40:0/2209532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2664105be0 0x7f2664106060 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2650009900 tx=0x7f265002a410 comp rx=0 tx=0).stop 2024-04-27T00:10:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.815+0000 7f2669c00640 1 -- 172.21.15.40:0/2209532 shutdown_connections 2024-04-27T00:10:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.815+0000 7f2669c00640 1 --2- 172.21.15.40:0/2209532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26641065a0 0x7f266410ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.815+0000 7f2669c00640 1 --2- 172.21.15.40:0/2209532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2664105be0 0x7f2664106060 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.815+0000 7f2669c00640 1 --2- 172.21.15.40:0/2209532 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f26641049f0 0x7f2664104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.815+0000 7f2669c00640 1 -- 172.21.15.40:0/2209532 >> 172.21.15.40:0/2209532 conn(0x7f26640fffa0 msgr2=0x7f26641023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.816+0000 7f2669c00640 1 -- 172.21.15.40:0/2209532 shutdown_connections 2024-04-27T00:10:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.816+0000 7f2669c00640 1 -- 172.21.15.40:0/2209532 wait complete. 2024-04-27T00:10:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.817+0000 7f2669c00640 1 Processor -- start 2024-04-27T00:10:23.821 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.817+0000 7f2669c00640 1 -- start start 2024-04-27T00:10:23.823 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.818+0000 7f2669c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26641049f0 0x7f266419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.818+0000 7f2669c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2664105be0 0x7f266419b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.818+0000 7f2669c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26641065a0 0x7f26641a2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.818+0000 7f2669c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2664110520 con 0x7f26641065a0 2024-04-27T00:10:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.818+0000 7f2669c00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f26641103a0 con 0x7f2664105be0 2024-04-27T00:10:23.824 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.818+0000 7f2669c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f26641106a0 con 0x7f26641049f0 2024-04-27T00:10:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.818+0000 7f2663e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26641049f0 0x7f266419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.818+0000 7f2663400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2664105be0 0x7f266419b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.818+0000 7f2663e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26641049f0 0x7f266419abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38570/0 (socket says 172.21.15.40:38570) 2024-04-27T00:10:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.818+0000 7f2663e00640 1 -- 172.21.15.40:0/822818925 learned_addr learned my addr 172.21.15.40:0/822818925 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:23.825 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.818+0000 7f2669200640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26641065a0 0x7f26641a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.826 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.819+0000 7f2663e00640 1 -- 172.21.15.40:0/822818925 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2664105be0 msgr2=0x7f266419b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:23.826 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.819+0000 7f2663e00640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2664105be0 0x7f266419b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.826 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.819+0000 7f2663e00640 1 -- 172.21.15.40:0/822818925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26641065a0 msgr2=0x7f26641a2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:23.826 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.819+0000 7f2663e00640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26641065a0 0x7f26641a2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.827 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.819+0000 7f2663e00640 1 -- 172.21.15.40:0/822818925 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26641a27f0 con 0x7f26641049f0 2024-04-27T00:10:23.827 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.819+0000 7f2663e00640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26641049f0 0x7f266419abd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f265400e9c0 tx=0x7f265400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:23.827 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.819+0000 7f2660c00640 1 -- 172.21.15.40:0/822818925 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265400cde0 con 0x7f26641049f0 2024-04-27T00:10:23.827 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.819+0000 7f2660c00640 1 -- 172.21.15.40:0/822818925 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26540108e0 con 0x7f26641049f0 2024-04-27T00:10:23.827 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.819+0000 7f2669c00640 1 -- 172.21.15.40:0/822818925 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26641a2ae0 con 0x7f26641049f0 2024-04-27T00:10:23.827 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.819+0000 7f2660c00640 1 -- 172.21.15.40:0/822818925 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26540188d0 con 0x7f26641049f0 2024-04-27T00:10:23.828 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.820+0000 7f2669c00640 1 -- 172.21.15.40:0/822818925 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26641a3000 con 0x7f26641049f0 2024-04-27T00:10:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.823+0000 7f2660c00640 1 -- 172.21.15.40:0/822818925 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f265401f020 con 0x7f26641049f0 2024-04-27T00:10:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.823+0000 7f266e029740 1 Processor -- start 2024-04-27T00:10:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.823+0000 7f266e029740 1 -- start start 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.823+0000 7f266e029740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 0x562b36cf6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.823+0000 7f266e029740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf67d0 0x562b36cfb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.823+0000 7f266e029740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 0x562b36cfddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.823+0000 7f266e029740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562b36c5a380 con 0x562b36cf5de0 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.823+0000 7f266e029740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x562b36bf6c30 con 0x562b36cfb940 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.823+0000 7f266e029740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562b36c56340 con 0x562b36cf67d0 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.824+0000 7f2669200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 0x562b36cf6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.824+0000 7f2669200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 0x562b36cf6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56406/0 (socket says 172.21.15.40:56406) 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.824+0000 7f2669200640 1 -- 172.21.15.40:0/189594653 learned_addr learned my addr 172.21.15.40:0/189594653 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.824+0000 7f2663e00640 1 --2- 172.21.15.40:0/189594653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf67d0 0x562b36cfb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.824+0000 7f2663400640 1 --2- 172.21.15.40:0/189594653 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 0x562b36cfddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.824+0000 7f2669200640 1 -- 172.21.15.40:0/189594653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf67d0 msgr2=0x562b36cfb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.824+0000 7f2669200640 1 --2- 172.21.15.40:0/189594653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf67d0 0x562b36cfb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.824+0000 7f2669200640 1 -- 172.21.15.40:0/189594653 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 msgr2=0x562b36cfddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.824+0000 7f2669200640 1 --2- 172.21.15.40:0/189594653 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 0x562b36cfddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.824+0000 7f2669200640 1 -- 172.21.15.40:0/189594653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b36c18940 con 0x562b36cf5de0 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.824+0000 7f2660c00640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f26480a3c30 0x7f26480a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.825+0000 7f2669200640 1 --2- 172.21.15.40:0/189594653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 0x562b36cf6200 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f265c009fd0 tx=0x7f265c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28975bd92fc4588 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.825+0000 7f2660c00640 1 -- 172.21.15.40:0/822818925 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f26540d0b00 con 0x7f26641049f0 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.825+0000 7f2663e00640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f26480a3c30 0x7f26480a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.825+0000 7f2659600640 1 -- 172.21.15.40:0/189594653 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265c004d20 con 0x562b36cf5de0 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.825+0000 7f2659600640 1 -- 172.21.15.40:0/189594653 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f265c004050 con 0x562b36cf5de0 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.825+0000 7f2659600640 1 -- 172.21.15.40:0/189594653 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265c005280 con 0x562b36cf5de0 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.826+0000 7f266e029740 1 -- 172.21.15.40:0/189594653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 msgr2=0x562b36cf6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.826+0000 7f2663e00640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f26480a3c30 0x7f26480a60f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x562b36cf6bd0 tx=0x7f26540d45b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.826+0000 7f266e029740 1 --2- 172.21.15.40:0/189594653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 0x562b36cf6200 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f265c009fd0 tx=0x7f265c0118d0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.826+0000 7f266e029740 1 -- 172.21.15.40:0/189594653 shutdown_connections 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.826+0000 7f266e029740 1 --2- 172.21.15.40:0/189594653 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 0x562b36cfddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.826+0000 7f266e029740 1 --2- 172.21.15.40:0/189594653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf67d0 0x562b36cfb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.826+0000 7f266e029740 1 --2- 172.21.15.40:0/189594653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 0x562b36cf6200 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.826+0000 7f266e029740 1 -- 172.21.15.40:0/189594653 >> 172.21.15.40:0/189594653 conn(0x562b36cf1450 msgr2=0x562b36cf1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.826+0000 7f266e029740 1 -- 172.21.15.40:0/189594653 shutdown_connections 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.826+0000 7f266e029740 1 -- 172.21.15.40:0/189594653 wait complete. 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.826+0000 7f266e029740 1 -- start start 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.827+0000 7f266e029740 1 Processor -- start 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.827+0000 7f266e029740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf5de0 0x562b36d00020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.827+0000 7f266e029740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cf67d0 0x562b36d00560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.827+0000 7f266e029740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cfb940 0x562b36d075a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.827+0000 7f266e029740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562b36cfe8f0 con 0x562b36cfb940 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.827+0000 7f266e029740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x562b36be1c60 con 0x562b36cf67d0 2024-04-27T00:10:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.827+0000 7f266e029740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562b36cfea70 con 0x562b36cf5de0 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.827+0000 7f266e029740 10 client.0 _ll_register_callbacks cb 0x562b36c0de50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.827+0000 7f2663400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cf67d0 0x562b36d00560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.827+0000 7f2663400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cf67d0 0x562b36d00560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:44534/0 (socket says 172.21.15.40:44534) 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.827+0000 7f2663400640 1 -- 172.21.15.40:0/618574106 learned_addr learned my addr 172.21.15.40:0/618574106 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.827+0000 7f2669200640 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf5de0 0x562b36d00020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.828+0000 7f2669200640 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cfb940 0x562b36d075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.828+0000 7f2663400640 1 -- 172.21.15.40:0/618574106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf5de0 msgr2=0x562b36d00020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.828+0000 7f2663400640 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf5de0 0x562b36d00020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.828+0000 7f2663400640 1 -- 172.21.15.40:0/618574106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cfb940 msgr2=0x562b36d075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.828+0000 7f2663400640 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cfb940 0x562b36d075a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.828+0000 7f2663400640 1 -- 172.21.15.40:0/618574106 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b36bf0510 con 0x562b36cf67d0 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.828+0000 7f2663400640 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cf67d0 0x562b36d00560 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2650009900 tx=0x7f2650031db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13cfb98e05ffa305 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.828+0000 7f264e000640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:10:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.828+0000 7f264e000640 1 -- 172.21.15.40:0/618574106 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2650035040 con 0x562b36cf67d0 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.828+0000 7f266e029740 1 -- 172.21.15.40:0/618574106 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f265c004050 con 0x562b36cf67d0 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.828+0000 7f264e000640 1 -- 172.21.15.40:0/618574106 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f2650004040 con 0x562b36cf67d0 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.829+0000 7f264e000640 1 -- 172.21.15.40:0/618574106 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f265004e050 con 0x562b36cf67d0 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.829+0000 7f266e029740 10 client.26063 fetch_fsmap learned FSMap version 24 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.829+0000 7f266e029740 1 -- 172.21.15.40:0/618574106 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562b3696c210 con 0x562b36cf67d0 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.829+0000 7f264e000640 1 -- 172.21.15.40:0/618574106 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f265c004050 con 0x562b36cf67d0 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.829+0000 7f264e000640 1 -- 172.21.15.40:0/618574106 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f265003c440 con 0x562b36cf67d0 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.829+0000 7f266e029740 10 client.26063 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.829+0000 7f266e029740 10 client.26063 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.829+0000 7f266e029740 20 client.26063 populate_metadata read hostname 'smithi040' 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.829+0000 7f266e029740 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b36d0bbc0 0x562b36d0dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.829+0000 7f266e029740 4 client.26063 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.829+0000 7f266e029740 1 -- 172.21.15.40:0/618574106 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b36c67f90 con 0x562b36d0bbc0 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.830+0000 7f2669200640 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b36d0bbc0 0x562b36d0dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.830+0000 7f2669200640 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b36d0bbc0 0x562b36d0dfb0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e5d324c97af6910 server_cookie=9bc7bbb4bccc4ca8 in_seq=0 out_seq=0 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.830+0000 7f264e000640 10 client.26063 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f264e000640 1 -- 172.21.15.40:0/618574106 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562b36c67f90 con 0x562b36d0bbc0 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f264e000640 10 client.26063 handle_command_reply: tid=0 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f266e029740 1 client.26063 shutdown 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f266e029740 1 -- 172.21.15.40:0/618574106 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cf67d0 msgr2=0x562b36d00560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f266e029740 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cf67d0 0x562b36d00560 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2650009900 tx=0x7f2650031db0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f266e029740 1 -- 172.21.15.40:0/618574106 shutdown_connections 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f266e029740 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b36d0bbc0 0x562b36d0dfb0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f266e029740 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cfb940 0x562b36d075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f266e029740 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cf67d0 0x562b36d00560 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f266e029740 1 --2- 172.21.15.40:0/618574106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf5de0 0x562b36d00020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f264e000640 0 client.26063 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f266e029740 1 -- 172.21.15.40:0/618574106 >> 172.21.15.40:0/618574106 conn(0x562b36cf1450 msgr2=0x562b36cf0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.831+0000 7f266e029740 1 -- 172.21.15.40:0/618574106 shutdown_connections 2024-04-27T00:10:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.832+0000 7f266e029740 1 -- 172.21.15.40:0/618574106 wait complete. 2024-04-27T00:10:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.832+0000 7f266e029740 20 client.26063 trim_cache size 0 max 16384 2024-04-27T00:10:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.848+0000 7f266e029740 1 Processor -- start 2024-04-27T00:10:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.848+0000 7f266e029740 1 -- start start 2024-04-27T00:10:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.849+0000 7f266e029740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf5de0 0x562b36d00020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.849+0000 7f266e029740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf67d0 0x562b36d00560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.849+0000 7f266e029740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 0x562b36d07560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.849+0000 7f266e029740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f26641afd20 con 0x562b36cf67d0 2024-04-27T00:10:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.849+0000 7f266e029740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f26641ad5f0 con 0x562b36cfb940 2024-04-27T00:10:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.849+0000 7f266e029740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f26641ab440 con 0x562b36cf5de0 2024-04-27T00:10:23.850 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.849+0000 7f2663400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf67d0 0x562b36d00560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.849+0000 7f2669200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf5de0 0x562b36d00020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.849+0000 7f2663400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf67d0 0x562b36d00560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56424/0 (socket says 172.21.15.40:56424) 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.849+0000 7f2669200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf5de0 0x562b36d00020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38602/0 (socket says 172.21.15.40:38602) 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.849+0000 7f2663400640 1 -- 172.21.15.40:0/1906123734 learned_addr learned my addr 172.21.15.40:0/1906123734 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.849+0000 7f2669200640 1 --2- 172.21.15.40:0/1906123734 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 0x562b36d07560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.850+0000 7f2663400640 1 -- 172.21.15.40:0/1906123734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf5de0 msgr2=0x562b36d00020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.850+0000 7f2663400640 1 --2- 172.21.15.40:0/1906123734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf5de0 0x562b36d00020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.850+0000 7f2663400640 1 -- 172.21.15.40:0/1906123734 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 msgr2=0x562b36d07560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.850+0000 7f2663400640 1 --2- 172.21.15.40:0/1906123734 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 0x562b36d07560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.850+0000 7f2663400640 1 -- 172.21.15.40:0/1906123734 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b36a38620 con 0x562b36cf67d0 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.850+0000 7f2669200640 1 --2- 172.21.15.40:0/1906123734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf5de0 0x562b36d00020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.850+0000 7f2663400640 1 --2- 172.21.15.40:0/1906123734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf67d0 0x562b36d00560 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f265003c8e0 tx=0x7f2650032fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8ba1b5c8205560 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.850+0000 7f2658c00640 1 -- 172.21.15.40:0/1906123734 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2650057070 con 0x562b36cf67d0 2024-04-27T00:10:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.850+0000 7f2658c00640 1 -- 172.21.15.40:0/1906123734 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f265003c440 con 0x562b36cf67d0 2024-04-27T00:10:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.850+0000 7f2658c00640 1 -- 172.21.15.40:0/1906123734 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2650035040 con 0x562b36cf67d0 2024-04-27T00:10:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.851+0000 7f266e029740 1 -- 172.21.15.40:0/1906123734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf67d0 msgr2=0x562b36d00560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.851+0000 7f266e029740 1 --2- 172.21.15.40:0/1906123734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf67d0 0x562b36d00560 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f265003c8e0 tx=0x7f2650032fc0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.851+0000 7f266e029740 1 -- 172.21.15.40:0/1906123734 shutdown_connections 2024-04-27T00:10:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.851+0000 7f266e029740 1 --2- 172.21.15.40:0/1906123734 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 0x562b36d07560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.851+0000 7f266e029740 1 --2- 172.21.15.40:0/1906123734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf67d0 0x562b36d00560 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.851+0000 7f266e029740 1 --2- 172.21.15.40:0/1906123734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf5de0 0x562b36d00020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.851+0000 7f266e029740 1 -- 172.21.15.40:0/1906123734 >> 172.21.15.40:0/1906123734 conn(0x562b36cf1450 msgr2=0x562b36cf1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.851+0000 7f266e029740 1 -- 172.21.15.40:0/1906123734 shutdown_connections 2024-04-27T00:10:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.851+0000 7f266e029740 1 -- 172.21.15.40:0/1906123734 wait complete. 2024-04-27T00:10:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.851+0000 7f266e029740 1 -- start start 2024-04-27T00:10:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f266e029740 1 Processor -- start 2024-04-27T00:10:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f266e029740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 0x562b36d04560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f266e029740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf67d0 0x562b36d04aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f266e029740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 0x562b36d70120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f266e029740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f26641abab0 con 0x562b36cf5de0 2024-04-27T00:10:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f266e029740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f26641acf20 con 0x562b36cfb940 2024-04-27T00:10:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f266e029740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f26641ac180 con 0x562b36cf67d0 2024-04-27T00:10:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f266e029740 10 client.0 _ll_register_callbacks cb 0x7f2650009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:10:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f2669200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 0x562b36d04560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f2663400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf67d0 0x562b36d04aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f2663400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf67d0 0x562b36d04aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38604/0 (socket says 172.21.15.40:38604) 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f2669200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 0x562b36d04560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56434/0 (socket says 172.21.15.40:56434) 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.852+0000 7f2663400640 1 -- 172.21.15.40:0/1800858272 learned_addr learned my addr 172.21.15.40:0/1800858272 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.853+0000 7f2669200640 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 0x562b36d70120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.853+0000 7f2669200640 1 -- 172.21.15.40:0/1800858272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf67d0 msgr2=0x562b36d04aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.853+0000 7f2669200640 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf67d0 0x562b36d04aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.853+0000 7f2669200640 1 -- 172.21.15.40:0/1800858272 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 msgr2=0x562b36d70120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.853+0000 7f2669200640 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 0x562b36d70120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.853+0000 7f2669200640 1 -- 172.21.15.40:0/1800858272 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b36c14cb0 con 0x562b36cf5de0 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.853+0000 7f2663400640 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf67d0 0x562b36d04aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.853+0000 7f2669200640 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 0x562b36d04560 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f265c009fa0 tx=0x7f2650005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2174c4886c530ced server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.853+0000 7f264e000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.853+0000 7f264e000640 1 -- 172.21.15.40:0/1800858272 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265c00fa70 con 0x562b36cf5de0 2024-04-27T00:10:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.853+0000 7f264e000640 1 -- 172.21.15.40:0/1800858272 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f265c01f040 con 0x562b36cf5de0 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.853+0000 7f266e029740 1 -- 172.21.15.40:0/1800858272 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f265003c440 con 0x562b36cf5de0 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.854+0000 7f264e000640 1 -- 172.21.15.40:0/1800858272 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f265c00fc10 con 0x562b36cf5de0 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.854+0000 7f266e029740 10 client.15957 fetch_fsmap learned FSMap version 24 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.854+0000 7f266e029740 1 -- 172.21.15.40:0/1800858272 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562b36d717a0 con 0x562b36cf5de0 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.854+0000 7f264e000640 1 -- 172.21.15.40:0/1800858272 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f265003c440 con 0x562b36cf5de0 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.854+0000 7f264e000640 1 -- 172.21.15.40:0/1800858272 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f265c026430 con 0x562b36cf5de0 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.854+0000 7f266e029740 10 client.15957 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.854+0000 7f266e029740 10 client.15957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.854+0000 7f266e029740 20 client.15957 populate_metadata read hostname 'smithi040' 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.854+0000 7f266e029740 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b36d71fc0 0x562b36d743e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.854+0000 7f266e029740 4 client.15957 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.854+0000 7f266e029740 1 -- 172.21.15.40:0/1800858272 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562b36c61e00 con 0x562b36d71fc0 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.855+0000 7f2663400640 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b36d71fc0 0x562b36d743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.855+0000 7f2663400640 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b36d71fc0 0x562b36d743e0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3ec0e5ecd9a90cd server_cookie=d506ec141ed1e4c8 in_seq=0 out_seq=0 2024-04-27T00:10:23.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.855+0000 7f264e000640 10 client.15957 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.870+0000 7f264e000640 1 -- 172.21.15.40:0/1800858272 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562b36c61e00 con 0x562b36d71fc0 2024-04-27T00:10:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.870+0000 7f264e000640 10 client.15957 handle_command_reply: tid=0 2024-04-27T00:10:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.870+0000 7f266e029740 1 client.15957 shutdown 2024-04-27T00:10:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.870+0000 7f266e029740 1 -- 172.21.15.40:0/1800858272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 msgr2=0x562b36d04560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.870+0000 7f266e029740 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 0x562b36d04560 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f265c009fa0 tx=0x7f2650005d50 comp rx=0 tx=0).stop 2024-04-27T00:10:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.870+0000 7f266e029740 1 -- 172.21.15.40:0/1800858272 shutdown_connections 2024-04-27T00:10:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.870+0000 7f266e029740 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562b36d71fc0 0x562b36d743e0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.870+0000 7f266e029740 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562b36cfb940 0x562b36d70120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.870+0000 7f266e029740 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562b36cf67d0 0x562b36d04aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.870+0000 7f266e029740 1 --2- 172.21.15.40:0/1800858272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562b36cf5de0 0x562b36d04560 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.870+0000 7f264e000640 0 client.15957 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.870+0000 7f266e029740 1 -- 172.21.15.40:0/1800858272 >> 172.21.15.40:0/1800858272 conn(0x562b36cf1450 msgr2=0x562b36d00020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.871+0000 7f266e029740 1 -- 172.21.15.40:0/1800858272 shutdown_connections 2024-04-27T00:10:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.871+0000 7f266e029740 1 -- 172.21.15.40:0/1800858272 wait complete. 2024-04-27T00:10:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.871+0000 7f266e029740 20 client.15957 trim_cache size 0 max 16384 2024-04-27T00:10:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:10:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:10:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:10:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:10:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.871+0000 7f2669c00640 1 -- 172.21.15.40:0/822818925 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f26480a3c30 msgr2=0x7f26480a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.871+0000 7f2669c00640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f26480a3c30 0x7f26480a60f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x562b36cf6bd0 tx=0x7f26540d45b0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.871+0000 7f2669c00640 1 -- 172.21.15.40:0/822818925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26641049f0 msgr2=0x7f266419abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.871+0000 7f2669c00640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26641049f0 0x7f266419abd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f265400e9c0 tx=0x7f265400ee80 comp rx=0 tx=0).stop 2024-04-27T00:10:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.872+0000 7f2669c00640 1 -- 172.21.15.40:0/822818925 shutdown_connections 2024-04-27T00:10:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.872+0000 7f2669c00640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26641065a0 0x7f26641a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.872+0000 7f2669c00640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2664105be0 0x7f266419b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.872+0000 7f2669c00640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f26480a3c30 0x7f26480a60f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.872+0000 7f2669c00640 1 --2- 172.21.15.40:0/822818925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26641049f0 0x7f266419abd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.872+0000 7f2669c00640 1 -- 172.21.15.40:0/822818925 >> 172.21.15.40:0/822818925 conn(0x7f26640fffa0 msgr2=0x7f2664108f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.872+0000 7f2669c00640 1 -- 172.21.15.40:0/822818925 shutdown_connections 2024-04-27T00:10:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:23.872+0000 7f2669c00640 1 -- 172.21.15.40:0/822818925 wait complete. 2024-04-27T00:10:24.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:24 smithi040 ceph-mon[28095]: pgmap v673: 129 pgs: 129 active+clean; 1.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:10:24.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:24 smithi080 ceph-mon[32362]: pgmap v673: 129 pgs: 129 active+clean; 1.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:10:24.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:24 smithi103 ceph-mon[32952]: pgmap v673: 129 pgs: 129 active+clean; 1.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:10:26.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:26 smithi040 ceph-mon[28095]: pgmap v674: 129 pgs: 129 active+clean; 1.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:10:26.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:26 smithi080 ceph-mon[32362]: pgmap v674: 129 pgs: 129 active+clean; 1.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:10:26.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:26 smithi103 ceph-mon[32952]: pgmap v674: 129 pgs: 129 active+clean; 1.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:10:28.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:28 smithi040 ceph-mon[28095]: pgmap v675: 129 pgs: 129 active+clean; 1.9 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:10:28.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:28 smithi080 ceph-mon[32362]: pgmap v675: 129 pgs: 129 active+clean; 1.9 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:10:28.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:28 smithi103 ceph-mon[32952]: pgmap v675: 129 pgs: 129 active+clean; 1.9 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:10:30.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:30 smithi040 ceph-mon[28095]: pgmap v676: 129 pgs: 129 active+clean; 1.9 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:30.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:30 smithi080 ceph-mon[32362]: pgmap v676: 129 pgs: 129 active+clean; 1.9 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:30.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:30 smithi103 ceph-mon[32952]: pgmap v676: 129 pgs: 129 active+clean; 1.9 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:32.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:32 smithi040 ceph-mon[28095]: pgmap v677: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:10:32.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:32 smithi080 ceph-mon[32362]: pgmap v677: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:10:32.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:32 smithi103 ceph-mon[32952]: pgmap v677: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:10:34.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:34 smithi040 ceph-mon[28095]: pgmap v678: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:10:34.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:34 smithi080 ceph-mon[32362]: pgmap v678: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:10:34.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:34 smithi103 ceph-mon[32952]: pgmap v678: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:10:36.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:36 smithi040 ceph-mon[28095]: pgmap v679: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:36.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:36 smithi080 ceph-mon[32362]: pgmap v679: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:36.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:36 smithi103 ceph-mon[32952]: pgmap v679: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.021+0000 7fde7a200640 1 -- 172.21.15.40:0/395772982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fde74075390 msgr2=0x7fde74075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.021+0000 7fde7a200640 1 --2- 172.21.15.40:0/395772982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fde74075390 0x7fde74075810 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fde60009860 tx=0x7fde6002a340 comp rx=0 tx=0).stop 2024-04-27T00:10:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.021+0000 7fde7a200640 1 -- 172.21.15.40:0/395772982 shutdown_connections 2024-04-27T00:10:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.021+0000 7fde7a200640 1 --2- 172.21.15.40:0/395772982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde7410a690 0x7fde7410cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.021+0000 7fde7a200640 1 --2- 172.21.15.40:0/395772982 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fde74075390 0x7fde74075810 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.021+0000 7fde7a200640 1 --2- 172.21.15.40:0/395772982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde74076970 0x7fde74074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.021+0000 7fde7a200640 1 -- 172.21.15.40:0/395772982 >> 172.21.15.40:0/395772982 conn(0x7fde740fdd40 msgr2=0x7fde74100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.021+0000 7fde7a200640 1 -- 172.21.15.40:0/395772982 shutdown_connections 2024-04-27T00:10:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.021+0000 7fde7a200640 1 -- 172.21.15.40:0/395772982 wait complete. 2024-04-27T00:10:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.022+0000 7fde7a200640 1 Processor -- start 2024-04-27T00:10:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.022+0000 7fde7a200640 1 -- start start 2024-04-27T00:10:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.022+0000 7fde7a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde74075390 0x7fde7419ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.022+0000 7fde7a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde74076970 0x7fde7419b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.022+0000 7fde7a200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fde7410a690 0x7fde741a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.022+0000 7fde7a200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde74110510 con 0x7fde74075390 2024-04-27T00:10:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.022+0000 7fde7a200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fde74110390 con 0x7fde7410a690 2024-04-27T00:10:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.022+0000 7fde7a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde74110690 con 0x7fde74076970 2024-04-27T00:10:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.023+0000 7fde78e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde74075390 0x7fde7419ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.023+0000 7fde73e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde74076970 0x7fde7419b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.023+0000 7fde78e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde74075390 0x7fde7419ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49412/0 (socket says 172.21.15.40:49412) 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.023+0000 7fde78e00640 1 -- 172.21.15.40:0/3773837219 learned_addr learned my addr 172.21.15.40:0/3773837219 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.023+0000 7fde79800640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fde7410a690 0x7fde741a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.023+0000 7fde78e00640 1 -- 172.21.15.40:0/3773837219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde74076970 msgr2=0x7fde7419b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.023+0000 7fde78e00640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde74076970 0x7fde7419b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.023+0000 7fde78e00640 1 -- 172.21.15.40:0/3773837219 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fde7410a690 msgr2=0x7fde741a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.023+0000 7fde78e00640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fde7410a690 0x7fde741a2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.023+0000 7fde78e00640 1 -- 172.21.15.40:0/3773837219 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde741a28d0 con 0x7fde74075390 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.024+0000 7fde78e00640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde74075390 0x7fde7419ac90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fde6800e9f0 tx=0x7fde6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.024+0000 7fde71600640 1 -- 172.21.15.40:0/3773837219 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6800cd80 con 0x7fde74075390 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.024+0000 7fde71600640 1 -- 172.21.15.40:0/3773837219 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde680108e0 con 0x7fde74075390 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.024+0000 7fde71600640 1 -- 172.21.15.40:0/3773837219 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde68018980 con 0x7fde74075390 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.024+0000 7fde7a200640 1 -- 172.21.15.40:0/3773837219 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde741a2bc0 con 0x7fde74075390 2024-04-27T00:10:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.024+0000 7fde7a200640 1 -- 172.21.15.40:0/3773837219 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde741a3090 con 0x7fde74075390 2024-04-27T00:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.027+0000 7fde7e764740 1 Processor -- start 2024-04-27T00:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.027+0000 7fde7e764740 1 -- start start 2024-04-27T00:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.028+0000 7fde71600640 1 -- 172.21.15.40:0/3773837219 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fde68010430 con 0x7fde74075390 2024-04-27T00:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.028+0000 7fde7e764740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e7de0 0x55729e9e8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.028+0000 7fde7e764740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e87d0 0x55729e9ed2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.028+0000 7fde7e764740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9ed940 0x55729e9efdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.028+0000 7fde7e764740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55729e94c380 con 0x55729e9ed940 2024-04-27T00:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.028+0000 7fde7e764740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55729e8e8c30 con 0x55729e9e7de0 2024-04-27T00:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.028+0000 7fde7e764740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55729e948340 con 0x55729e9e87d0 2024-04-27T00:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.028+0000 7fde79800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e7de0 0x55729e9e8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.028+0000 7fde73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9ed940 0x55729e9efdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.028+0000 7fde79800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e7de0 0x55729e9e8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:44724/0 (socket says 172.21.15.40:44724) 2024-04-27T00:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.028+0000 7fde79800640 1 -- 172.21.15.40:0/3396335957 learned_addr learned my addr 172.21.15.40:0/3396335957 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.028+0000 7fde78e00640 1 --2- 172.21.15.40:0/3396335957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e87d0 0x55729e9ed2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.029+0000 7fde79800640 1 -- 172.21.15.40:0/3396335957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e87d0 msgr2=0x55729e9ed2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.029+0000 7fde79800640 1 --2- 172.21.15.40:0/3396335957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e87d0 0x55729e9ed2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.029+0000 7fde79800640 1 -- 172.21.15.40:0/3396335957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9ed940 msgr2=0x55729e9efdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.029+0000 7fde79800640 1 --2- 172.21.15.40:0/3396335957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9ed940 0x55729e9efdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.029+0000 7fde79800640 1 -- 172.21.15.40:0/3396335957 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55729e90a940 con 0x55729e9e7de0 2024-04-27T00:10:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.029+0000 7fde78e00640 1 --2- 172.21.15.40:0/3396335957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e87d0 0x55729e9ed2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T00:10:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.029+0000 7fde79800640 1 --2- 172.21.15.40:0/3396335957 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e7de0 0x55729e9e8200 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00e450 tx=0x7fde6c0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=494faef2f7903d0a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.029+0000 7fde66000640 1 -- 172.21.15.40:0/3396335957 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c00d910 con 0x55729e9e7de0 2024-04-27T00:10:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.029+0000 7fde66000640 1 -- 172.21.15.40:0/3396335957 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde6c01c070 con 0x55729e9e7de0 2024-04-27T00:10:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.029+0000 7fde66000640 1 -- 172.21.15.40:0/3396335957 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c017e10 con 0x55729e9e7de0 2024-04-27T00:10:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.029+0000 7fde71600640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fde5c0a3c80 0x7fde5c0a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.030+0000 7fde71600640 1 -- 172.21.15.40:0/3773837219 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fde680d0c40 con 0x7fde74075390 2024-04-27T00:10:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.030+0000 7fde78e00640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fde5c0a3c80 0x7fde5c0a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.030+0000 7fde7e764740 1 -- 172.21.15.40:0/3396335957 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e7de0 msgr2=0x55729e9e8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.030+0000 7fde7e764740 1 --2- 172.21.15.40:0/3396335957 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e7de0 0x55729e9e8200 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00e450 tx=0x7fde6c0095c0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.030+0000 7fde7e764740 1 -- 172.21.15.40:0/3396335957 shutdown_connections 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.030+0000 7fde7e764740 1 --2- 172.21.15.40:0/3396335957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9ed940 0x55729e9efdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.030+0000 7fde7e764740 1 --2- 172.21.15.40:0/3396335957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e87d0 0x55729e9ed2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.030+0000 7fde7e764740 1 --2- 172.21.15.40:0/3396335957 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e7de0 0x55729e9e8200 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.030+0000 7fde7e764740 1 -- 172.21.15.40:0/3396335957 >> 172.21.15.40:0/3396335957 conn(0x55729e9e3450 msgr2=0x55729e9e3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.030+0000 7fde7e764740 1 -- 172.21.15.40:0/3396335957 shutdown_connections 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.030+0000 7fde78e00640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fde5c0a3c80 0x7fde5c0a6140 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x55729e9e8bd0 tx=0x7fde68000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.030+0000 7fde7e764740 1 -- 172.21.15.40:0/3396335957 wait complete. 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.031+0000 7fde7e764740 1 -- start start 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.031+0000 7fde7e764740 1 Processor -- start 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde7e764740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e7de0 0x55729e9f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde7e764740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e87d0 0x55729e9f2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde7e764740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9ed940 0x55729e9f9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde7e764740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55729e9f08f0 con 0x55729e9e87d0 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde7e764740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55729e8d3c60 con 0x55729e9ed940 2024-04-27T00:10:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde7e764740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55729e9f0a70 con 0x55729e9e7de0 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde7e764740 10 client.0 _ll_register_callbacks cb 0x55729e8ffe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e87d0 0x55729e9f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde73e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e87d0 0x55729e9f2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49428/0 (socket says 172.21.15.40:49428) 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde73e00640 1 -- 172.21.15.40:0/1583729865 learned_addr learned my addr 172.21.15.40:0/1583729865 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde79800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e7de0 0x55729e9f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde79800640 1 --2- 172.21.15.40:0/1583729865 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9ed940 0x55729e9f9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde73e00640 1 -- 172.21.15.40:0/1583729865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e7de0 msgr2=0x55729e9f1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde73e00640 1 --2- 172.21.15.40:0/1583729865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e7de0 0x55729e9f1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde73e00640 1 -- 172.21.15.40:0/1583729865 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9ed940 msgr2=0x55729e9f9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde73e00640 1 --2- 172.21.15.40:0/1583729865 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9ed940 0x55729e9f9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.032+0000 7fde73e00640 1 -- 172.21.15.40:0/1583729865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55729e8e2510 con 0x55729e9e87d0 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.033+0000 7fde73e00640 1 --2- 172.21.15.40:0/1583729865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e87d0 0x55729e9f2530 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fde60009860 tx=0x7fde60030570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa474e0b358eec08 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:37.036 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.033+0000 7fde5aa00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.033+0000 7fde5aa00640 1 -- 172.21.15.40:0/1583729865 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde60037040 con 0x55729e9e87d0 2024-04-27T00:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.033+0000 7fde5aa00640 1 -- 172.21.15.40:0/1583729865 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fde60004240 con 0x55729e9e87d0 2024-04-27T00:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.033+0000 7fde7e764740 1 -- 172.21.15.40:0/1583729865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde6c01c070 con 0x55729e9e87d0 2024-04-27T00:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.033+0000 7fde5aa00640 1 -- 172.21.15.40:0/1583729865 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fde6004e050 con 0x55729e9e87d0 2024-04-27T00:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.033+0000 7fde7e764740 10 client.15969 fetch_fsmap learned FSMap version 24 2024-04-27T00:10:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.033+0000 7fde7e764740 1 -- 172.21.15.40:0/1583729865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55729e65e210 con 0x55729e9e87d0 2024-04-27T00:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.034+0000 7fde5aa00640 1 -- 172.21.15.40:0/1583729865 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde6c01c070 con 0x55729e9e87d0 2024-04-27T00:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.034+0000 7fde5aa00640 1 -- 172.21.15.40:0/1583729865 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fde6003d550 con 0x55729e9e87d0 2024-04-27T00:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.034+0000 7fde7e764740 10 client.15969 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.034+0000 7fde7e764740 10 client.15969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.034+0000 7fde7e764740 20 client.15969 populate_metadata read hostname 'smithi040' 2024-04-27T00:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.034+0000 7fde7e764740 1 --2- 172.21.15.40:0/1583729865 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55729e9fdaf0 0x55729e9fff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.034+0000 7fde7e764740 4 client.15969 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.034+0000 7fde7e764740 1 -- 172.21.15.40:0/1583729865 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55729e959f90 con 0x55729e9fdaf0 2024-04-27T00:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.035+0000 7fde79800640 1 --2- 172.21.15.40:0/1583729865 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55729e9fdaf0 0x55729e9fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.035+0000 7fde79800640 1 --2- 172.21.15.40:0/1583729865 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55729e9fdaf0 0x55729e9fff00 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d69fdfc9d9be7b08 server_cookie=8054cdd0fdd77ef2 in_seq=0 out_seq=0 2024-04-27T00:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.035+0000 7fde5aa00640 10 client.15969 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde5aa00640 1 -- 172.21.15.40:0/1583729865 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55729e959f90 con 0x55729e9fdaf0 2024-04-27T00:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde5aa00640 10 client.15969 handle_command_reply: tid=0 2024-04-27T00:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde7e764740 1 client.15969 shutdown 2024-04-27T00:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde7e764740 1 -- 172.21.15.40:0/1583729865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e87d0 msgr2=0x55729e9f2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde7e764740 1 --2- 172.21.15.40:0/1583729865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e87d0 0x55729e9f2530 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fde60009860 tx=0x7fde60030570 comp rx=0 tx=0).stop 2024-04-27T00:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde7e764740 1 -- 172.21.15.40:0/1583729865 shutdown_connections 2024-04-27T00:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde7e764740 1 --2- 172.21.15.40:0/1583729865 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55729e9fdaf0 0x55729e9fff00 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde7e764740 1 --2- 172.21.15.40:0/1583729865 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9ed940 0x55729e9f9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde7e764740 1 --2- 172.21.15.40:0/1583729865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e87d0 0x55729e9f2530 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde7e764740 1 --2- 172.21.15.40:0/1583729865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e7de0 0x55729e9f1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde7e764740 1 -- 172.21.15.40:0/1583729865 >> 172.21.15.40:0/1583729865 conn(0x55729e9e3450 msgr2=0x55729e9e2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde5aa00640 0 client.15969 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde7e764740 1 -- 172.21.15.40:0/1583729865 shutdown_connections 2024-04-27T00:10:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.036+0000 7fde7e764740 1 -- 172.21.15.40:0/1583729865 wait complete. 2024-04-27T00:10:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.037+0000 7fde7e764740 20 client.15969 trim_cache size 0 max 16384 2024-04-27T00:10:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.053+0000 7fde7e764740 1 Processor -- start 2024-04-27T00:10:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.053+0000 7fde7e764740 1 -- start start 2024-04-27T00:10:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.053+0000 7fde7e764740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e7de0 0x55729e9e99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.053+0000 7fde7e764740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e87d0 0x55729e9e9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.053+0000 7fde7e764740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9ed940 0x55729e9f9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.053+0000 7fde7e764740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde741afee0 con 0x55729e9e7de0 2024-04-27T00:10:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.053+0000 7fde7e764740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fde741ad7b0 con 0x55729e9e87d0 2024-04-27T00:10:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.053+0000 7fde7e764740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde741ab600 con 0x55729e9ed940 2024-04-27T00:10:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.054+0000 7fde79800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e7de0 0x55729e9e99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.054+0000 7fde73e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e87d0 0x55729e9e9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.054+0000 7fde79800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e7de0 0x55729e9e99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49444/0 (socket says 172.21.15.40:49444) 2024-04-27T00:10:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.054+0000 7fde79800640 1 -- 172.21.15.40:0/2914496640 learned_addr learned my addr 172.21.15.40:0/2914496640 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.054+0000 7fde79800640 1 --2- 172.21.15.40:0/2914496640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9ed940 0x55729e9f9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.054+0000 7fde73e00640 1 -- 172.21.15.40:0/2914496640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9ed940 msgr2=0x55729e9f9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.054+0000 7fde73e00640 1 --2- 172.21.15.40:0/2914496640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9ed940 0x55729e9f9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.054+0000 7fde73e00640 1 -- 172.21.15.40:0/2914496640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e7de0 msgr2=0x55729e9e99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.054+0000 7fde73e00640 1 --2- 172.21.15.40:0/2914496640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e7de0 0x55729e9e99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.054+0000 7fde73e00640 1 -- 172.21.15.40:0/2914496640 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55729e72a620 con 0x55729e9e87d0 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.054+0000 7fde79800640 1 --2- 172.21.15.40:0/2914496640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9ed940 0x55729e9f9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.055+0000 7fde73e00640 1 --2- 172.21.15.40:0/2914496640 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e87d0 0x55729e9e9f30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fde6003db60 tx=0x7fde60030ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8581932d65c6e67e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.055+0000 7fde65600640 1 -- 172.21.15.40:0/2914496640 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde60037070 con 0x55729e9e87d0 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.055+0000 7fde65600640 1 -- 172.21.15.40:0/2914496640 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde60004040 con 0x55729e9e87d0 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.055+0000 7fde65600640 1 -- 172.21.15.40:0/2914496640 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde60000e60 con 0x55729e9e87d0 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.055+0000 7fde7e764740 1 -- 172.21.15.40:0/2914496640 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e87d0 msgr2=0x55729e9e9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.055+0000 7fde7e764740 1 --2- 172.21.15.40:0/2914496640 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e87d0 0x55729e9e9f30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fde6003db60 tx=0x7fde60030ed0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.056+0000 7fde7e764740 1 -- 172.21.15.40:0/2914496640 shutdown_connections 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.056+0000 7fde7e764740 1 --2- 172.21.15.40:0/2914496640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9ed940 0x55729e9f9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.056+0000 7fde7e764740 1 --2- 172.21.15.40:0/2914496640 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9e87d0 0x55729e9e9f30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.056+0000 7fde7e764740 1 --2- 172.21.15.40:0/2914496640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e7de0 0x55729e9e99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.056+0000 7fde7e764740 1 -- 172.21.15.40:0/2914496640 >> 172.21.15.40:0/2914496640 conn(0x55729e9e3450 msgr2=0x55729e9f1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.056+0000 7fde7e764740 1 -- 172.21.15.40:0/2914496640 shutdown_connections 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.056+0000 7fde7e764740 1 -- 172.21.15.40:0/2914496640 wait complete. 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.056+0000 7fde7e764740 1 -- start start 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.057+0000 7fde7e764740 1 Processor -- start 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.057+0000 7fde7e764740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e7de0 0x55729e9f6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.057+0000 7fde7e764740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e87d0 0x55729e9f6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.057+0000 7fde7e764740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9ed940 0x55729e9f7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.057+0000 7fde7e764740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde741abc70 con 0x55729e9e7de0 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.057+0000 7fde7e764740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fde741ad0e0 con 0x55729e9ed940 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.057+0000 7fde7e764740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde741ac340 con 0x55729e9e87d0 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.057+0000 7fde7e764740 10 client.0 _ll_register_callbacks cb 0x7fde6c009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.058+0000 7fde79800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e7de0 0x55729e9f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.058+0000 7fde79800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e7de0 0x55729e9f6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49450/0 (socket says 172.21.15.40:49450) 2024-04-27T00:10:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.058+0000 7fde79800640 1 -- 172.21.15.40:0/889712173 learned_addr learned my addr 172.21.15.40:0/889712173 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.058+0000 7fde73e00640 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e87d0 0x55729e9f6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.058+0000 7fde79800640 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9ed940 0x55729e9f7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.058+0000 7fde73e00640 1 -- 172.21.15.40:0/889712173 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9ed940 msgr2=0x55729e9f7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.058+0000 7fde73e00640 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9ed940 0x55729e9f7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.058+0000 7fde73e00640 1 -- 172.21.15.40:0/889712173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e7de0 msgr2=0x55729e9f6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.058+0000 7fde73e00640 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e7de0 0x55729e9f6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.058+0000 7fde73e00640 1 -- 172.21.15.40:0/889712173 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55729e906cb0 con 0x55729e9e87d0 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.058+0000 7fde73e00640 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e87d0 0x55729e9f6e70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fde6003dae0 tx=0x7fde6002a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f0c95a90a3c854 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.058+0000 7fde5aa00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.059+0000 7fde5aa00640 1 -- 172.21.15.40:0/889712173 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde60037070 con 0x55729e9e87d0 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.059+0000 7fde5aa00640 1 -- 172.21.15.40:0/889712173 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fde6003dbe0 con 0x55729e9e87d0 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.059+0000 7fde7e764740 1 -- 172.21.15.40:0/889712173 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde60004040 con 0x55729e9e87d0 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.059+0000 7fde5aa00640 1 -- 172.21.15.40:0/889712173 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fde6002a9c0 con 0x55729e9e87d0 2024-04-27T00:10:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.059+0000 7fde7e764740 10 client.26116 fetch_fsmap learned FSMap version 24 2024-04-27T00:10:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.059+0000 7fde7e764740 1 -- 172.21.15.40:0/889712173 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55729e9f95b0 con 0x55729e9e87d0 2024-04-27T00:10:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.059+0000 7fde5aa00640 1 -- 172.21.15.40:0/889712173 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde60004040 con 0x55729e9e87d0 2024-04-27T00:10:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.060+0000 7fde5aa00640 1 -- 172.21.15.40:0/889712173 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fde60032440 con 0x55729e9e87d0 2024-04-27T00:10:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.060+0000 7fde7e764740 10 client.26116 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:10:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.060+0000 7fde7e764740 10 client.26116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:10:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.060+0000 7fde7e764740 20 client.26116 populate_metadata read hostname 'smithi040' 2024-04-27T00:10:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.060+0000 7fde7e764740 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55729ea64490 0x55729ea668b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.060+0000 7fde7e764740 4 client.26116 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:10:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.060+0000 7fde7e764740 1 -- 172.21.15.40:0/889712173 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55729e953e00 con 0x55729ea64490 2024-04-27T00:10:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.060+0000 7fde79800640 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55729ea64490 0x55729ea668b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.060+0000 7fde79800640 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55729ea64490 0x55729ea668b0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3148932b624a82ed server_cookie=c5b0a9970bdca5eb in_seq=0 out_seq=0 2024-04-27T00:10:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.061+0000 7fde5aa00640 10 client.26116 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:37.077 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.076+0000 7fde5aa00640 1 -- 172.21.15.40:0/889712173 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55729e953e00 con 0x55729ea64490 2024-04-27T00:10:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.076+0000 7fde5aa00640 10 client.26116 handle_command_reply: tid=0 2024-04-27T00:10:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.076+0000 7fde7e764740 1 client.26116 shutdown 2024-04-27T00:10:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.077+0000 7fde7e764740 1 -- 172.21.15.40:0/889712173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e87d0 msgr2=0x55729e9f6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.077+0000 7fde7e764740 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e87d0 0x55729e9f6e70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fde6003dae0 tx=0x7fde6002a340 comp rx=0 tx=0).stop 2024-04-27T00:10:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.077+0000 7fde7e764740 1 -- 172.21.15.40:0/889712173 shutdown_connections 2024-04-27T00:10:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.077+0000 7fde7e764740 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55729ea64490 0x55729ea668b0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.077+0000 7fde7e764740 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55729e9ed940 0x55729e9f7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.077+0000 7fde7e764740 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55729e9e87d0 0x55729e9f6e70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.077+0000 7fde7e764740 1 --2- 172.21.15.40:0/889712173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55729e9e7de0 0x55729e9f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.077+0000 7fde7e764740 1 -- 172.21.15.40:0/889712173 >> 172.21.15.40:0/889712173 conn(0x55729e9e3450 msgr2=0x55729e9f1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.077+0000 7fde5aa00640 0 client.26116 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.077+0000 7fde7e764740 1 -- 172.21.15.40:0/889712173 shutdown_connections 2024-04-27T00:10:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.077+0000 7fde7e764740 1 -- 172.21.15.40:0/889712173 wait complete. 2024-04-27T00:10:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.077+0000 7fde7e764740 20 client.26116 trim_cache size 0 max 16384 2024-04-27T00:10:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:10:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:10:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:10:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:10:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.078+0000 7fde7a200640 1 -- 172.21.15.40:0/3773837219 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fde5c0a3c80 msgr2=0x7fde5c0a6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.078+0000 7fde7a200640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fde5c0a3c80 0x7fde5c0a6140 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x55729e9e8bd0 tx=0x7fde68000bc0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.078+0000 7fde7a200640 1 -- 172.21.15.40:0/3773837219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde74075390 msgr2=0x7fde7419ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.078+0000 7fde7a200640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde74075390 0x7fde7419ac90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fde6800e9f0 tx=0x7fde6800eeb0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.078+0000 7fde7a200640 1 -- 172.21.15.40:0/3773837219 shutdown_connections 2024-04-27T00:10:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.078+0000 7fde7a200640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fde7410a690 0x7fde741a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.078+0000 7fde7a200640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fde5c0a3c80 0x7fde5c0a6140 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.078+0000 7fde7a200640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde74076970 0x7fde7419b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.078+0000 7fde7a200640 1 --2- 172.21.15.40:0/3773837219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde74075390 0x7fde7419ac90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.078+0000 7fde7a200640 1 -- 172.21.15.40:0/3773837219 >> 172.21.15.40:0/3773837219 conn(0x7fde740fdd40 msgr2=0x7fde74106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:37.081 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.078+0000 7fde7a200640 1 -- 172.21.15.40:0/3773837219 shutdown_connections 2024-04-27T00:10:37.081 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:37.078+0000 7fde7a200640 1 -- 172.21.15.40:0/3773837219 wait complete. 2024-04-27T00:10:38.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:38 smithi040 ceph-mon[28095]: pgmap v680: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:10:38.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:38 smithi080 ceph-mon[32362]: pgmap v680: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:10:38.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:38 smithi103 ceph-mon[32952]: pgmap v680: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:10:40.149 INFO:tasks.workunit.client.0.smithi040.stdout:375/600: open fd = 3 2024-04-27T00:10:40.150 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.150 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.150 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.150 INFO:tasks.workunit.client.0.smithi040.stdout:376/600: open fd = 3 2024-04-27T00:10:40.150 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.150 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.150 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.150 INFO:tasks.workunit.client.0.smithi040.stdout:377/600: open fd = 3 2024-04-27T00:10:40.150 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.150 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.150 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:378/600: open fd = 3 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:379/600: open fd = 3 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:380/600: open fd = 3 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:381/600: open fd = 3 2024-04-27T00:10:40.151 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:382/600: open fd = 3 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:383/600: open fd = 3 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:384/600: open fd = 3 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.152 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:385/600: open fd = 3 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:386/600: open fd = 3 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:387/600: open fd = 3 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:388/600: open fd = 3 2024-04-27T00:10:40.153 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:389/600: open fd = 3 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:390/600: open fd = 3 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:391/600: open fd = 3 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.154 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:392/600: open fd = 3 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:393/600: open fd = 3 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:394/600: open fd = 3 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.155 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:395/600: open fd = 3 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:396/600: open fd = 3 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:397/600: open fd = 3 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:398/600: open fd = 3 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.156 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:399/600: open fd = 3 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:400/600: open fd = 3 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:401/600: open fd = 3 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.157 INFO:tasks.workunit.client.0.smithi040.stdout:402/600: open fd = 3 2024-04-27T00:10:40.158 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.158 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.158 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.158 INFO:tasks.workunit.client.0.smithi040.stdout:403/600: open fd = 3 2024-04-27T00:10:40.158 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.158 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.158 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.158 INFO:tasks.workunit.client.0.smithi040.stdout:404/600: open fd = 3 2024-04-27T00:10:40.158 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.158 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.158 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.159 INFO:tasks.workunit.client.0.smithi040.stdout:405/600: open fd = 3 2024-04-27T00:10:40.159 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.159 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.159 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.159 INFO:tasks.workunit.client.0.smithi040.stdout:406/600: open fd = 3 2024-04-27T00:10:40.159 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.159 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.159 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.159 INFO:tasks.workunit.client.0.smithi040.stdout:407/600: open fd = 3 2024-04-27T00:10:40.159 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:408/600: open fd = 3 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:409/600: open fd = 3 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:410/600: open fd = 3 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.160 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:411/600: open fd = 3 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:412/600: open fd = 3 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:413/600: open fd = 3 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:414/600: open fd = 3 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.161 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:415/600: open fd = 3 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:416/600: open fd = 3 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:417/600: open fd = 3 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:418/600: open fd = 3 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.162 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:419/600: open fd = 3 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:420/600: open fd = 3 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:421/600: open fd = 3 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:422/600: open fd = 3 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.163 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:423/600: open fd = 3 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:424/600: open fd = 3 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:425/600: open fd = 3 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:426/600: open fd = 3 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.164 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:427/600: open fd = 3 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:428/600: open fd = 3 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:429/600: open fd = 3 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:430/600: open fd = 3 2024-04-27T00:10:40.165 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:431/600: open fd = 3 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:432/600: open fd = 3 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:433/600: open fd = 3 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.166 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.167 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.167 INFO:tasks.workunit.client.0.smithi040.stdout:434/600: open fd = 3 2024-04-27T00:10:40.167 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.167 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.167 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.167 INFO:tasks.workunit.client.0.smithi040.stdout:435/600: open fd = 3 2024-04-27T00:10:40.167 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.167 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.167 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.167 INFO:tasks.workunit.client.0.smithi040.stdout:436/600: open fd = 3 2024-04-27T00:10:40.168 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.168 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:10:40.168 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:10:40.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:40 smithi040 ceph-mon[28095]: pgmap v681: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:10:40.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:40 smithi080 ceph-mon[32362]: pgmap v681: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:10:40.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:40 smithi103 ceph-mon[32952]: pgmap v681: 129 pgs: 129 active+clean; 2.0 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:10:42.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:42 smithi040 ceph-mon[28095]: pgmap v682: 129 pgs: 129 active+clean; 2.1 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:10:42.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:42 smithi080 ceph-mon[32362]: pgmap v682: 129 pgs: 129 active+clean; 2.1 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:10:42.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:42 smithi103 ceph-mon[32952]: pgmap v682: 129 pgs: 129 active+clean; 2.1 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:10:44.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:44 smithi040 ceph-mon[28095]: pgmap v683: 129 pgs: 129 active+clean; 2.1 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:10:44.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:44 smithi080 ceph-mon[32362]: pgmap v683: 129 pgs: 129 active+clean; 2.1 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:10:44.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:44 smithi103 ceph-mon[32952]: pgmap v683: 129 pgs: 129 active+clean; 2.1 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:10:46.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:46 smithi040 ceph-mon[28095]: pgmap v684: 129 pgs: 129 active+clean; 2.1 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:46.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:46 smithi080 ceph-mon[32362]: pgmap v684: 129 pgs: 129 active+clean; 2.1 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:46.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:46 smithi103 ceph-mon[32952]: pgmap v684: 129 pgs: 129 active+clean; 2.1 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:48.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:48 smithi080 ceph-mon[32362]: pgmap v685: 129 pgs: 129 active+clean; 2.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:10:48.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:48 smithi103 ceph-mon[32952]: pgmap v685: 129 pgs: 129 active+clean; 2.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:10:48.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:48 smithi040 ceph-mon[28095]: pgmap v685: 129 pgs: 129 active+clean; 2.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:10:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.230+0000 7fa61f200640 1 -- 172.21.15.40:0/1154732063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6201027c0 msgr2=0x7fa620102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.230+0000 7fa61f200640 1 --2- 172.21.15.40:0/1154732063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6201027c0 0x7fa620102be0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa610009900 tx=0x7fa61002a3f0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.231+0000 7fa61f200640 1 -- 172.21.15.40:0/1154732063 shutdown_connections 2024-04-27T00:10:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.231+0000 7fa61f200640 1 --2- 172.21.15.40:0/1154732063 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa620104370 0x7fa62010ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.231+0000 7fa61f200640 1 --2- 172.21.15.40:0/1154732063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6201039b0 0x7fa620103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.231+0000 7fa61f200640 1 --2- 172.21.15.40:0/1154732063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6201027c0 0x7fa620102be0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.231+0000 7fa61f200640 1 -- 172.21.15.40:0/1154732063 >> 172.21.15.40:0/1154732063 conn(0x7fa6200fdd30 msgr2=0x7fa620100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.231+0000 7fa61f200640 1 -- 172.21.15.40:0/1154732063 shutdown_connections 2024-04-27T00:10:50.232 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.231+0000 7fa61f200640 1 -- 172.21.15.40:0/1154732063 wait complete. 2024-04-27T00:10:50.233 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.232+0000 7fa61f200640 1 Processor -- start 2024-04-27T00:10:50.233 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.233+0000 7fa61f200640 1 -- start start 2024-04-27T00:10:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.233+0000 7fa61f200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6201027c0 0x7fa62019ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.233+0000 7fa61f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6201039b0 0x7fa62019b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.233+0000 7fa61f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa620104370 0x7fa6201a21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.233+0000 7fa61f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa62010e340 con 0x7fa6201039b0 2024-04-27T00:10:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.233+0000 7fa61f200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa62010e1c0 con 0x7fa6201027c0 2024-04-27T00:10:50.234 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.233+0000 7fa61f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa62010e4c0 con 0x7fa620104370 2024-04-27T00:10:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.234+0000 7fa61e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa620104370 0x7fa6201a21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.234+0000 7fa61de00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6201027c0 0x7fa62019ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.234+0000 7fa61e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa620104370 0x7fa6201a21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:45894/0 (socket says 172.21.15.40:45894) 2024-04-27T00:10:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.234+0000 7fa61e800640 1 -- 172.21.15.40:0/4248621331 learned_addr learned my addr 172.21.15.40:0/4248621331 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:50.235 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.234+0000 7fa61d400640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6201039b0 0x7fa62019b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.236 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.234+0000 7fa61e800640 1 -- 172.21.15.40:0/4248621331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6201027c0 msgr2=0x7fa62019ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:50.236 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.234+0000 7fa61e800640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6201027c0 0x7fa62019ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.236 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.234+0000 7fa61e800640 1 -- 172.21.15.40:0/4248621331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6201039b0 msgr2=0x7fa62019b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:50.236 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.234+0000 7fa61e800640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6201039b0 0x7fa62019b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.236 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.234+0000 7fa61e800640 1 -- 172.21.15.40:0/4248621331 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6201a2850 con 0x7fa620104370 2024-04-27T00:10:50.236 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.234+0000 7fa61d400640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6201039b0 0x7fa62019b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:10:50.236 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.235+0000 7fa61e800640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa620104370 0x7fa6201a21b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa61800eb10 tx=0x7fa61800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:50.236 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.235+0000 7fa616a00640 1 -- 172.21.15.40:0/4248621331 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa618004b20 con 0x7fa620104370 2024-04-27T00:10:50.236 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.235+0000 7fa616a00640 1 -- 172.21.15.40:0/4248621331 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa618004cc0 con 0x7fa620104370 2024-04-27T00:10:50.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.235+0000 7fa61f200640 1 -- 172.21.15.40:0/4248621331 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6201a2b40 con 0x7fa620104370 2024-04-27T00:10:50.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.235+0000 7fa616a00640 1 -- 172.21.15.40:0/4248621331 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa618010690 con 0x7fa620104370 2024-04-27T00:10:50.237 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.235+0000 7fa61f200640 1 -- 172.21.15.40:0/4248621331 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6201a3060 con 0x7fa620104370 2024-04-27T00:10:50.239 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.239+0000 7fa616a00640 1 -- 172.21.15.40:0/4248621331 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa618005020 con 0x7fa620104370 2024-04-27T00:10:50.239 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.239+0000 7fa627722740 1 Processor -- start 2024-04-27T00:10:50.240 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.239+0000 7fa627722740 1 -- start start 2024-04-27T00:10:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.240+0000 7fa627722740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5613123195f0 0x561312319a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.240+0000 7fa627722740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 0x56131231eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.240+0000 7fa627722740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56131231f150 0x5613123215e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.240+0000 7fa627722740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561312275380 con 0x56131231f150 2024-04-27T00:10:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.240+0000 7fa627722740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x561312211c30 con 0x561312319fe0 2024-04-27T00:10:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.240+0000 7fa627722740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561312271340 con 0x5613123195f0 2024-04-27T00:10:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.240+0000 7fa61d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5613123195f0 0x561312319a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.240+0000 7fa61de00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 0x56131231eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.240+0000 7fa61de00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 0x56131231eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:42656/0 (socket says 172.21.15.40:42656) 2024-04-27T00:10:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.240+0000 7fa61de00640 1 -- 172.21.15.40:0/2589014041 learned_addr learned my addr 172.21.15.40:0/2589014041 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:50.241 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.240+0000 7fa61d400640 1 --2- 172.21.15.40:0/2589014041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56131231f150 0x5613123215e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.241+0000 7fa61de00640 1 -- 172.21.15.40:0/2589014041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5613123195f0 msgr2=0x561312319a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.241+0000 7fa61de00640 1 --2- 172.21.15.40:0/2589014041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5613123195f0 0x561312319a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.241+0000 7fa61de00640 1 -- 172.21.15.40:0/2589014041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56131231f150 msgr2=0x5613123215e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.241+0000 7fa61de00640 1 --2- 172.21.15.40:0/2589014041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56131231f150 0x5613123215e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.241+0000 7fa616a00640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa6040a3c30 0x7fa6040a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.241+0000 7fa61de00640 1 -- 172.21.15.40:0/2589014041 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561312233940 con 0x561312319fe0 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.241+0000 7fa61d400640 1 --2- 172.21.15.40:0/2589014041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56131231f150 0x5613123215e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.241+0000 7fa616a00640 1 -- 172.21.15.40:0/4248621331 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fa6180d0ff0 con 0x7fa620104370 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.241+0000 7fa61e800640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa6040a3c30 0x7fa6040a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.241+0000 7fa61de00640 1 --2- 172.21.15.40:0/2589014041 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 0x56131231eae0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa610031500 tx=0x7fa610004170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0f0aafe8d150fe0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.242+0000 7fa60b000640 1 -- 172.21.15.40:0/2589014041 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610035030 con 0x561312319fe0 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.242+0000 7fa60b000640 1 -- 172.21.15.40:0/2589014041 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa610040070 con 0x561312319fe0 2024-04-27T00:10:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.242+0000 7fa60b000640 1 -- 172.21.15.40:0/2589014041 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610031a00 con 0x561312319fe0 2024-04-27T00:10:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.242+0000 7fa627722740 1 -- 172.21.15.40:0/2589014041 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 msgr2=0x56131231eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.242+0000 7fa61e800640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa6040a3c30 0x7fa6040a60f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa61809ca90 tx=0x7fa61800cf70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.242+0000 7fa627722740 1 --2- 172.21.15.40:0/2589014041 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 0x56131231eae0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa610031500 tx=0x7fa610004170 comp rx=0 tx=0).stop 2024-04-27T00:10:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.243+0000 7fa627722740 1 -- 172.21.15.40:0/2589014041 shutdown_connections 2024-04-27T00:10:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.243+0000 7fa627722740 1 --2- 172.21.15.40:0/2589014041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56131231f150 0x5613123215e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.243+0000 7fa627722740 1 --2- 172.21.15.40:0/2589014041 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 0x56131231eae0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.243+0000 7fa627722740 1 --2- 172.21.15.40:0/2589014041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5613123195f0 0x561312319a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.243+0000 7fa627722740 1 -- 172.21.15.40:0/2589014041 >> 172.21.15.40:0/2589014041 conn(0x56131230c450 msgr2=0x56131230c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.243+0000 7fa627722740 1 -- 172.21.15.40:0/2589014041 shutdown_connections 2024-04-27T00:10:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.243+0000 7fa627722740 1 -- 172.21.15.40:0/2589014041 wait complete. 2024-04-27T00:10:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.243+0000 7fa627722740 1 -- start start 2024-04-27T00:10:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.243+0000 7fa627722740 1 Processor -- start 2024-04-27T00:10:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa627722740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 0x5613123237d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa627722740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561312319fe0 0x561312323d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa627722740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56131231f150 0x56131232ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa627722740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5613123220a0 con 0x561312319fe0 2024-04-27T00:10:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa627722740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5613121fcc60 con 0x5613123195f0 2024-04-27T00:10:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa627722740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561312322220 con 0x56131231f150 2024-04-27T00:10:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa627722740 10 client.0 _ll_register_callbacks cb 0x561312228e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:10:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa61de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561312319fe0 0x561312323d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa61d400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 0x5613123237d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa61de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561312319fe0 0x561312323d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33594/0 (socket says 172.21.15.40:33594) 2024-04-27T00:10:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa61de00640 1 -- 172.21.15.40:0/179611308 learned_addr learned my addr 172.21.15.40:0/179611308 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa61d400640 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56131231f150 0x56131232ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa61de00640 1 -- 172.21.15.40:0/179611308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56131231f150 msgr2=0x56131232ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa61de00640 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56131231f150 0x56131232ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa61de00640 1 -- 172.21.15.40:0/179611308 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 msgr2=0x5613123237d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.244+0000 7fa61de00640 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 0x5613123237d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.245+0000 7fa61de00640 1 -- 172.21.15.40:0/179611308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56131220b510 con 0x561312319fe0 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.245+0000 7fa61d400640 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 0x5613123237d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.245+0000 7fa61de00640 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561312319fe0 0x561312323d10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa610005ab0 tx=0x7fa610004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd18f8661057625a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.245+0000 7fa5ff400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.245+0000 7fa5ff400640 1 -- 172.21.15.40:0/179611308 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610053060 con 0x561312319fe0 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.245+0000 7fa627722740 1 -- 172.21.15.40:0/179611308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa610040070 con 0x561312319fe0 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.245+0000 7fa5ff400640 1 -- 172.21.15.40:0/179611308 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fa610031c50 con 0x561312319fe0 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.245+0000 7fa5ff400640 1 -- 172.21.15.40:0/179611308 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa61005a050 con 0x561312319fe0 2024-04-27T00:10:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.245+0000 7fa627722740 10 client.15987 fetch_fsmap learned FSMap version 24 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.245+0000 7fa627722740 1 -- 172.21.15.40:0/179611308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561311f87210 con 0x561312319fe0 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.246+0000 7fa5ff400640 1 -- 172.21.15.40:0/179611308 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa610040070 con 0x561312319fe0 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.246+0000 7fa5ff400640 1 -- 172.21.15.40:0/179611308 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa61004d030 con 0x561312319fe0 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.246+0000 7fa627722740 10 client.15987 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.246+0000 7fa627722740 10 client.15987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.246+0000 7fa627722740 20 client.15987 populate_metadata read hostname 'smithi040' 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.246+0000 7fa627722740 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56131232eeb0 0x5613123312a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.246+0000 7fa627722740 4 client.15987 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.246+0000 7fa627722740 1 -- 172.21.15.40:0/179611308 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561312282f90 con 0x56131232eeb0 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.246+0000 7fa61d400640 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56131232eeb0 0x5613123312a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.247+0000 7fa61d400640 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56131232eeb0 0x5613123312a0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0f0f1dc517cdc server_cookie=1898c1f8476a9637 in_seq=0 out_seq=0 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.247+0000 7fa5ff400640 10 client.15987 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa5ff400640 1 -- 172.21.15.40:0/179611308 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561312282f90 con 0x56131232eeb0 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa5ff400640 10 client.15987 handle_command_reply: tid=0 2024-04-27T00:10:50.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa627722740 1 client.15987 shutdown 2024-04-27T00:10:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa627722740 1 -- 172.21.15.40:0/179611308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561312319fe0 msgr2=0x561312323d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa627722740 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561312319fe0 0x561312323d10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa610005ab0 tx=0x7fa610004590 comp rx=0 tx=0).stop 2024-04-27T00:10:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa627722740 1 -- 172.21.15.40:0/179611308 shutdown_connections 2024-04-27T00:10:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa627722740 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x56131232eeb0 0x5613123312a0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa627722740 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56131231f150 0x56131232ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa627722740 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561312319fe0 0x561312323d10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa627722740 1 --2- 172.21.15.40:0/179611308 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 0x5613123237d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa627722740 1 -- 172.21.15.40:0/179611308 >> 172.21.15.40:0/179611308 conn(0x56131230c450 msgr2=0x56131230b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa5ff400640 0 client.15987 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa627722740 1 -- 172.21.15.40:0/179611308 shutdown_connections 2024-04-27T00:10:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa627722740 1 -- 172.21.15.40:0/179611308 wait complete. 2024-04-27T00:10:50.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.248+0000 7fa627722740 20 client.15987 trim_cache size 0 max 16384 2024-04-27T00:10:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.265+0000 7fa627722740 1 Processor -- start 2024-04-27T00:10:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.265+0000 7fa627722740 1 -- start start 2024-04-27T00:10:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.265+0000 7fa627722740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 0x56131230adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.265+0000 7fa627722740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561312319fe0 0x56131232ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.265+0000 7fa627722740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56131231f150 0x56131230b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.265+0000 7fa627722740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa6201ab9e0 con 0x561312319fe0 2024-04-27T00:10:50.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.265+0000 7fa627722740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa6201a92b0 con 0x5613123195f0 2024-04-27T00:10:50.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.265+0000 7fa627722740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa6201a7100 con 0x56131231f150 2024-04-27T00:10:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.266+0000 7fa61d400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 0x56131230adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.266+0000 7fa61d400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 0x56131230adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:42684/0 (socket says 172.21.15.40:42684) 2024-04-27T00:10:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.266+0000 7fa61d400640 1 -- 172.21.15.40:0/3947888866 learned_addr learned my addr 172.21.15.40:0/3947888866 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.266+0000 7fa61de00640 1 --2- 172.21.15.40:0/3947888866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561312319fe0 0x56131232ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.266+0000 7fa61d400640 1 --2- 172.21.15.40:0/3947888866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56131231f150 0x56131230b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.266+0000 7fa61d400640 1 -- 172.21.15.40:0/3947888866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56131231f150 msgr2=0x56131230b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.266+0000 7fa61d400640 1 --2- 172.21.15.40:0/3947888866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56131231f150 0x56131230b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.266+0000 7fa61d400640 1 -- 172.21.15.40:0/3947888866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561312319fe0 msgr2=0x56131232ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.266+0000 7fa61d400640 1 --2- 172.21.15.40:0/3947888866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561312319fe0 0x56131232ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.266+0000 7fa61d400640 1 -- 172.21.15.40:0/3947888866 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561312053620 con 0x5613123195f0 2024-04-27T00:10:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.266+0000 7fa61d400640 1 --2- 172.21.15.40:0/3947888866 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 0x56131230adc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00c840 tx=0x7fa60c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=896a4a909ce3d7c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.267+0000 7fa60a600640 1 -- 172.21.15.40:0/3947888866 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c011ea0 con 0x5613123195f0 2024-04-27T00:10:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.267+0000 7fa60a600640 1 -- 172.21.15.40:0/3947888866 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa60c021900 con 0x5613123195f0 2024-04-27T00:10:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.267+0000 7fa60a600640 1 -- 172.21.15.40:0/3947888866 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c003cd0 con 0x5613123195f0 2024-04-27T00:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.267+0000 7fa627722740 1 -- 172.21.15.40:0/3947888866 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 msgr2=0x56131230adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.267+0000 7fa627722740 1 --2- 172.21.15.40:0/3947888866 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 0x56131230adc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00c840 tx=0x7fa60c0046c0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.268+0000 7fa627722740 1 -- 172.21.15.40:0/3947888866 shutdown_connections 2024-04-27T00:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.268+0000 7fa627722740 1 --2- 172.21.15.40:0/3947888866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56131231f150 0x56131230b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.268+0000 7fa627722740 1 --2- 172.21.15.40:0/3947888866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561312319fe0 0x56131232ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.268+0000 7fa627722740 1 --2- 172.21.15.40:0/3947888866 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5613123195f0 0x56131230adc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.268+0000 7fa627722740 1 -- 172.21.15.40:0/3947888866 >> 172.21.15.40:0/3947888866 conn(0x56131230c450 msgr2=0x56131231b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.268+0000 7fa627722740 1 -- 172.21.15.40:0/3947888866 shutdown_connections 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.268+0000 7fa627722740 1 -- 172.21.15.40:0/3947888866 wait complete. 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.268+0000 7fa627722740 1 -- start start 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.268+0000 7fa627722740 1 Processor -- start 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.269+0000 7fa627722740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5613123195f0 0x561312328360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.269+0000 7fa627722740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 0x5613123288a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.269+0000 7fa627722740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56131231f150 0x561312329810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.269+0000 7fa627722740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa6201a7770 con 0x56131231f150 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.269+0000 7fa627722740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa6201a8be0 con 0x561312319fe0 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.269+0000 7fa627722740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa6201a7e40 con 0x5613123195f0 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.269+0000 7fa627722740 10 client.0 _ll_register_callbacks cb 0x7fa60c00cab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.269+0000 7fa61de00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 0x5613123288a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.269+0000 7fa61de00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 0x5613123288a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:42692/0 (socket says 172.21.15.40:42692) 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.269+0000 7fa61de00640 1 -- 172.21.15.40:0/2304043187 learned_addr learned my addr 172.21.15.40:0/2304043187 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:10:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.269+0000 7fa61d400640 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56131231f150 0x561312329810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.269+0000 7fa61d400640 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5613123195f0 0x561312328360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.270+0000 7fa61de00640 1 -- 172.21.15.40:0/2304043187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5613123195f0 msgr2=0x561312328360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.270+0000 7fa61de00640 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5613123195f0 0x561312328360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.270+0000 7fa61de00640 1 -- 172.21.15.40:0/2304043187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56131231f150 msgr2=0x561312329810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.270+0000 7fa61de00640 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56131231f150 0x561312329810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.270+0000 7fa61de00640 1 -- 172.21.15.40:0/2304043187 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56131222fcb0 con 0x561312319fe0 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.270+0000 7fa61de00640 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 0x5613123288a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa6100096c0 tx=0x7fa610031c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e081909b3286fe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.270+0000 7fa5ff400640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.270+0000 7fa5ff400640 1 -- 172.21.15.40:0/2304043187 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610063060 con 0x561312319fe0 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.270+0000 7fa627722740 1 -- 172.21.15.40:0/2304043187 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa60c021900 con 0x561312319fe0 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.270+0000 7fa5ff400640 1 -- 172.21.15.40:0/2304043187 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fa610045b20 con 0x561312319fe0 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.271+0000 7fa5ff400640 1 -- 172.21.15.40:0/2304043187 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa61006a050 con 0x561312319fe0 2024-04-27T00:10:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.271+0000 7fa627722740 10 client.26105 fetch_fsmap learned FSMap version 24 2024-04-27T00:10:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.271+0000 7fa627722740 1 -- 172.21.15.40:0/2304043187 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56131232af40 con 0x561312319fe0 2024-04-27T00:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.271+0000 7fa5ff400640 1 -- 172.21.15.40:0/2304043187 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa60c021900 con 0x561312319fe0 2024-04-27T00:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.271+0000 7fa5ff400640 1 -- 172.21.15.40:0/2304043187 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa61004d070 con 0x561312319fe0 2024-04-27T00:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.271+0000 7fa627722740 10 client.26105 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.271+0000 7fa627722740 10 client.26105 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.271+0000 7fa627722740 20 client.26105 populate_metadata read hostname 'smithi040' 2024-04-27T00:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.271+0000 7fa627722740 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561312395950 0x561312397d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.271+0000 7fa627722740 4 client.26105 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.271+0000 7fa627722740 1 -- 172.21.15.40:0/2304043187 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56131227ce00 con 0x561312395950 2024-04-27T00:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.272+0000 7fa61d400640 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561312395950 0x561312397d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.272+0000 7fa61d400640 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561312395950 0x561312397d70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac1ae7c68464509b server_cookie=1ff343bab7fcf66 in_seq=0 out_seq=0 2024-04-27T00:10:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.272+0000 7fa5ff400640 10 client.26105 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.286+0000 7fa5ff400640 1 -- 172.21.15.40:0/2304043187 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56131227ce00 con 0x561312395950 2024-04-27T00:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.286+0000 7fa5ff400640 10 client.26105 handle_command_reply: tid=0 2024-04-27T00:10:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.286+0000 7fa627722740 1 client.26105 shutdown 2024-04-27T00:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.287+0000 7fa627722740 1 -- 172.21.15.40:0/2304043187 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 msgr2=0x5613123288a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.287+0000 7fa627722740 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 0x5613123288a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa6100096c0 tx=0x7fa610031c80 comp rx=0 tx=0).stop 2024-04-27T00:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.287+0000 7fa627722740 1 -- 172.21.15.40:0/2304043187 shutdown_connections 2024-04-27T00:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.287+0000 7fa627722740 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561312395950 0x561312397d70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.287+0000 7fa627722740 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56131231f150 0x561312329810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.287+0000 7fa627722740 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561312319fe0 0x5613123288a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.287+0000 7fa627722740 1 --2- 172.21.15.40:0/2304043187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5613123195f0 0x561312328360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.287+0000 7fa5ff400640 0 client.26105 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.287+0000 7fa627722740 1 -- 172.21.15.40:0/2304043187 >> 172.21.15.40:0/2304043187 conn(0x56131230c450 msgr2=0x56131230f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.287+0000 7fa627722740 1 -- 172.21.15.40:0/2304043187 shutdown_connections 2024-04-27T00:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.287+0000 7fa627722740 1 -- 172.21.15.40:0/2304043187 wait complete. 2024-04-27T00:10:50.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.287+0000 7fa627722740 20 client.26105 trim_cache size 0 max 16384 2024-04-27T00:10:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:10:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:10:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:10:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:10:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.288+0000 7fa61f200640 1 -- 172.21.15.40:0/4248621331 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa6040a3c30 msgr2=0x7fa6040a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.288+0000 7fa61f200640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa6040a3c30 0x7fa6040a60f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa61809ca90 tx=0x7fa61800cf70 comp rx=0 tx=0).stop 2024-04-27T00:10:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.288+0000 7fa61f200640 1 -- 172.21.15.40:0/4248621331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa620104370 msgr2=0x7fa6201a21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:10:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.288+0000 7fa61f200640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa620104370 0x7fa6201a21b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa61800eb10 tx=0x7fa61800c2d0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.288+0000 7fa61f200640 1 -- 172.21.15.40:0/4248621331 shutdown_connections 2024-04-27T00:10:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.288+0000 7fa61f200640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa6040a3c30 0x7fa6040a60f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.288+0000 7fa61f200640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa620104370 0x7fa6201a21b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.288+0000 7fa61f200640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6201039b0 0x7fa62019b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.288+0000 7fa61f200640 1 --2- 172.21.15.40:0/4248621331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa6201027c0 0x7fa62019ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:10:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.288+0000 7fa61f200640 1 -- 172.21.15.40:0/4248621331 >> 172.21.15.40:0/4248621331 conn(0x7fa6200fdd30 msgr2=0x7fa620074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:10:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.288+0000 7fa61f200640 1 -- 172.21.15.40:0/4248621331 shutdown_connections 2024-04-27T00:10:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:10:50.288+0000 7fa61f200640 1 -- 172.21.15.40:0/4248621331 wait complete. 2024-04-27T00:10:50.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:50 smithi080 ceph-mon[32362]: pgmap v686: 129 pgs: 129 active+clean; 2.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:50.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:50 smithi103 ceph-mon[32952]: pgmap v686: 129 pgs: 129 active+clean; 2.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:50.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:50 smithi040 ceph-mon[28095]: pgmap v686: 129 pgs: 129 active+clean; 2.1 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:52.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:52 smithi080 ceph-mon[32362]: pgmap v687: 129 pgs: 129 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-27T00:10:52.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:52 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:10:52.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:52 smithi103 ceph-mon[32952]: pgmap v687: 129 pgs: 129 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-27T00:10:52.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:52 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:10:52.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:52 smithi040 ceph-mon[28095]: pgmap v687: 129 pgs: 129 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-27T00:10:52.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:52 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:10:54.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:54 smithi080 ceph-mon[32362]: pgmap v688: 129 pgs: 129 active+clean; 2.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:10:54.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:54 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:10:54.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:54 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:10:54.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:54 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:10:54.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:54 smithi103 ceph-mon[32952]: pgmap v688: 129 pgs: 129 active+clean; 2.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:10:54.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:10:54.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:10:54.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:10:54.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:54 smithi040 ceph-mon[28095]: pgmap v688: 129 pgs: 129 active+clean; 2.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:10:54.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:10:54.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:10:54.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:10:56.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:56 smithi080 ceph-mon[32362]: pgmap v689: 129 pgs: 129 active+clean; 2.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:56.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:56 smithi103 ceph-mon[32952]: pgmap v689: 129 pgs: 129 active+clean; 2.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:56.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:56 smithi040 ceph-mon[28095]: pgmap v689: 129 pgs: 129 active+clean; 2.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:10:58.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:10:58 smithi080 ceph-mon[32362]: pgmap v690: 129 pgs: 129 active+clean; 2.3 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:10:58.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:10:58 smithi103 ceph-mon[32952]: pgmap v690: 129 pgs: 129 active+clean; 2.3 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:10:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:10:58 smithi040 ceph-mon[28095]: pgmap v690: 129 pgs: 129 active+clean; 2.3 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:11:00.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:00 smithi080 ceph-mon[32362]: pgmap v691: 129 pgs: 129 active+clean; 2.3 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:11:00.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:00 smithi103 ceph-mon[32952]: pgmap v691: 129 pgs: 129 active+clean; 2.3 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:11:00.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:00 smithi040 ceph-mon[28095]: pgmap v691: 129 pgs: 129 active+clean; 2.3 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:11:02.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:02 smithi080 ceph-mon[32362]: pgmap v692: 129 pgs: 129 active+clean; 2.3 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:11:02.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:02 smithi103 ceph-mon[32952]: pgmap v692: 129 pgs: 129 active+clean; 2.3 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:11:02.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:02 smithi040 ceph-mon[28095]: pgmap v692: 129 pgs: 129 active+clean; 2.3 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:11:03.438 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.438+0000 7fa57f800640 1 -- 172.21.15.40:0/218406662 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa578108fc0 msgr2=0x7fa5781093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.438+0000 7fa57f800640 1 --2- 172.21.15.40:0/218406662 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa578108fc0 0x7fa5781093e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa56c009900 tx=0x7fa56c02a3f0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.438+0000 7fa57f800640 1 -- 172.21.15.40:0/218406662 shutdown_connections 2024-04-27T00:11:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.438+0000 7fa57f800640 1 --2- 172.21.15.40:0/218406662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa57810ab70 0x7fa578111400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.438+0000 7fa57f800640 1 --2- 172.21.15.40:0/218406662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa57810a1b0 0x7fa57810a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.438+0000 7fa57f800640 1 --2- 172.21.15.40:0/218406662 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa578108fc0 0x7fa5781093e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.438+0000 7fa57f800640 1 -- 172.21.15.40:0/218406662 >> 172.21.15.40:0/218406662 conn(0x7fa578077f80 msgr2=0x7fa57807a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.438+0000 7fa57f800640 1 -- 172.21.15.40:0/218406662 shutdown_connections 2024-04-27T00:11:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.438+0000 7fa57f800640 1 -- 172.21.15.40:0/218406662 wait complete. 2024-04-27T00:11:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.439+0000 7fa57f800640 1 Processor -- start 2024-04-27T00:11:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.439+0000 7fa57f800640 1 -- start start 2024-04-27T00:11:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.440+0000 7fa57f800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa578108fc0 0x7fa57819f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.440+0000 7fa57f800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa57810a1b0 0x7fa57819f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.440+0000 7fa57f800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa57810ab70 0x7fa5781a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.440+0000 7fa57f800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa578114b50 con 0x7fa57810ab70 2024-04-27T00:11:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.440+0000 7fa57f800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa5781149d0 con 0x7fa57810a1b0 2024-04-27T00:11:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.440+0000 7fa57f800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa578114cd0 con 0x7fa578108fc0 2024-04-27T00:11:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.440+0000 7fa57d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa57810ab70 0x7fa5781a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.440+0000 7fa57ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa578108fc0 0x7fa57819f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.440+0000 7fa577e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa57810a1b0 0x7fa57819f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.440+0000 7fa57ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa578108fc0 0x7fa57819f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:53096/0 (socket says 172.21.15.40:53096) 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.440+0000 7fa57ce00640 1 -- 172.21.15.40:0/1404451083 learned_addr learned my addr 172.21.15.40:0/1404451083 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.441+0000 7fa57ce00640 1 -- 172.21.15.40:0/1404451083 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa57810a1b0 msgr2=0x7fa57819f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.441+0000 7fa57ce00640 1 --2- 172.21.15.40:0/1404451083 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa57810a1b0 0x7fa57819f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.441+0000 7fa57ce00640 1 -- 172.21.15.40:0/1404451083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa57810ab70 msgr2=0x7fa5781a6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.441+0000 7fa57ce00640 1 --2- 172.21.15.40:0/1404451083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa57810ab70 0x7fa5781a6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.441+0000 7fa57ce00640 1 -- 172.21.15.40:0/1404451083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5781a6d70 con 0x7fa578108fc0 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.441+0000 7fa57ce00640 1 --2- 172.21.15.40:0/1404451083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa578108fc0 0x7fa57819f150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa56c031500 tx=0x7fa56c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.441+0000 7fa575600640 1 -- 172.21.15.40:0/1404451083 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c03f070 con 0x7fa578108fc0 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.441+0000 7fa57f800640 1 -- 172.21.15.40:0/1404451083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5781a7000 con 0x7fa578108fc0 2024-04-27T00:11:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.441+0000 7fa575600640 1 -- 172.21.15.40:0/1404451083 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa56c031c50 con 0x7fa578108fc0 2024-04-27T00:11:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.441+0000 7fa575600640 1 -- 172.21.15.40:0/1404451083 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c035030 con 0x7fa578108fc0 2024-04-27T00:11:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.442+0000 7fa57f800640 1 -- 172.21.15.40:0/1404451083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5781a74f0 con 0x7fa578108fc0 2024-04-27T00:11:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.445+0000 7fa582622740 1 Processor -- start 2024-04-27T00:11:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.445+0000 7fa582622740 1 -- start start 2024-04-27T00:11:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.445+0000 7fa582622740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cbbd0 0x5562ee9cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.445+0000 7fa582622740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9d10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.445+0000 7fa582622740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9d3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.445+0000 7fa582622740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5562ee930380 con 0x5562ee9d1730 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.445+0000 7fa582622740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5562ee8ccc30 con 0x5562ee9cc5c0 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.445+0000 7fa582622740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5562ee92c340 con 0x5562ee9cbbd0 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.446+0000 7fa577e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9d3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.446+0000 7fa577e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9d3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37928/0 (socket says 172.21.15.40:37928) 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.446+0000 7fa577e00640 1 -- 172.21.15.40:0/2029980523 learned_addr learned my addr 172.21.15.40:0/2029980523 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.446+0000 7fa575600640 1 -- 172.21.15.40:0/1404451083 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa56c042070 con 0x7fa578108fc0 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.446+0000 7fa57d800640 1 --2- 172.21.15.40:0/2029980523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cbbd0 0x5562ee9cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.446+0000 7fa577e00640 1 -- 172.21.15.40:0/2029980523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cbbd0 msgr2=0x5562ee9cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.446+0000 7fa577e00640 1 --2- 172.21.15.40:0/2029980523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cbbd0 0x5562ee9cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.446+0000 7fa577e00640 1 -- 172.21.15.40:0/2029980523 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 msgr2=0x5562ee9d10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.446+0000 7fa577e00640 1 --2- 172.21.15.40:0/2029980523 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9d10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.446+0000 7fa577e00640 1 -- 172.21.15.40:0/2029980523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562ee8ee940 con 0x5562ee9d1730 2024-04-27T00:11:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.447+0000 7fa577e00640 1 --2- 172.21.15.40:0/2029980523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9d3bc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa568009ca0 tx=0x7fa568011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76dbb86a4ae39073 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.447+0000 7fa566000640 1 -- 172.21.15.40:0/2029980523 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568005220 con 0x5562ee9d1730 2024-04-27T00:11:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.447+0000 7fa566000640 1 -- 172.21.15.40:0/2029980523 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa568004c50 con 0x5562ee9d1730 2024-04-27T00:11:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.447+0000 7fa566000640 1 -- 172.21.15.40:0/2029980523 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568019710 con 0x5562ee9d1730 2024-04-27T00:11:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.447+0000 7fa575600640 1 --2- 172.21.15.40:0/1404451083 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa5600a3cf0 0x7fa5600a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.448+0000 7fa582622740 1 -- 172.21.15.40:0/2029980523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 msgr2=0x5562ee9d3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.448+0000 7fa582622740 1 --2- 172.21.15.40:0/2029980523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9d3bc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa568009ca0 tx=0x7fa568011810 comp rx=0 tx=0).stop 2024-04-27T00:11:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.448+0000 7fa575600640 1 -- 172.21.15.40:0/1404451083 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fa56c0f1020 con 0x7fa578108fc0 2024-04-27T00:11:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.448+0000 7fa57d800640 1 --2- 172.21.15.40:0/1404451083 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa5600a3cf0 0x7fa5600a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.448+0000 7fa582622740 1 -- 172.21.15.40:0/2029980523 shutdown_connections 2024-04-27T00:11:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.448+0000 7fa582622740 1 --2- 172.21.15.40:0/2029980523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9d3bc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.448+0000 7fa582622740 1 --2- 172.21.15.40:0/2029980523 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9d10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.448+0000 7fa582622740 1 --2- 172.21.15.40:0/2029980523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cbbd0 0x5562ee9cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.448+0000 7fa582622740 1 -- 172.21.15.40:0/2029980523 >> 172.21.15.40:0/2029980523 conn(0x5562ee9c7240 msgr2=0x5562ee9c7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:03.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.448+0000 7fa582622740 1 -- 172.21.15.40:0/2029980523 shutdown_connections 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.448+0000 7fa582622740 1 -- 172.21.15.40:0/2029980523 wait complete. 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.448+0000 7fa582622740 1 -- start start 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.449+0000 7fa57d800640 1 --2- 172.21.15.40:0/1404451083 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa5600a3cf0 0x7fa5600a61b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x5562ee9cce20 tx=0x7fa570009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.449+0000 7fa582622740 1 Processor -- start 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.449+0000 7fa582622740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cbbd0 0x5562ee9d5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.449+0000 7fa582622740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9d6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.449+0000 7fa582622740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9dd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.449+0000 7fa582622740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5562ee9d46e0 con 0x5562ee9d1730 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.449+0000 7fa582622740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5562ee8b7c60 con 0x5562ee9cc5c0 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.449+0000 7fa582622740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5562ee9d4860 con 0x5562ee9cbbd0 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.449+0000 7fa582622740 10 client.0 _ll_register_callbacks cb 0x5562ee8e3e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.449+0000 7fa57d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9dd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.449+0000 7fa57ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cbbd0 0x5562ee9d5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.449+0000 7fa577e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9d6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.450+0000 7fa57d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9dd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37940/0 (socket says 172.21.15.40:37940) 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.450+0000 7fa57ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cbbd0 0x5562ee9d5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:53106/0 (socket says 172.21.15.40:53106) 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.450+0000 7fa57d800640 1 -- 172.21.15.40:0/3430444345 learned_addr learned my addr 172.21.15.40:0/3430444345 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.450+0000 7fa577e00640 1 -- 172.21.15.40:0/3430444345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cbbd0 msgr2=0x5562ee9d5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.450+0000 7fa577e00640 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cbbd0 0x5562ee9d5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.450+0000 7fa577e00640 1 -- 172.21.15.40:0/3430444345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 msgr2=0x5562ee9dd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.450+0000 7fa577e00640 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9dd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.450+0000 7fa577e00640 1 -- 172.21.15.40:0/3430444345 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562ee8c6510 con 0x5562ee9cc5c0 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.450+0000 7fa57ce00640 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cbbd0 0x5562ee9d5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.450+0000 7fa57d800640 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9dd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.450+0000 7fa577e00640 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9d6380 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa568009c70 tx=0x7fa5680050b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fae1f6ca26af331 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.450+0000 7fa55aa00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:11:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.451+0000 7fa55aa00640 1 -- 172.21.15.40:0/3430444345 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568029070 con 0x5562ee9cc5c0 2024-04-27T00:11:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.451+0000 7fa582622740 1 -- 172.21.15.40:0/3430444345 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa568004c50 con 0x5562ee9cc5c0 2024-04-27T00:11:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.451+0000 7fa55aa00640 1 -- 172.21.15.40:0/3430444345 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fa568023840 con 0x5562ee9cc5c0 2024-04-27T00:11:03.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.451+0000 7fa55aa00640 1 -- 172.21.15.40:0/3430444345 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa568035050 con 0x5562ee9cc5c0 2024-04-27T00:11:03.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.451+0000 7fa582622740 10 client.26114 fetch_fsmap learned FSMap version 24 2024-04-27T00:11:03.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.451+0000 7fa582622740 1 -- 172.21.15.40:0/3430444345 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5562ee642210 con 0x5562ee9cc5c0 2024-04-27T00:11:03.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.452+0000 7fa55aa00640 1 -- 172.21.15.40:0/3430444345 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa568004c50 con 0x5562ee9cc5c0 2024-04-27T00:11:03.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.452+0000 7fa55aa00640 1 -- 172.21.15.40:0/3430444345 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa5680195f0 con 0x5562ee9cc5c0 2024-04-27T00:11:03.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.452+0000 7fa582622740 10 client.26114 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:11:03.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.452+0000 7fa582622740 10 client.26114 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:11:03.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.452+0000 7fa582622740 20 client.26114 populate_metadata read hostname 'smithi040' 2024-04-27T00:11:03.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.452+0000 7fa582622740 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562ee9e1620 0x5562ee9e3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.452+0000 7fa582622740 4 client.26114 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:11:03.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.452+0000 7fa582622740 1 -- 172.21.15.40:0/3430444345 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562ee93df90 con 0x5562ee9e1620 2024-04-27T00:11:03.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.452+0000 7fa57ce00640 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562ee9e1620 0x5562ee9e3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.453+0000 7fa57ce00640 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562ee9e1620 0x5562ee9e3a10 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f80b86639928591 server_cookie=80d1db93a00842d1 in_seq=0 out_seq=0 2024-04-27T00:11:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.453+0000 7fa55aa00640 10 client.26114 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa55aa00640 1 -- 172.21.15.40:0/3430444345 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5562ee93df90 con 0x5562ee9e1620 2024-04-27T00:11:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa55aa00640 10 client.26114 handle_command_reply: tid=0 2024-04-27T00:11:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa582622740 1 client.26114 shutdown 2024-04-27T00:11:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa582622740 1 -- 172.21.15.40:0/3430444345 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 msgr2=0x5562ee9d6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa582622740 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9d6380 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa568009c70 tx=0x7fa5680050b0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa582622740 1 -- 172.21.15.40:0/3430444345 shutdown_connections 2024-04-27T00:11:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa582622740 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562ee9e1620 0x5562ee9e3a10 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa582622740 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9dd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa55aa00640 0 client.26114 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:03.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa582622740 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9d6380 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa582622740 1 --2- 172.21.15.40:0/3430444345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cbbd0 0x5562ee9d5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa582622740 1 -- 172.21.15.40:0/3430444345 >> 172.21.15.40:0/3430444345 conn(0x5562ee9c7240 msgr2=0x5562ee9c6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:03.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa582622740 1 -- 172.21.15.40:0/3430444345 shutdown_connections 2024-04-27T00:11:03.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa582622740 1 -- 172.21.15.40:0/3430444345 wait complete. 2024-04-27T00:11:03.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.454+0000 7fa582622740 20 client.26114 trim_cache size 0 max 16384 2024-04-27T00:11:03.473 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa582622740 1 Processor -- start 2024-04-27T00:11:03.473 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa582622740 1 -- start start 2024-04-27T00:11:03.473 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa582622740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cbbd0 0x5562ee9cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.473 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa582622740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cc5c0 0x5562ee9cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.473 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa582622740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9dd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.473 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa582622740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa5781b42d0 con 0x5562ee9d1730 2024-04-27T00:11:03.473 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa582622740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa5781b1ba0 con 0x5562ee9cbbd0 2024-04-27T00:11:03.473 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa582622740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa5781af9f0 con 0x5562ee9cc5c0 2024-04-27T00:11:03.473 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa577e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cc5c0 0x5562ee9cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.473 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa577e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cc5c0 0x5562ee9cdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:53120/0 (socket says 172.21.15.40:53120) 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa577e00640 1 -- 172.21.15.40:0/567356976 learned_addr learned my addr 172.21.15.40:0/567356976 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa57ce00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cbbd0 0x5562ee9cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.471+0000 7fa57d800640 1 --2- 172.21.15.40:0/567356976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9dd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.472+0000 7fa577e00640 1 -- 172.21.15.40:0/567356976 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cbbd0 msgr2=0x5562ee9cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.472+0000 7fa577e00640 1 --2- 172.21.15.40:0/567356976 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cbbd0 0x5562ee9cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.472+0000 7fa577e00640 1 -- 172.21.15.40:0/567356976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 msgr2=0x5562ee9dd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.472+0000 7fa577e00640 1 --2- 172.21.15.40:0/567356976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9dd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.472+0000 7fa577e00640 1 -- 172.21.15.40:0/567356976 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562ee70e620 con 0x5562ee9cc5c0 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.472+0000 7fa57ce00640 1 --2- 172.21.15.40:0/567356976 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cbbd0 0x5562ee9cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.472+0000 7fa577e00640 1 --2- 172.21.15.40:0/567356976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cc5c0 0x5562ee9cdd80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa568011850 tx=0x7fa568005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20ce22fd777e3914 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.472+0000 7fa565600640 1 -- 172.21.15.40:0/567356976 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568032020 con 0x5562ee9cc5c0 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.472+0000 7fa565600640 1 -- 172.21.15.40:0/567356976 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa568005360 con 0x5562ee9cc5c0 2024-04-27T00:11:03.474 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.472+0000 7fa565600640 1 -- 172.21.15.40:0/567356976 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568029070 con 0x5562ee9cc5c0 2024-04-27T00:11:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.473+0000 7fa582622740 1 -- 172.21.15.40:0/567356976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cc5c0 msgr2=0x5562ee9cdd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.473+0000 7fa582622740 1 --2- 172.21.15.40:0/567356976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cc5c0 0x5562ee9cdd80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa568011850 tx=0x7fa568005970 comp rx=0 tx=0).stop 2024-04-27T00:11:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.473+0000 7fa582622740 1 -- 172.21.15.40:0/567356976 shutdown_connections 2024-04-27T00:11:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.473+0000 7fa582622740 1 --2- 172.21.15.40:0/567356976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9d1730 0x5562ee9dd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.473+0000 7fa582622740 1 --2- 172.21.15.40:0/567356976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9cc5c0 0x5562ee9cdd80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.473+0000 7fa582622740 1 --2- 172.21.15.40:0/567356976 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cbbd0 0x5562ee9cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.473+0000 7fa582622740 1 -- 172.21.15.40:0/567356976 >> 172.21.15.40:0/567356976 conn(0x5562ee9c7240 msgr2=0x5562ee9d5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.473+0000 7fa582622740 1 -- 172.21.15.40:0/567356976 shutdown_connections 2024-04-27T00:11:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.473+0000 7fa582622740 1 -- 172.21.15.40:0/567356976 wait complete. 2024-04-27T00:11:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.474+0000 7fa582622740 1 -- start start 2024-04-27T00:11:03.478 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.474+0000 7fa582622740 1 Processor -- start 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.474+0000 7fa582622740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9cbbd0 0x5562ee9da4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.474+0000 7fa582622740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9da9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.474+0000 7fa582622740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9d1730 0x5562ee9db9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.474+0000 7fa582622740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa5781b0060 con 0x5562ee9cbbd0 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.474+0000 7fa582622740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa5781b14d0 con 0x5562ee9cc5c0 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.474+0000 7fa582622740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa5781b0730 con 0x5562ee9d1730 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.474+0000 7fa582622740 10 client.0 _ll_register_callbacks cb 0x7fa570009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.474+0000 7fa57ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9cbbd0 0x5562ee9da4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.474+0000 7fa577e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9da9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.475+0000 7fa57ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9cbbd0 0x5562ee9da4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37952/0 (socket says 172.21.15.40:37952) 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.475+0000 7fa57ce00640 1 -- 172.21.15.40:0/2397553406 learned_addr learned my addr 172.21.15.40:0/2397553406 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.475+0000 7fa57d800640 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9d1730 0x5562ee9db9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.475+0000 7fa577e00640 1 -- 172.21.15.40:0/2397553406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9d1730 msgr2=0x5562ee9db9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.475+0000 7fa577e00640 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9d1730 0x5562ee9db9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.475+0000 7fa577e00640 1 -- 172.21.15.40:0/2397553406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9cbbd0 msgr2=0x5562ee9da4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.475+0000 7fa577e00640 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9cbbd0 0x5562ee9da4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.475+0000 7fa577e00640 1 -- 172.21.15.40:0/2397553406 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562ee8eacb0 con 0x5562ee9cc5c0 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.475+0000 7fa57ce00640 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9cbbd0 0x5562ee9da4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.475+0000 7fa577e00640 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9da9e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa568019c00 tx=0x7fa568009f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c33cd8da9291d76a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.475+0000 7fa55aa00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.476+0000 7fa55aa00640 1 -- 172.21.15.40:0/2397553406 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568032030 con 0x5562ee9cc5c0 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.476+0000 7fa55aa00640 1 -- 172.21.15.40:0/2397553406 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fa568019430 con 0x5562ee9cc5c0 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.476+0000 7fa582622740 1 -- 172.21.15.40:0/2397553406 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa568005360 con 0x5562ee9cc5c0 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.476+0000 7fa55aa00640 1 -- 172.21.15.40:0/2397553406 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa568029040 con 0x5562ee9cc5c0 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.476+0000 7fa582622740 10 client.26120 fetch_fsmap learned FSMap version 24 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.476+0000 7fa582622740 1 -- 172.21.15.40:0/2397553406 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5562ee9dd120 con 0x5562ee9cc5c0 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.476+0000 7fa55aa00640 1 -- 172.21.15.40:0/2397553406 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa568005360 con 0x5562ee9cc5c0 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.476+0000 7fa55aa00640 1 -- 172.21.15.40:0/2397553406 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa568023640 con 0x5562ee9cc5c0 2024-04-27T00:11:03.480 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.476+0000 7fa582622740 10 client.26120 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:11:03.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.476+0000 7fa582622740 10 client.26120 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:11:03.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.477+0000 7fa582622740 20 client.26120 populate_metadata read hostname 'smithi040' 2024-04-27T00:11:03.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.477+0000 7fa582622740 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562eea47dc0 0x5562eea4a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:03.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.477+0000 7fa582622740 4 client.26120 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:11:03.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.477+0000 7fa582622740 1 -- 172.21.15.40:0/2397553406 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5562ee937e00 con 0x5562eea47dc0 2024-04-27T00:11:03.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.477+0000 7fa57ce00640 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562eea47dc0 0x5562eea4a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:03.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.477+0000 7fa57ce00640 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562eea47dc0 0x5562eea4a1b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=faece54541d0aa1d server_cookie=abb3d5a8285377e0 in_seq=0 out_seq=0 2024-04-27T00:11:03.481 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.477+0000 7fa55aa00640 10 client.26120 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:03.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.492+0000 7fa55aa00640 1 -- 172.21.15.40:0/2397553406 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5562ee937e00 con 0x5562eea47dc0 2024-04-27T00:11:03.493 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.492+0000 7fa55aa00640 10 client.26120 handle_command_reply: tid=0 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.492+0000 7fa582622740 1 client.26120 shutdown 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.492+0000 7fa582622740 1 -- 172.21.15.40:0/2397553406 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 msgr2=0x5562ee9da9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.492+0000 7fa582622740 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9da9e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa568019c00 tx=0x7fa568009f10 comp rx=0 tx=0).stop 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.492+0000 7fa582622740 1 -- 172.21.15.40:0/2397553406 shutdown_connections 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.492+0000 7fa582622740 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5562eea47dc0 0x5562eea4a1b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.492+0000 7fa582622740 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5562ee9d1730 0x5562ee9db9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.492+0000 7fa55aa00640 0 client.26120 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.492+0000 7fa582622740 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5562ee9cc5c0 0x5562ee9da9e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.492+0000 7fa582622740 1 --2- 172.21.15.40:0/2397553406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5562ee9cbbd0 0x5562ee9da4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.492+0000 7fa582622740 1 -- 172.21.15.40:0/2397553406 >> 172.21.15.40:0/2397553406 conn(0x5562ee9c7240 msgr2=0x5562ee9cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.493+0000 7fa582622740 1 -- 172.21.15.40:0/2397553406 shutdown_connections 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.493+0000 7fa582622740 1 -- 172.21.15.40:0/2397553406 wait complete. 2024-04-27T00:11:03.494 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.493+0000 7fa582622740 20 client.26120 trim_cache size 0 max 16384 2024-04-27T00:11:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:11:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:11:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:11:03.495 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:11:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.494+0000 7fa57f800640 1 -- 172.21.15.40:0/1404451083 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa5600a3cf0 msgr2=0x7fa5600a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.494+0000 7fa57f800640 1 --2- 172.21.15.40:0/1404451083 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa5600a3cf0 0x7fa5600a61b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x5562ee9cce20 tx=0x7fa570009500 comp rx=0 tx=0).stop 2024-04-27T00:11:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.494+0000 7fa57f800640 1 -- 172.21.15.40:0/1404451083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa578108fc0 msgr2=0x7fa57819f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.494+0000 7fa57f800640 1 --2- 172.21.15.40:0/1404451083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa578108fc0 0x7fa57819f150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa56c031500 tx=0x7fa56c003c10 comp rx=0 tx=0).stop 2024-04-27T00:11:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.494+0000 7fa57f800640 1 -- 172.21.15.40:0/1404451083 shutdown_connections 2024-04-27T00:11:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.494+0000 7fa57f800640 1 --2- 172.21.15.40:0/1404451083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa57810ab70 0x7fa5781a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.494+0000 7fa57f800640 1 --2- 172.21.15.40:0/1404451083 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa57810a1b0 0x7fa57819f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.494+0000 7fa57f800640 1 --2- 172.21.15.40:0/1404451083 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa5600a3cf0 0x7fa5600a61b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.496 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.494+0000 7fa57f800640 1 --2- 172.21.15.40:0/1404451083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa578108fc0 0x7fa57819f150 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.494+0000 7fa57f800640 1 -- 172.21.15.40:0/1404451083 >> 172.21.15.40:0/1404451083 conn(0x7fa578077f80 msgr2=0x7fa57810d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.494+0000 7fa57f800640 1 -- 172.21.15.40:0/1404451083 shutdown_connections 2024-04-27T00:11:03.497 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:03.494+0000 7fa57f800640 1 -- 172.21.15.40:0/1404451083 wait complete. 2024-04-27T00:11:04.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:04 smithi080 ceph-mon[32362]: pgmap v693: 129 pgs: 129 active+clean; 2.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:11:04.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:04 smithi103 ceph-mon[32952]: pgmap v693: 129 pgs: 129 active+clean; 2.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:11:04.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:04 smithi040 ceph-mon[28095]: pgmap v693: 129 pgs: 129 active+clean; 2.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:11:06.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:06 smithi080 ceph-mon[32362]: pgmap v694: 129 pgs: 129 active+clean; 2.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:11:06.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:06 smithi103 ceph-mon[32952]: pgmap v694: 129 pgs: 129 active+clean; 2.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:11:06.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:06 smithi040 ceph-mon[28095]: pgmap v694: 129 pgs: 129 active+clean; 2.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:11:08.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:08 smithi080 ceph-mon[32362]: pgmap v695: 129 pgs: 129 active+clean; 2.4 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:11:08.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:08 smithi103 ceph-mon[32952]: pgmap v695: 129 pgs: 129 active+clean; 2.4 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:11:08.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:08 smithi040 ceph-mon[28095]: pgmap v695: 129 pgs: 129 active+clean; 2.4 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:11:10.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:10 smithi080 ceph-mon[32362]: pgmap v696: 129 pgs: 129 active+clean; 2.4 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:10.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:10 smithi103 ceph-mon[32952]: pgmap v696: 129 pgs: 129 active+clean; 2.4 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:10.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:10 smithi040 ceph-mon[28095]: pgmap v696: 129 pgs: 129 active+clean; 2.4 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:12.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:12 smithi080 ceph-mon[32362]: pgmap v697: 129 pgs: 129 active+clean; 2.5 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:11:12.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:12 smithi103 ceph-mon[32952]: pgmap v697: 129 pgs: 129 active+clean; 2.5 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:11:12.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:12 smithi040 ceph-mon[28095]: pgmap v697: 129 pgs: 129 active+clean; 2.5 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:11:14.550 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:14 smithi080 ceph-mon[32362]: pgmap v698: 129 pgs: 129 active+clean; 2.5 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:11:14.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:14 smithi103 ceph-mon[32952]: pgmap v698: 129 pgs: 129 active+clean; 2.5 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:11:14.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:14 smithi040 ceph-mon[28095]: pgmap v698: 129 pgs: 129 active+clean; 2.5 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:11:16.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:16 smithi080 ceph-mon[32362]: pgmap v699: 129 pgs: 129 active+clean; 2.5 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:16.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:16 smithi103 ceph-mon[32952]: pgmap v699: 129 pgs: 129 active+clean; 2.5 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.643+0000 7fd0ef200640 1 -- 172.21.15.40:0/3350438137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0f01049d0 msgr2=0x7fd0f0104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.643+0000 7fd0ef200640 1 --2- 172.21.15.40:0/3350438137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0f01049d0 0x7fd0f0104df0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0009900 tx=0x7fd0e002a400 comp rx=0 tx=0).stop 2024-04-27T00:11:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.643+0000 7fd0ef200640 1 -- 172.21.15.40:0/3350438137 shutdown_connections 2024-04-27T00:11:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.643+0000 7fd0ef200640 1 --2- 172.21.15.40:0/3350438137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0f0106580 0x7fd0f010ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.643+0000 7fd0ef200640 1 --2- 172.21.15.40:0/3350438137 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd0f0105bc0 0x7fd0f0106040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.643+0000 7fd0ef200640 1 --2- 172.21.15.40:0/3350438137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0f01049d0 0x7fd0f0104df0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.643+0000 7fd0ef200640 1 -- 172.21.15.40:0/3350438137 >> 172.21.15.40:0/3350438137 conn(0x7fd0f00fff40 msgr2=0x7fd0f0102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.643+0000 7fd0ef200640 1 -- 172.21.15.40:0/3350438137 shutdown_connections 2024-04-27T00:11:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.643+0000 7fd0ef200640 1 -- 172.21.15.40:0/3350438137 wait complete. 2024-04-27T00:11:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.644+0000 7fd0ef200640 1 Processor -- start 2024-04-27T00:11:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.644+0000 7fd0ef200640 1 -- start start 2024-04-27T00:11:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.644+0000 7fd0ef200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0f01049d0 0x7fd0f019abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.644+0000 7fd0ef200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd0f0105bc0 0x7fd0f019b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.644+0000 7fd0ef200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0f0106580 0x7fd0f01a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.644+0000 7fd0ef200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f0110520 con 0x7fd0f0106580 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.644+0000 7fd0ef200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01103a0 con 0x7fd0f0105bc0 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.644+0000 7fd0ef200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01106a0 con 0x7fd0f01049d0 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.645+0000 7fd0e5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd0f0105bc0 0x7fd0f019b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.645+0000 7fd0e5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd0f0105bc0 0x7fd0f019b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:34196/0 (socket says 172.21.15.40:34196) 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.645+0000 7fd0e5400640 1 -- 172.21.15.40:0/4251997416 learned_addr learned my addr 172.21.15.40:0/4251997416 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.645+0000 7fd0ee800640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0f0106580 0x7fd0f01a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.645+0000 7fd0ede00640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0f01049d0 0x7fd0f019abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.645+0000 7fd0e5400640 1 -- 172.21.15.40:0/4251997416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0f01049d0 msgr2=0x7fd0f019abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.645+0000 7fd0e5400640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0f01049d0 0x7fd0f019abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.647 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.645+0000 7fd0e5400640 1 -- 172.21.15.40:0/4251997416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0f0106580 msgr2=0x7fd0f01a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:16.648 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.645+0000 7fd0e5400640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0f0106580 0x7fd0f01a2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.648 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.645+0000 7fd0e5400640 1 -- 172.21.15.40:0/4251997416 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f01a27e0 con 0x7fd0f0105bc0 2024-04-27T00:11:16.648 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.646+0000 7fd0e5400640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd0f0105bc0 0x7fd0f019b100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800e9f0 tx=0x7fd0d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:16.648 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.646+0000 7fd0e7400640 1 -- 172.21.15.40:0/4251997416 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d800cd80 con 0x7fd0f0105bc0 2024-04-27T00:11:16.648 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.646+0000 7fd0e7400640 1 -- 172.21.15.40:0/4251997416 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0d80108e0 con 0x7fd0f0105bc0 2024-04-27T00:11:16.648 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.646+0000 7fd0ef200640 1 -- 172.21.15.40:0/4251997416 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f01a2ad0 con 0x7fd0f0105bc0 2024-04-27T00:11:16.648 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.646+0000 7fd0e7400640 1 -- 172.21.15.40:0/4251997416 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8018980 con 0x7fd0f0105bc0 2024-04-27T00:11:16.648 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.646+0000 7fd0ef200640 1 -- 172.21.15.40:0/4251997416 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f01a3020 con 0x7fd0f0105bc0 2024-04-27T00:11:16.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.649+0000 7fd0f7983740 1 Processor -- start 2024-04-27T00:11:16.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.649+0000 7fd0f7983740 1 -- start start 2024-04-27T00:11:16.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.649+0000 7fd0e7400640 1 -- 172.21.15.40:0/4251997416 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd0d8010430 con 0x7fd0f0105bc0 2024-04-27T00:11:16.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0f7983740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9bede0 0x561aaa9bf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0f7983740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 0x561aaa9c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0f7983740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9c4940 0x561aaa9c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0f7983740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561aaa923380 con 0x561aaa9bf7d0 2024-04-27T00:11:16.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0f7983740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x561aaa8bfc30 con 0x561aaa9c4940 2024-04-27T00:11:16.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0f7983740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561aaa91f340 con 0x561aaa9bede0 2024-04-27T00:11:16.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0ede00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 0x561aaa9c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0ee800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9bede0 0x561aaa9bf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.652 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0ede00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 0x561aaa9c42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60720/0 (socket says 172.21.15.40:60720) 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0ede00640 1 -- 172.21.15.40:0/3406770975 learned_addr learned my addr 172.21.15.40:0/3406770975 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0e5400640 1 --2- 172.21.15.40:0/3406770975 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9c4940 0x561aaa9c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0ee800640 1 -- 172.21.15.40:0/3406770975 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9c4940 msgr2=0x561aaa9c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0ee800640 1 --2- 172.21.15.40:0/3406770975 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9c4940 0x561aaa9c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0ee800640 1 -- 172.21.15.40:0/3406770975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 msgr2=0x561aaa9c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0ee800640 1 --2- 172.21.15.40:0/3406770975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 0x561aaa9c42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.650+0000 7fd0ee800640 1 -- 172.21.15.40:0/3406770975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aaa8e1940 con 0x561aaa9bede0 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.651+0000 7fd0e5400640 1 --2- 172.21.15.40:0/3406770975 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9c4940 0x561aaa9c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.651+0000 7fd0ee800640 1 --2- 172.21.15.40:0/3406770975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9bede0 0x561aaa9bf200 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8009f00 tx=0x7fd0e8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74c25919b3c7cf87 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.651+0000 7fd0e7400640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd0d40a3c30 0x7fd0d40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.651+0000 7fd0df000640 1 -- 172.21.15.40:0/3406770975 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8004d20 con 0x561aaa9bede0 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.651+0000 7fd0e7400640 1 -- 172.21.15.40:0/4251997416 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fd0d80d0c40 con 0x7fd0f0105bc0 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.651+0000 7fd0df000640 1 -- 172.21.15.40:0/3406770975 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0e8004030 con 0x561aaa9bede0 2024-04-27T00:11:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.651+0000 7fd0df000640 1 -- 172.21.15.40:0/3406770975 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8005260 con 0x561aaa9bede0 2024-04-27T00:11:16.654 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.651+0000 7fd0ede00640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd0d40a3c30 0x7fd0d40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.652+0000 7fd0f7983740 1 -- 172.21.15.40:0/3406770975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9bede0 msgr2=0x561aaa9bf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.652+0000 7fd0f7983740 1 --2- 172.21.15.40:0/3406770975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9bede0 0x561aaa9bf200 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8009f00 tx=0x7fd0e8011810 comp rx=0 tx=0).stop 2024-04-27T00:11:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.652+0000 7fd0f7983740 1 -- 172.21.15.40:0/3406770975 shutdown_connections 2024-04-27T00:11:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.652+0000 7fd0f7983740 1 --2- 172.21.15.40:0/3406770975 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9c4940 0x561aaa9c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.652+0000 7fd0f7983740 1 --2- 172.21.15.40:0/3406770975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 0x561aaa9c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.652+0000 7fd0f7983740 1 --2- 172.21.15.40:0/3406770975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9bede0 0x561aaa9bf200 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.652+0000 7fd0f7983740 1 -- 172.21.15.40:0/3406770975 >> 172.21.15.40:0/3406770975 conn(0x561aaa9ba450 msgr2=0x561aaa9ba850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.652+0000 7fd0ede00640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd0d40a3c30 0x7fd0d40a60f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x561aaa9c0240 tx=0x7fd0e0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.652+0000 7fd0f7983740 1 -- 172.21.15.40:0/3406770975 shutdown_connections 2024-04-27T00:11:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.652+0000 7fd0f7983740 1 -- 172.21.15.40:0/3406770975 wait complete. 2024-04-27T00:11:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.653+0000 7fd0f7983740 1 -- start start 2024-04-27T00:11:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.653+0000 7fd0f7983740 1 Processor -- start 2024-04-27T00:11:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.653+0000 7fd0f7983740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9bede0 0x561aaa9c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.653+0000 7fd0f7983740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bf7d0 0x561aaa9c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.653+0000 7fd0f7983740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9c4940 0x561aaa9d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.653+0000 7fd0f7983740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561aaa9c78f0 con 0x561aaa9c4940 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.653+0000 7fd0f7983740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x561aaa8aac60 con 0x561aaa9bf7d0 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.653+0000 7fd0f7983740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561aaa9c7a70 con 0x561aaa9bede0 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.653+0000 7fd0f7983740 10 client.0 _ll_register_callbacks cb 0x561aaa8d6e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.654+0000 7fd0ee800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9c4940 0x561aaa9d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.654+0000 7fd0ee800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9c4940 0x561aaa9d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60722/0 (socket says 172.21.15.40:60722) 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.654+0000 7fd0ee800640 1 -- 172.21.15.40:0/2549366321 learned_addr learned my addr 172.21.15.40:0/2549366321 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.654+0000 7fd0ede00640 1 --2- 172.21.15.40:0/2549366321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9bede0 0x561aaa9c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.654+0000 7fd0e5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bf7d0 0x561aaa9c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.654+0000 7fd0ee800640 1 -- 172.21.15.40:0/2549366321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9bede0 msgr2=0x561aaa9c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:16.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.654+0000 7fd0ee800640 1 --2- 172.21.15.40:0/2549366321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9bede0 0x561aaa9c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.654+0000 7fd0ee800640 1 -- 172.21.15.40:0/2549366321 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bf7d0 msgr2=0x561aaa9c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.654+0000 7fd0ee800640 1 --2- 172.21.15.40:0/2549366321 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bf7d0 0x561aaa9c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.654+0000 7fd0ee800640 1 -- 172.21.15.40:0/2549366321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aaa8b9510 con 0x561aaa9c4940 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.655+0000 7fd0ee800640 1 --2- 172.21.15.40:0/2549366321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9c4940 0x561aaa9d0520 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8000f80 tx=0x7fd0e8009cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45f3a76de71cbc80 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.655+0000 7fd0d3400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.655+0000 7fd0d3400640 1 -- 172.21.15.40:0/2549366321 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8027020 con 0x561aaa9c4940 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.655+0000 7fd0d3400640 1 -- 172.21.15.40:0/2549366321 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fd0e8003a30 con 0x561aaa9c4940 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.655+0000 7fd0f7983740 1 -- 172.21.15.40:0/2549366321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0e8004030 con 0x561aaa9c4940 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.655+0000 7fd0d3400640 1 -- 172.21.15.40:0/2549366321 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd0e8032050 con 0x561aaa9c4940 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.655+0000 7fd0f7983740 10 client.16023 fetch_fsmap learned FSMap version 24 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.655+0000 7fd0f7983740 1 -- 172.21.15.40:0/2549366321 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561aaa635210 con 0x561aaa9c4940 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.655+0000 7fd0d3400640 1 -- 172.21.15.40:0/2549366321 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0e8004030 con 0x561aaa9c4940 2024-04-27T00:11:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.656+0000 7fd0d3400640 1 -- 172.21.15.40:0/2549366321 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd0e8018070 con 0x561aaa9c4940 2024-04-27T00:11:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.656+0000 7fd0f7983740 10 client.16023 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:11:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.656+0000 7fd0f7983740 10 client.16023 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:11:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.656+0000 7fd0f7983740 20 client.16023 populate_metadata read hostname 'smithi040' 2024-04-27T00:11:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.656+0000 7fd0f7983740 1 --2- 172.21.15.40:0/2549366321 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561aaa9d4900 0x561aaa9d6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.656+0000 7fd0f7983740 4 client.16023 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:11:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.656+0000 7fd0f7983740 1 -- 172.21.15.40:0/2549366321 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561aaa930f90 con 0x561aaa9d4900 2024-04-27T00:11:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.656+0000 7fd0ede00640 1 --2- 172.21.15.40:0/2549366321 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561aaa9d4900 0x561aaa9d6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.657+0000 7fd0ede00640 1 --2- 172.21.15.40:0/2549366321 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561aaa9d4900 0x561aaa9d6cf0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d94acb6095b49ac5 server_cookie=5eb0f2036aff273d in_seq=0 out_seq=0 2024-04-27T00:11:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.657+0000 7fd0d3400640 10 client.16023 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:16.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0d3400640 1 -- 172.21.15.40:0/2549366321 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561aaa930f90 con 0x561aaa9d4900 2024-04-27T00:11:16.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0d3400640 10 client.16023 handle_command_reply: tid=0 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0f7983740 1 client.16023 shutdown 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0f7983740 1 -- 172.21.15.40:0/2549366321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9c4940 msgr2=0x561aaa9d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0f7983740 1 --2- 172.21.15.40:0/2549366321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9c4940 0x561aaa9d0520 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8000f80 tx=0x7fd0e8009cf0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0f7983740 1 -- 172.21.15.40:0/2549366321 shutdown_connections 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0f7983740 1 --2- 172.21.15.40:0/2549366321 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561aaa9d4900 0x561aaa9d6cf0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0f7983740 1 --2- 172.21.15.40:0/2549366321 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9c4940 0x561aaa9d0520 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0f7983740 1 --2- 172.21.15.40:0/2549366321 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bf7d0 0x561aaa9c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0f7983740 1 --2- 172.21.15.40:0/2549366321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9bede0 0x561aaa9c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0d3400640 0 client.16023 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0f7983740 1 -- 172.21.15.40:0/2549366321 >> 172.21.15.40:0/2549366321 conn(0x561aaa9ba450 msgr2=0x561aaa9b9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0f7983740 1 -- 172.21.15.40:0/2549366321 shutdown_connections 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0f7983740 1 -- 172.21.15.40:0/2549366321 wait complete. 2024-04-27T00:11:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.658+0000 7fd0f7983740 20 client.16023 trim_cache size 0 max 16384 2024-04-27T00:11:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.675+0000 7fd0f7983740 1 Processor -- start 2024-04-27T00:11:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.675+0000 7fd0f7983740 1 -- start start 2024-04-27T00:11:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.675+0000 7fd0f7983740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bede0 0x561aaa9b8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.675+0000 7fd0f7983740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bf7d0 0x561aaa9d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.675+0000 7fd0f7983740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9c4940 0x561aaa9b9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.675+0000 7fd0f7983740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01afd90 con 0x561aaa9bede0 2024-04-27T00:11:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.675+0000 7fd0f7983740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01ad660 con 0x561aaa9bf7d0 2024-04-27T00:11:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.675+0000 7fd0f7983740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01ab4b0 con 0x561aaa9c4940 2024-04-27T00:11:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.676+0000 7fd0ede00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bede0 0x561aaa9b8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.676+0000 7fd0ee800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9c4940 0x561aaa9b9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.676+0000 7fd0e5400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bf7d0 0x561aaa9d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.676+0000 7fd0ede00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bede0 0x561aaa9b8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60738/0 (socket says 172.21.15.40:60738) 2024-04-27T00:11:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.676+0000 7fd0ede00640 1 -- 172.21.15.40:0/3900513820 learned_addr learned my addr 172.21.15.40:0/3900513820 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.676+0000 7fd0ee800640 1 -- 172.21.15.40:0/3900513820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bf7d0 msgr2=0x561aaa9d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.676+0000 7fd0ee800640 1 --2- 172.21.15.40:0/3900513820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bf7d0 0x561aaa9d04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.676+0000 7fd0ee800640 1 -- 172.21.15.40:0/3900513820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bede0 msgr2=0x561aaa9b8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.676+0000 7fd0ee800640 1 --2- 172.21.15.40:0/3900513820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bede0 0x561aaa9b8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.676+0000 7fd0ee800640 1 -- 172.21.15.40:0/3900513820 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aaa701620 con 0x561aaa9c4940 2024-04-27T00:11:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.676+0000 7fd0ede00640 1 --2- 172.21.15.40:0/3900513820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bede0 0x561aaa9b8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:11:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.677+0000 7fd0ee800640 1 --2- 172.21.15.40:0/3900513820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9c4940 0x561aaa9b9300 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8005af0 tx=0x7fd0e80021f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10ddcdb7e062f224 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.677+0000 7fd0de600640 1 -- 172.21.15.40:0/3900513820 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8027070 con 0x561aaa9c4940 2024-04-27T00:11:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.677+0000 7fd0de600640 1 -- 172.21.15.40:0/3900513820 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0e8011d10 con 0x561aaa9c4940 2024-04-27T00:11:16.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.677+0000 7fd0de600640 1 -- 172.21.15.40:0/3900513820 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8003a30 con 0x561aaa9c4940 2024-04-27T00:11:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.677+0000 7fd0f7983740 1 -- 172.21.15.40:0/3900513820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9c4940 msgr2=0x561aaa9b9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.677+0000 7fd0f7983740 1 --2- 172.21.15.40:0/3900513820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9c4940 0x561aaa9b9300 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8005af0 tx=0x7fd0e80021f0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.678+0000 7fd0f7983740 1 -- 172.21.15.40:0/3900513820 shutdown_connections 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.678+0000 7fd0f7983740 1 --2- 172.21.15.40:0/3900513820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9c4940 0x561aaa9b9300 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.678+0000 7fd0f7983740 1 --2- 172.21.15.40:0/3900513820 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bf7d0 0x561aaa9d04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.678+0000 7fd0f7983740 1 --2- 172.21.15.40:0/3900513820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bede0 0x561aaa9b8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.678+0000 7fd0f7983740 1 -- 172.21.15.40:0/3900513820 >> 172.21.15.40:0/3900513820 conn(0x561aaa9ba450 msgr2=0x561aaa9c8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.678+0000 7fd0f7983740 1 -- 172.21.15.40:0/3900513820 shutdown_connections 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.678+0000 7fd0f7983740 1 -- 172.21.15.40:0/3900513820 wait complete. 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.678+0000 7fd0f7983740 1 -- start start 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.678+0000 7fd0f7983740 1 Processor -- start 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0f7983740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bede0 0x561aaa9cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0f7983740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 0x561aaa9ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0f7983740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9c4940 0x561aaa9cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0f7983740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01abb20 con 0x561aaa9bf7d0 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0f7983740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01acf90 con 0x561aaa9bede0 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0f7983740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01ac1f0 con 0x561aaa9c4940 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0f7983740 10 client.0 _ll_register_callbacks cb 0x7fd0e80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0e5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 0x561aaa9ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0e5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 0x561aaa9ce150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60750/0 (socket says 172.21.15.40:60750) 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0ee800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9c4940 0x561aaa9cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0e5400640 1 -- 172.21.15.40:0/2381141323 learned_addr learned my addr 172.21.15.40:0/2381141323 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0ede00640 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bede0 0x561aaa9cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0e5400640 1 -- 172.21.15.40:0/2381141323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9c4940 msgr2=0x561aaa9cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0e5400640 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9c4940 0x561aaa9cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0e5400640 1 -- 172.21.15.40:0/2381141323 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bede0 msgr2=0x561aaa9cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0e5400640 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bede0 0x561aaa9cdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.679+0000 7fd0e5400640 1 -- 172.21.15.40:0/2381141323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aaa8ddcb0 con 0x561aaa9bf7d0 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.680+0000 7fd0ee800640 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9c4940 0x561aaa9cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.680+0000 7fd0ede00640 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bede0 0x561aaa9cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.680+0000 7fd0e5400640 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 0x561aaa9ce150 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8099440 tx=0x7fd0d801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d63b81fe621318 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.680+0000 7fd0d3400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.680+0000 7fd0d3400640 1 -- 172.21.15.40:0/2381141323 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d809ca70 con 0x561aaa9bf7d0 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.680+0000 7fd0d3400640 1 -- 172.21.15.40:0/2381141323 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fd0d80bbb80 con 0x561aaa9bf7d0 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.680+0000 7fd0f7983740 1 -- 172.21.15.40:0/2381141323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0e8011d10 con 0x561aaa9bf7d0 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.680+0000 7fd0d3400640 1 -- 172.21.15.40:0/2381141323 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd0d80a0d70 con 0x561aaa9bf7d0 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.680+0000 7fd0f7983740 10 client.16029 fetch_fsmap learned FSMap version 24 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.680+0000 7fd0f7983740 1 -- 172.21.15.40:0/2381141323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561aaaa3ae20 con 0x561aaa9bf7d0 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.681+0000 7fd0d3400640 1 -- 172.21.15.40:0/2381141323 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0e8011d10 con 0x561aaa9bf7d0 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.681+0000 7fd0d3400640 1 -- 172.21.15.40:0/2381141323 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fd0d80d7070 con 0x561aaa9bf7d0 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.681+0000 7fd0f7983740 10 client.16029 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:11:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.681+0000 7fd0f7983740 10 client.16029 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:11:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.681+0000 7fd0f7983740 20 client.16029 populate_metadata read hostname 'smithi040' 2024-04-27T00:11:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.681+0000 7fd0f7983740 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561aaaa3b660 0x561aaaa3da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.681+0000 7fd0f7983740 4 client.16029 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:11:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.681+0000 7fd0f7983740 1 -- 172.21.15.40:0/2381141323 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561aaa9c8d90 con 0x561aaaa3b660 2024-04-27T00:11:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.681+0000 7fd0ede00640 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561aaaa3b660 0x561aaaa3da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.682+0000 7fd0ede00640 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561aaaa3b660 0x561aaaa3da80 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76a3cec41e29d298 server_cookie=c013ef0039136b83 in_seq=0 out_seq=0 2024-04-27T00:11:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.682+0000 7fd0d3400640 10 client.16029 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.697+0000 7fd0d3400640 1 -- 172.21.15.40:0/2381141323 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561aaa9c8d90 con 0x561aaaa3b660 2024-04-27T00:11:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.697+0000 7fd0d3400640 10 client.16029 handle_command_reply: tid=0 2024-04-27T00:11:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.697+0000 7fd0f7983740 1 client.16029 shutdown 2024-04-27T00:11:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.697+0000 7fd0f7983740 1 -- 172.21.15.40:0/2381141323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 msgr2=0x561aaa9ce150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.697+0000 7fd0f7983740 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 0x561aaa9ce150 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8099440 tx=0x7fd0d801f000 comp rx=0 tx=0).stop 2024-04-27T00:11:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.697+0000 7fd0f7983740 1 -- 172.21.15.40:0/2381141323 shutdown_connections 2024-04-27T00:11:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.697+0000 7fd0f7983740 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x561aaaa3b660 0x561aaaa3da80 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.697+0000 7fd0f7983740 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561aaa9c4940 0x561aaa9cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.697+0000 7fd0d3400640 0 client.16029 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.697+0000 7fd0f7983740 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561aaa9bf7d0 0x561aaa9ce150 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.697+0000 7fd0f7983740 1 --2- 172.21.15.40:0/2381141323 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x561aaa9bede0 0x561aaa9cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.697+0000 7fd0f7983740 1 -- 172.21.15.40:0/2381141323 >> 172.21.15.40:0/2381141323 conn(0x561aaa9ba450 msgr2=0x561aaa9bd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.698+0000 7fd0f7983740 1 -- 172.21.15.40:0/2381141323 shutdown_connections 2024-04-27T00:11:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.698+0000 7fd0f7983740 1 -- 172.21.15.40:0/2381141323 wait complete. 2024-04-27T00:11:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.698+0000 7fd0f7983740 20 client.16029 trim_cache size 0 max 16384 2024-04-27T00:11:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:11:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:11:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:11:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:11:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.699+0000 7fd0ef200640 1 -- 172.21.15.40:0/4251997416 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd0d40a3c30 msgr2=0x7fd0d40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.699+0000 7fd0ef200640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd0d40a3c30 0x7fd0d40a60f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x561aaa9c0240 tx=0x7fd0e0005a20 comp rx=0 tx=0).stop 2024-04-27T00:11:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.699+0000 7fd0ef200640 1 -- 172.21.15.40:0/4251997416 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd0f0105bc0 msgr2=0x7fd0f019b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.699+0000 7fd0ef200640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd0f0105bc0 0x7fd0f019b100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800e9f0 tx=0x7fd0d800eeb0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.699+0000 7fd0ef200640 1 -- 172.21.15.40:0/4251997416 shutdown_connections 2024-04-27T00:11:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.699+0000 7fd0ef200640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0f0106580 0x7fd0f01a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.699+0000 7fd0ef200640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd0f0105bc0 0x7fd0f019b100 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.699+0000 7fd0ef200640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd0d40a3c30 0x7fd0d40a60f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.699+0000 7fd0ef200640 1 --2- 172.21.15.40:0/4251997416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0f01049d0 0x7fd0f019abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.699+0000 7fd0ef200640 1 -- 172.21.15.40:0/4251997416 >> 172.21.15.40:0/4251997416 conn(0x7fd0f00fff40 msgr2=0x7fd0f0108f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.699+0000 7fd0ef200640 1 -- 172.21.15.40:0/4251997416 shutdown_connections 2024-04-27T00:11:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:16.699+0000 7fd0ef200640 1 -- 172.21.15.40:0/4251997416 wait complete. 2024-04-27T00:11:16.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:16 smithi040 ceph-mon[28095]: pgmap v699: 129 pgs: 129 active+clean; 2.5 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:18.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:18 smithi080 ceph-mon[32362]: pgmap v700: 129 pgs: 129 active+clean; 2.5 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:11:18.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:18 smithi103 ceph-mon[32952]: pgmap v700: 129 pgs: 129 active+clean; 2.5 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:11:18.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:18 smithi040 ceph-mon[28095]: pgmap v700: 129 pgs: 129 active+clean; 2.5 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:11:20.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:20 smithi103 ceph-mon[32952]: pgmap v701: 129 pgs: 129 active+clean; 2.5 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:11:20.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:20 smithi040 ceph-mon[28095]: pgmap v701: 129 pgs: 129 active+clean; 2.5 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:11:20.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:20 smithi080 ceph-mon[32362]: pgmap v701: 129 pgs: 129 active+clean; 2.5 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:11:22.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:22 smithi103 ceph-mon[32952]: pgmap v702: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-04-27T00:11:22.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:22 smithi040 ceph-mon[28095]: pgmap v702: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-04-27T00:11:22.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:22 smithi080 ceph-mon[32362]: pgmap v702: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-04-27T00:11:24.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:24 smithi103 ceph-mon[32952]: pgmap v703: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:11:24.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:24 smithi040 ceph-mon[28095]: pgmap v703: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:11:24.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:24 smithi080 ceph-mon[32362]: pgmap v703: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:11:26.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:26 smithi103 ceph-mon[32952]: pgmap v704: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:26.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:26 smithi040 ceph-mon[28095]: pgmap v704: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:26.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:26 smithi080 ceph-mon[32362]: pgmap v704: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:28.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:27 smithi080 ceph-mon[32362]: pgmap v705: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:11:28.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:27 smithi103 ceph-mon[32952]: pgmap v705: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:11:28.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:27 smithi040 ceph-mon[28095]: pgmap v705: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:11:29.850 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.849+0000 7fa112400640 1 -- 172.21.15.40:0/1294125316 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa10c1049d0 msgr2=0x7fa10c104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.849+0000 7fa112400640 1 --2- 172.21.15.40:0/1294125316 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa10c1049d0 0x7fa10c104df0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc009900 tx=0x7fa0fc02a400 comp rx=0 tx=0).stop 2024-04-27T00:11:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.849+0000 7fa112400640 1 -- 172.21.15.40:0/1294125316 shutdown_connections 2024-04-27T00:11:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.849+0000 7fa112400640 1 --2- 172.21.15.40:0/1294125316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa10c106580 0x7fa10c10ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.849+0000 7fa112400640 1 --2- 172.21.15.40:0/1294125316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa10c105bc0 0x7fa10c106040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.849+0000 7fa112400640 1 --2- 172.21.15.40:0/1294125316 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa10c1049d0 0x7fa10c104df0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.849+0000 7fa112400640 1 -- 172.21.15.40:0/1294125316 >> 172.21.15.40:0/1294125316 conn(0x7fa10c0fff40 msgr2=0x7fa10c102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.849+0000 7fa112400640 1 -- 172.21.15.40:0/1294125316 shutdown_connections 2024-04-27T00:11:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.849+0000 7fa112400640 1 -- 172.21.15.40:0/1294125316 wait complete. 2024-04-27T00:11:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.851+0000 7fa112400640 1 Processor -- start 2024-04-27T00:11:29.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.851+0000 7fa112400640 1 -- start start 2024-04-27T00:11:29.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.851+0000 7fa112400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa10c1049d0 0x7fa10c19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.851+0000 7fa112400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa10c105bc0 0x7fa10c19b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.851+0000 7fa112400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa10c106580 0x7fa10c1a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.851+0000 7fa112400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c110550 con 0x7fa10c106580 2024-04-27T00:11:29.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.851+0000 7fa112400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1103d0 con 0x7fa10c1049d0 2024-04-27T00:11:29.854 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.851+0000 7fa112400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1106d0 con 0x7fa10c105bc0 2024-04-27T00:11:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.852+0000 7fa111000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa10c1049d0 0x7fa10c19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.852+0000 7fa111000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa10c1049d0 0x7fa10c19abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:41004/0 (socket says 172.21.15.40:41004) 2024-04-27T00:11:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.852+0000 7fa111000640 1 -- 172.21.15.40:0/2454140327 learned_addr learned my addr 172.21.15.40:0/2454140327 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.852+0000 7fa111a00640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa10c106580 0x7fa10c1a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.852+0000 7fa103e00640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa10c105bc0 0x7fa10c19b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.852+0000 7fa111000640 1 -- 172.21.15.40:0/2454140327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa10c105bc0 msgr2=0x7fa10c19b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.852+0000 7fa111000640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa10c105bc0 0x7fa10c19b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.852+0000 7fa111000640 1 -- 172.21.15.40:0/2454140327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa10c106580 msgr2=0x7fa10c1a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.852+0000 7fa111000640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa10c106580 0x7fa10c1a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.852+0000 7fa111000640 1 -- 172.21.15.40:0/2454140327 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10c1a2810 con 0x7fa10c1049d0 2024-04-27T00:11:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.852+0000 7fa111000640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa10c1049d0 0x7fa10c19abf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc031500 tx=0x7fa0fc003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.853+0000 7fa10a000640 1 -- 172.21.15.40:0/2454140327 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc03f070 con 0x7fa10c1049d0 2024-04-27T00:11:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.853+0000 7fa112400640 1 -- 172.21.15.40:0/2454140327 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa10c1a2aa0 con 0x7fa10c1049d0 2024-04-27T00:11:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.853+0000 7fa10a000640 1 -- 172.21.15.40:0/2454140327 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0fc031c50 con 0x7fa10c1049d0 2024-04-27T00:11:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.853+0000 7fa10a000640 1 -- 172.21.15.40:0/2454140327 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc035030 con 0x7fa10c1049d0 2024-04-27T00:11:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.853+0000 7fa112400640 1 -- 172.21.15.40:0/2454140327 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa10c1a2fe0 con 0x7fa10c1049d0 2024-04-27T00:11:29.857 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.857+0000 7fa116779740 1 Processor -- start 2024-04-27T00:11:29.858 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.857+0000 7fa116779740 1 -- start start 2024-04-27T00:11:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.858+0000 7fa116779740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb6de0 0x55ead1bb7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.858+0000 7fa116779740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bb77d0 0x55ead1bbc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.858+0000 7fa116779740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bbc940 0x55ead1bbedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.858+0000 7fa116779740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ead1b1b380 con 0x55ead1bbc940 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.858+0000 7fa116779740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55ead1ab7c30 con 0x55ead1bb6de0 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.858+0000 7fa116779740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ead1b17340 con 0x55ead1bb77d0 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.858+0000 7fa10a000640 1 -- 172.21.15.40:0/2454140327 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa0fc042070 con 0x7fa10c1049d0 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.858+0000 7fa103e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bbc940 0x55ead1bbedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.858+0000 7fa111a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb6de0 0x55ead1bb7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.858+0000 7fa111a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb6de0 0x55ead1bb7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:54262/0 (socket says 172.21.15.40:54262) 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.858+0000 7fa111a00640 1 -- 172.21.15.40:0/1779178454 learned_addr learned my addr 172.21.15.40:0/1779178454 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.858+0000 7fa103e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bbc940 0x55ead1bbedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40814/0 (socket says 172.21.15.40:40814) 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.859+0000 7fa111000640 1 --2- 172.21.15.40:0/1779178454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bb77d0 0x55ead1bbc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.859+0000 7fa111a00640 1 -- 172.21.15.40:0/1779178454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bb77d0 msgr2=0x55ead1bbc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.859+0000 7fa111a00640 1 --2- 172.21.15.40:0/1779178454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bb77d0 0x55ead1bbc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.859+0000 7fa111a00640 1 -- 172.21.15.40:0/1779178454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bbc940 msgr2=0x55ead1bbedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.859+0000 7fa111a00640 1 --2- 172.21.15.40:0/1779178454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bbc940 0x55ead1bbedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.859+0000 7fa111a00640 1 -- 172.21.15.40:0/1779178454 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ead1ad9940 con 0x55ead1bb6de0 2024-04-27T00:11:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.859+0000 7fa103e00640 1 --2- 172.21.15.40:0/1779178454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bbc940 0x55ead1bbedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:11:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.859+0000 7fa111a00640 1 --2- 172.21.15.40:0/1779178454 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb6de0 0x55ead1bb7200 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa104009fd0 tx=0x7fa1040118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8fe794511a0177b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.859+0000 7fa102000640 1 -- 172.21.15.40:0/1779178454 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104004d20 con 0x55ead1bb6de0 2024-04-27T00:11:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.859+0000 7fa102000640 1 -- 172.21.15.40:0/1779178454 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa104004050 con 0x55ead1bb6de0 2024-04-27T00:11:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.859+0000 7fa102000640 1 -- 172.21.15.40:0/1779178454 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104005280 con 0x55ead1bb6de0 2024-04-27T00:11:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.860+0000 7fa10a000640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa0f00a3cb0 0x7fa0f00a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.860+0000 7fa10a000640 1 -- 172.21.15.40:0/2454140327 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fa0fc0f0ff0 con 0x7fa10c1049d0 2024-04-27T00:11:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.860+0000 7fa116779740 1 -- 172.21.15.40:0/1779178454 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb6de0 msgr2=0x55ead1bb7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.860+0000 7fa116779740 1 --2- 172.21.15.40:0/1779178454 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb6de0 0x55ead1bb7200 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa104009fd0 tx=0x7fa1040118d0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.860+0000 7fa111000640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa0f00a3cb0 0x7fa0f00a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.860+0000 7fa116779740 1 -- 172.21.15.40:0/1779178454 shutdown_connections 2024-04-27T00:11:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.860+0000 7fa116779740 1 --2- 172.21.15.40:0/1779178454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bbc940 0x55ead1bbedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.860+0000 7fa116779740 1 --2- 172.21.15.40:0/1779178454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bb77d0 0x55ead1bbc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.860+0000 7fa116779740 1 --2- 172.21.15.40:0/1779178454 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb6de0 0x55ead1bb7200 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.860+0000 7fa116779740 1 -- 172.21.15.40:0/1779178454 >> 172.21.15.40:0/1779178454 conn(0x55ead1bb2450 msgr2=0x55ead1bb2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.860+0000 7fa116779740 1 -- 172.21.15.40:0/1779178454 shutdown_connections 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.861+0000 7fa116779740 1 -- 172.21.15.40:0/1779178454 wait complete. 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.861+0000 7fa116779740 1 -- start start 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.861+0000 7fa111000640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa0f00a3cb0 0x7fa0f00a6170 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55ead1bb8240 tx=0x7fa0fc009560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.861+0000 7fa116779740 1 Processor -- start 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa116779740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bb6de0 0x55ead1bc0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa116779740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa116779740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bbc940 0x55ead1bc8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa116779740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ead1bbf8f0 con 0x55ead1bbc940 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa116779740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55ead1aa2c60 con 0x55ead1bb77d0 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa116779740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ead1bbfa70 con 0x55ead1bb6de0 2024-04-27T00:11:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa116779740 10 client.0 _ll_register_callbacks cb 0x55ead1acee50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa111a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bbc940 0x55ead1bc8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa111a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bb6de0 0x55ead1bc0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa103e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa103e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:54264/0 (socket says 172.21.15.40:54264) 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa111a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bb6de0 0x55ead1bc0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48740/0 (socket says 172.21.15.40:48740) 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa103e00640 1 -- 172.21.15.40:0/904257872 learned_addr learned my addr 172.21.15.40:0/904257872 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa103e00640 1 -- 172.21.15.40:0/904257872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bb6de0 msgr2=0x55ead1bc0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa103e00640 1 --2- 172.21.15.40:0/904257872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bb6de0 0x55ead1bc0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa103e00640 1 -- 172.21.15.40:0/904257872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bbc940 msgr2=0x55ead1bc8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa103e00640 1 --2- 172.21.15.40:0/904257872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bbc940 0x55ead1bc8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.862+0000 7fa103e00640 1 -- 172.21.15.40:0/904257872 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ead1ab1510 con 0x55ead1bb77d0 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.863+0000 7fa103e00640 1 --2- 172.21.15.40:0/904257872 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc1510 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8012700 tx=0x7fa0f8012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14045111febbfb63 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.863+0000 7fa0f6a00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:11:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.863+0000 7fa0f6a00640 1 -- 172.21.15.40:0/904257872 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8004030 con 0x55ead1bb77d0 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.863+0000 7fa116779740 1 -- 172.21.15.40:0/904257872 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa104004050 con 0x55ead1bb77d0 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.863+0000 7fa0f6a00640 1 -- 172.21.15.40:0/904257872 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fa0f80041d0 con 0x55ead1bb77d0 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.864+0000 7fa0f6a00640 1 -- 172.21.15.40:0/904257872 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa0f8025050 con 0x55ead1bb77d0 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.864+0000 7fa116779740 10 client.26150 fetch_fsmap learned FSMap version 24 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.864+0000 7fa116779740 1 -- 172.21.15.40:0/904257872 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ead182d210 con 0x55ead1bb77d0 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.864+0000 7fa0f6a00640 1 -- 172.21.15.40:0/904257872 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa104004050 con 0x55ead1bb77d0 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.864+0000 7fa0f6a00640 1 -- 172.21.15.40:0/904257872 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa0f8003950 con 0x55ead1bb77d0 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.864+0000 7fa116779740 10 client.26150 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.864+0000 7fa116779740 10 client.26150 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.864+0000 7fa116779740 20 client.26150 populate_metadata read hostname 'smithi040' 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.864+0000 7fa116779740 1 --2- 172.21.15.40:0/904257872 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ead1bccb10 0x55ead1bcef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.864+0000 7fa116779740 4 client.26150 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:11:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.864+0000 7fa116779740 1 -- 172.21.15.40:0/904257872 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ead1b28f90 con 0x55ead1bccb10 2024-04-27T00:11:29.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.865+0000 7fa111a00640 1 --2- 172.21.15.40:0/904257872 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ead1bccb10 0x55ead1bcef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.865+0000 7fa111a00640 1 --2- 172.21.15.40:0/904257872 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ead1bccb10 0x55ead1bcef30 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bf530166c7cbe84 server_cookie=43328a5fc96c29a1 in_seq=0 out_seq=0 2024-04-27T00:11:29.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.865+0000 7fa0f6a00640 10 client.26150 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:29.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.866+0000 7fa0f6a00640 1 -- 172.21.15.40:0/904257872 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ead1b28f90 con 0x55ead1bccb10 2024-04-27T00:11:29.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.866+0000 7fa0f6a00640 10 client.26150 handle_command_reply: tid=0 2024-04-27T00:11:29.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.866+0000 7fa116779740 1 client.26150 shutdown 2024-04-27T00:11:29.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.866+0000 7fa116779740 1 -- 172.21.15.40:0/904257872 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 msgr2=0x55ead1bc1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:29.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.866+0000 7fa116779740 1 --2- 172.21.15.40:0/904257872 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc1510 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8012700 tx=0x7fa0f8012bc0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.866+0000 7fa116779740 1 -- 172.21.15.40:0/904257872 shutdown_connections 2024-04-27T00:11:29.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.866+0000 7fa116779740 1 --2- 172.21.15.40:0/904257872 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ead1bccb10 0x55ead1bcef30 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.866+0000 7fa116779740 1 --2- 172.21.15.40:0/904257872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bbc940 0x55ead1bc8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.866+0000 7fa116779740 1 --2- 172.21.15.40:0/904257872 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc1510 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.866+0000 7fa116779740 1 --2- 172.21.15.40:0/904257872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bb6de0 0x55ead1bc0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.866+0000 7fa116779740 1 -- 172.21.15.40:0/904257872 >> 172.21.15.40:0/904257872 conn(0x55ead1bb2450 msgr2=0x55ead1bb1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:29.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.866+0000 7fa0f6a00640 0 client.26150 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:29.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.867+0000 7fa116779740 1 -- 172.21.15.40:0/904257872 shutdown_connections 2024-04-27T00:11:29.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.867+0000 7fa116779740 1 -- 172.21.15.40:0/904257872 wait complete. 2024-04-27T00:11:29.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.867+0000 7fa116779740 20 client.26150 trim_cache size 0 max 16384 2024-04-27T00:11:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.883+0000 7fa116779740 1 Processor -- start 2024-04-27T00:11:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.883+0000 7fa116779740 1 -- start start 2024-04-27T00:11:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.883+0000 7fa116779740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 0x55ead1bb0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.883+0000 7fa116779740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa116779740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bbc940 0x55ead1bb1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa116779740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1afdd0 con 0x55ead1bb6de0 2024-04-27T00:11:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa116779740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1ad6a0 con 0x55ead1bb77d0 2024-04-27T00:11:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa116779740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1ab4f0 con 0x55ead1bbc940 2024-04-27T00:11:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa103e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa103e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:54272/0 (socket says 172.21.15.40:54272) 2024-04-27T00:11:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa103e00640 1 -- 172.21.15.40:0/3072443218 learned_addr learned my addr 172.21.15.40:0/3072443218 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa111a00640 1 --2- 172.21.15.40:0/3072443218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 0x55ead1bb0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa111a00640 1 --2- 172.21.15.40:0/3072443218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bbc940 0x55ead1bb1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa103e00640 1 -- 172.21.15.40:0/3072443218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bbc940 msgr2=0x55ead1bb1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa103e00640 1 --2- 172.21.15.40:0/3072443218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bbc940 0x55ead1bb1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa103e00640 1 -- 172.21.15.40:0/3072443218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 msgr2=0x55ead1bb0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa103e00640 1 --2- 172.21.15.40:0/3072443218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 0x55ead1bb0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa103e00640 1 -- 172.21.15.40:0/3072443218 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ead18f9620 con 0x55ead1bb77d0 2024-04-27T00:11:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.884+0000 7fa111a00640 1 --2- 172.21.15.40:0/3072443218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 0x55ead1bb0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:11:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.885+0000 7fa103e00640 1 --2- 172.21.15.40:0/3072443218 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc8510 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8003da0 tx=0x7fa0f80096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d454e99cd42dc60 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.885+0000 7fa101600640 1 -- 172.21.15.40:0/3072443218 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f802e070 con 0x55ead1bb77d0 2024-04-27T00:11:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.885+0000 7fa101600640 1 -- 172.21.15.40:0/3072443218 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0f8003aa0 con 0x55ead1bb77d0 2024-04-27T00:11:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.885+0000 7fa101600640 1 -- 172.21.15.40:0/3072443218 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f800f8c0 con 0x55ead1bb77d0 2024-04-27T00:11:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.886+0000 7fa116779740 1 -- 172.21.15.40:0/3072443218 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 msgr2=0x55ead1bc8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.886+0000 7fa116779740 1 --2- 172.21.15.40:0/3072443218 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc8510 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8003da0 tx=0x7fa0f80096c0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.886+0000 7fa116779740 1 -- 172.21.15.40:0/3072443218 shutdown_connections 2024-04-27T00:11:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.886+0000 7fa116779740 1 --2- 172.21.15.40:0/3072443218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bbc940 0x55ead1bb1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.886+0000 7fa116779740 1 --2- 172.21.15.40:0/3072443218 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc8510 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.886+0000 7fa116779740 1 --2- 172.21.15.40:0/3072443218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 0x55ead1bb0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.886+0000 7fa116779740 1 -- 172.21.15.40:0/3072443218 >> 172.21.15.40:0/3072443218 conn(0x55ead1bb2450 msgr2=0x55ead1bb89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.886+0000 7fa116779740 1 -- 172.21.15.40:0/3072443218 shutdown_connections 2024-04-27T00:11:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.886+0000 7fa116779740 1 -- 172.21.15.40:0/3072443218 wait complete. 2024-04-27T00:11:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.886+0000 7fa116779740 1 -- start start 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.886+0000 7fa116779740 1 Processor -- start 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.887+0000 7fa116779740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 0x55ead1bc5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.887+0000 7fa116779740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.887+0000 7fa116779740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bbc940 0x55ead1bc71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.887+0000 7fa116779740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1abb60 con 0x55ead1bb6de0 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.887+0000 7fa116779740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1acfd0 con 0x55ead1bb77d0 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.887+0000 7fa116779740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1ac230 con 0x55ead1bbc940 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.887+0000 7fa116779740 10 client.0 _ll_register_callbacks cb 0x7fa0f8009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.887+0000 7fa111a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 0x55ead1bc5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.887+0000 7fa103e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.887+0000 7fa111a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 0x55ead1bc5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40840/0 (socket says 172.21.15.40:40840) 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.887+0000 7fa111a00640 1 -- 172.21.15.40:0/3219455059 learned_addr learned my addr 172.21.15.40:0/3219455059 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.887+0000 7fa111a00640 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bbc940 0x55ead1bc71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa103e00640 1 -- 172.21.15.40:0/3219455059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bbc940 msgr2=0x55ead1bc71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa103e00640 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bbc940 0x55ead1bc71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa103e00640 1 -- 172.21.15.40:0/3219455059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 msgr2=0x55ead1bc5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa103e00640 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 0x55ead1bc5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa103e00640 1 -- 172.21.15.40:0/3219455059 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ead1ad5cb0 con 0x55ead1bb77d0 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa111a00640 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 0x55ead1bc5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa103e00640 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc61c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8003d20 tx=0x7fa0f8003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dba98e5dc2e0b2fb server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa0f6a00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa0f6a00640 1 -- 172.21.15.40:0/3219455059 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f802e070 con 0x55ead1bb77d0 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa0f6a00640 1 -- 172.21.15.40:0/3219455059 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fa0f8005280 con 0x55ead1bb77d0 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa116779740 1 -- 172.21.15.40:0/3219455059 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0f8003aa0 con 0x55ead1bb77d0 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa0f6a00640 1 -- 172.21.15.40:0/3219455059 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa0f803a050 con 0x55ead1bb77d0 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa116779740 10 client.26156 fetch_fsmap learned FSMap version 24 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.888+0000 7fa116779740 1 -- 172.21.15.40:0/3219455059 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ead1c330b0 con 0x55ead1bb77d0 2024-04-27T00:11:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.889+0000 7fa0f6a00640 1 -- 172.21.15.40:0/3219455059 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0f8003aa0 con 0x55ead1bb77d0 2024-04-27T00:11:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.889+0000 7fa0f6a00640 1 -- 172.21.15.40:0/3219455059 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa0f8010770 con 0x55ead1bb77d0 2024-04-27T00:11:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.889+0000 7fa116779740 10 client.26156 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:11:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.889+0000 7fa116779740 10 client.26156 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:11:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.889+0000 7fa116779740 20 client.26156 populate_metadata read hostname 'smithi040' 2024-04-27T00:11:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.889+0000 7fa116779740 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ead1c33850 0x55ead1c35c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.889+0000 7fa116779740 4 client.26156 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:11:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.889+0000 7fa116779740 1 -- 172.21.15.40:0/3219455059 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ead1b22e00 con 0x55ead1c33850 2024-04-27T00:11:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.889+0000 7fa111a00640 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ead1c33850 0x55ead1c35c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:29.894 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.890+0000 7fa111a00640 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ead1c33850 0x55ead1c35c70 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b071bbb0b3ee54b2 server_cookie=1949a36dfdeae363 in_seq=0 out_seq=0 2024-04-27T00:11:29.894 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.890+0000 7fa0f6a00640 10 client.26156 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:29.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa0f6a00640 1 -- 172.21.15.40:0/3219455059 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ead1b22e00 con 0x55ead1c33850 2024-04-27T00:11:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa0f6a00640 10 client.26156 handle_command_reply: tid=0 2024-04-27T00:11:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa116779740 1 client.26156 shutdown 2024-04-27T00:11:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa116779740 1 -- 172.21.15.40:0/3219455059 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 msgr2=0x55ead1bc61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa116779740 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc61c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8003d20 tx=0x7fa0f8003820 comp rx=0 tx=0).stop 2024-04-27T00:11:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa116779740 1 -- 172.21.15.40:0/3219455059 shutdown_connections 2024-04-27T00:11:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa116779740 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55ead1c33850 0x55ead1c35c70 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa116779740 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ead1bbc940 0x55ead1bc71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa116779740 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55ead1bb77d0 0x55ead1bc61c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa116779740 1 --2- 172.21.15.40:0/3219455059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ead1bb6de0 0x55ead1bc5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa0f6a00640 0 client.26156 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:29.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa116779740 1 -- 172.21.15.40:0/3219455059 >> 172.21.15.40:0/3219455059 conn(0x55ead1bb2450 msgr2=0x55ead1bb54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:29.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa116779740 1 -- 172.21.15.40:0/3219455059 shutdown_connections 2024-04-27T00:11:29.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.905+0000 7fa116779740 1 -- 172.21.15.40:0/3219455059 wait complete. 2024-04-27T00:11:29.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.906+0000 7fa116779740 20 client.26156 trim_cache size 0 max 16384 2024-04-27T00:11:29.909 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:11:29.909 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:11:29.909 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:11:29.909 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:11:29.910 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.906+0000 7fa112400640 1 -- 172.21.15.40:0/2454140327 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa0f00a3cb0 msgr2=0x7fa0f00a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:29.910 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.906+0000 7fa112400640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa0f00a3cb0 0x7fa0f00a6170 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55ead1bb8240 tx=0x7fa0fc009560 comp rx=0 tx=0).stop 2024-04-27T00:11:29.910 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.906+0000 7fa112400640 1 -- 172.21.15.40:0/2454140327 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa10c1049d0 msgr2=0x7fa10c19abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.906+0000 7fa112400640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa10c1049d0 0x7fa10c19abf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc031500 tx=0x7fa0fc003c10 comp rx=0 tx=0).stop 2024-04-27T00:11:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.906+0000 7fa112400640 1 -- 172.21.15.40:0/2454140327 shutdown_connections 2024-04-27T00:11:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.906+0000 7fa112400640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa10c106580 0x7fa10c1a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.906+0000 7fa112400640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fa0f00a3cb0 0x7fa0f00a6170 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.906+0000 7fa112400640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa10c105bc0 0x7fa10c19b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.906+0000 7fa112400640 1 --2- 172.21.15.40:0/2454140327 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fa10c1049d0 0x7fa10c19abf0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.906+0000 7fa112400640 1 -- 172.21.15.40:0/2454140327 >> 172.21.15.40:0/2454140327 conn(0x7fa10c0fff40 msgr2=0x7fa10c108f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.907+0000 7fa112400640 1 -- 172.21.15.40:0/2454140327 shutdown_connections 2024-04-27T00:11:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:29.907+0000 7fa112400640 1 -- 172.21.15.40:0/2454140327 wait complete. 2024-04-27T00:11:30.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:30 smithi040 ceph-mon[28095]: pgmap v706: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:30.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:30 smithi080 ceph-mon[32362]: pgmap v706: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:30.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:30 smithi103 ceph-mon[32952]: pgmap v706: 129 pgs: 129 active+clean; 2.6 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:32.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:32 smithi040 ceph-mon[28095]: pgmap v707: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:11:32.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:32 smithi080 ceph-mon[32362]: pgmap v707: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:11:32.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:32 smithi103 ceph-mon[32952]: pgmap v707: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:11:34.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:34 smithi040 ceph-mon[28095]: pgmap v708: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:11:34.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:34 smithi080 ceph-mon[32362]: pgmap v708: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:11:34.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:34 smithi103 ceph-mon[32952]: pgmap v708: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-04-27T00:11:36.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:36 smithi040 ceph-mon[28095]: pgmap v709: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:11:36.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:36 smithi080 ceph-mon[32362]: pgmap v709: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:11:36.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:36 smithi103 ceph-mon[32952]: pgmap v709: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:11:38.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:38 smithi040 ceph-mon[28095]: pgmap v710: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 16 op/s 2024-04-27T00:11:38.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:38 smithi080 ceph-mon[32362]: pgmap v710: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 16 op/s 2024-04-27T00:11:38.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:38 smithi103 ceph-mon[32952]: pgmap v710: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 16 op/s 2024-04-27T00:11:40.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:40 smithi040 ceph-mon[28095]: pgmap v711: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:11:40.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:40 smithi080 ceph-mon[32362]: pgmap v711: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:11:40.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:40 smithi103 ceph-mon[32952]: pgmap v711: 129 pgs: 129 active+clean; 2.7 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:11:42.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:42 smithi040 ceph-mon[28095]: pgmap v712: 129 pgs: 129 active+clean; 2.8 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:11:42.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:42 smithi080 ceph-mon[32362]: pgmap v712: 129 pgs: 129 active+clean; 2.8 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:11:42.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:42 smithi103 ceph-mon[32952]: pgmap v712: 129 pgs: 129 active+clean; 2.8 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:11:42.701 INFO:tasks.workunit.client.0.smithi040.stdout:437/600: open fd = 3 2024-04-27T00:11:42.701 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.701 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.702 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.702 INFO:tasks.workunit.client.0.smithi040.stdout:438/600: open fd = 3 2024-04-27T00:11:42.702 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.702 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.702 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.702 INFO:tasks.workunit.client.0.smithi040.stdout:439/600: open fd = 3 2024-04-27T00:11:42.702 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.702 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.702 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.702 INFO:tasks.workunit.client.0.smithi040.stdout:440/600: open fd = 3 2024-04-27T00:11:42.702 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.702 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:441/600: open fd = 3 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:442/600: open fd = 3 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:443/600: open fd = 3 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:444/600: open fd = 3 2024-04-27T00:11:42.703 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:445/600: open fd = 3 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:446/600: open fd = 3 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:447/600: open fd = 3 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:448/600: open fd = 3 2024-04-27T00:11:42.704 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:449/600: open fd = 3 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:450/600: open fd = 3 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:451/600: open fd = 3 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:452/600: open fd = 3 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:453/600: open fd = 3 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:454/600: open fd = 3 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:455/600: open fd = 3 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:456/600: open fd = 3 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:457/600: open fd = 3 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:458/600: open fd = 3 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.707 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:459/600: open fd = 3 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:460/600: open fd = 3 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:461/600: open fd = 3 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:462/600: open fd = 3 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.708 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:463/600: open fd = 3 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:464/600: open fd = 3 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:465/600: open fd = 3 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:466/600: open fd = 3 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.709 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:467/600: open fd = 3 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:468/600: open fd = 3 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:469/600: open fd = 3 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.710 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:470/600: open fd = 3 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:471/600: open fd = 3 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:472/600: open fd = 3 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:473/600: open fd = 3 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.711 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:474/600: open fd = 3 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:475/600: open fd = 3 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:476/600: open fd = 3 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:477/600: open fd = 3 2024-04-27T00:11:42.712 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.713 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.713 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.713 INFO:tasks.workunit.client.0.smithi040.stdout:478/600: open fd = 3 2024-04-27T00:11:42.713 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.713 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.713 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.713 INFO:tasks.workunit.client.0.smithi040.stdout:479/600: open fd = 3 2024-04-27T00:11:42.713 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.713 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.713 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.713 INFO:tasks.workunit.client.0.smithi040.stdout:480/600: open fd = 3 2024-04-27T00:11:42.713 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:481/600: open fd = 3 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:482/600: open fd = 3 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:483/600: open fd = 3 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.714 INFO:tasks.workunit.client.0.smithi040.stdout:484/600: open fd = 3 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:485/600: open fd = 3 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:486/600: open fd = 3 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:487/600: open fd = 3 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.715 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:488/600: open fd = 3 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:489/600: open fd = 3 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:490/600: open fd = 3 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:491/600: open fd = 3 2024-04-27T00:11:42.716 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:492/600: open fd = 3 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:493/600: open fd = 3 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:494/600: open fd = 3 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.717 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:495/600: open fd = 3 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:496/600: open fd = 3 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:497/600: open fd = 3 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:498/600: open fd = 3 2024-04-27T00:11:42.718 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.719 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:11:42.719 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:11:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.056+0000 7f602d800640 1 -- 172.21.15.40:0/2340840266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f602810ab70 msgr2=0x7f6028111400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.056+0000 7f602d800640 1 --2- 172.21.15.40:0/2340840266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f602810ab70 0x7f6028111400 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6020009ca0 tx=0x7f602002a8c0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.056+0000 7f602d800640 1 -- 172.21.15.40:0/2340840266 shutdown_connections 2024-04-27T00:11:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.056+0000 7f602d800640 1 --2- 172.21.15.40:0/2340840266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f602810ab70 0x7f6028111400 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.056+0000 7f602d800640 1 --2- 172.21.15.40:0/2340840266 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602810a1b0 0x7f602810a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.056+0000 7f602d800640 1 --2- 172.21.15.40:0/2340840266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6028108fc0 0x7f60281093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.056+0000 7f602d800640 1 -- 172.21.15.40:0/2340840266 >> 172.21.15.40:0/2340840266 conn(0x7f6028077f80 msgr2=0x7f602807a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.056+0000 7f602d800640 1 -- 172.21.15.40:0/2340840266 shutdown_connections 2024-04-27T00:11:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.056+0000 7f602d800640 1 -- 172.21.15.40:0/2340840266 wait complete. 2024-04-27T00:11:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.057+0000 7f602d800640 1 Processor -- start 2024-04-27T00:11:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.057+0000 7f602d800640 1 -- start start 2024-04-27T00:11:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.058+0000 7f602d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6028108fc0 0x7f602819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.058+0000 7f602d800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602810a1b0 0x7f602819f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.058+0000 7f602d800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602810ab70 0x7f60281a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.058+0000 7f602d800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6028114b50 con 0x7f6028108fc0 2024-04-27T00:11:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.058+0000 7f602d800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f60281149d0 con 0x7f602810ab70 2024-04-27T00:11:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.058+0000 7f602d800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6028114cd0 con 0x7f602810a1b0 2024-04-27T00:11:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.058+0000 7f6026000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602810a1b0 0x7f602819f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.058+0000 7f6026a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6028108fc0 0x7f602819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.058+0000 7f6026000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602810a1b0 0x7f602819f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36522/0 (socket says 172.21.15.40:36522) 2024-04-27T00:11:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.058+0000 7f6026000640 1 -- 172.21.15.40:0/137347588 learned_addr learned my addr 172.21.15.40:0/137347588 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.058+0000 7f6027400640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602810ab70 0x7f60281a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.059+0000 7f6026000640 1 -- 172.21.15.40:0/137347588 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602810ab70 msgr2=0x7f60281a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.059+0000 7f6026000640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602810ab70 0x7f60281a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.059+0000 7f6026000640 1 -- 172.21.15.40:0/137347588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6028108fc0 msgr2=0x7f602819f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.059+0000 7f6026000640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6028108fc0 0x7f602819f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.059+0000 7f6026000640 1 -- 172.21.15.40:0/137347588 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60281a6cc0 con 0x7f602810a1b0 2024-04-27T00:11:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.059+0000 7f6026a00640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6028108fc0 0x7f602819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:11:43.060 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.059+0000 7f6026000640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602810a1b0 0x7f602819f5e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f601400e9f0 tx=0x7f601400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:43.060 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.059+0000 7f601f400640 1 -- 172.21.15.40:0/137347588 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601400cd80 con 0x7f602810a1b0 2024-04-27T00:11:43.060 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.059+0000 7f602d800640 1 -- 172.21.15.40:0/137347588 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60281a6fb0 con 0x7f602810a1b0 2024-04-27T00:11:43.060 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.059+0000 7f601f400640 1 -- 172.21.15.40:0/137347588 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60140108e0 con 0x7f602810a1b0 2024-04-27T00:11:43.060 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.059+0000 7f601f400640 1 -- 172.21.15.40:0/137347588 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6014018980 con 0x7f602810a1b0 2024-04-27T00:11:43.061 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.060+0000 7f602d800640 1 -- 172.21.15.40:0/137347588 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60281a74c0 con 0x7f602810a1b0 2024-04-27T00:11:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.063+0000 7f603062d740 1 Processor -- start 2024-04-27T00:11:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.063+0000 7f603062d740 1 -- start start 2024-04-27T00:11:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.063+0000 7f603062d740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 0x562e454b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.063+0000 7f603062d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b55c0 0x562e454ba0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.063+0000 7f603062d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454ba730 0x562e454bcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.063+0000 7f603062d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562e45419380 con 0x562e454ba730 2024-04-27T00:11:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.063+0000 7f603062d740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x562e453b5c30 con 0x562e454b4bd0 2024-04-27T00:11:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.063+0000 7f603062d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562e45415340 con 0x562e454b55c0 2024-04-27T00:11:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.063+0000 7f6027400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 0x562e454b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.063+0000 7f6027400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 0x562e454b4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:59448/0 (socket says 172.21.15.40:59448) 2024-04-27T00:11:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.063+0000 7f6027400640 1 -- 172.21.15.40:0/1295384002 learned_addr learned my addr 172.21.15.40:0/1295384002 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.064+0000 7f6026a00640 1 --2- 172.21.15.40:0/1295384002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b55c0 0x562e454ba0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.064+0000 7f6027400640 1 -- 172.21.15.40:0/1295384002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b55c0 msgr2=0x562e454ba0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.064+0000 7f6027400640 1 --2- 172.21.15.40:0/1295384002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b55c0 0x562e454ba0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.064+0000 7f6027400640 1 -- 172.21.15.40:0/1295384002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454ba730 msgr2=0x562e454bcbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-27T00:11:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.064+0000 7f6027400640 1 --2- 172.21.15.40:0/1295384002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454ba730 0x562e454bcbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.064+0000 7f6027400640 1 -- 172.21.15.40:0/1295384002 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e453d7940 con 0x562e454b4bd0 2024-04-27T00:11:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.064+0000 7f601f400640 1 -- 172.21.15.40:0/137347588 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6014010430 con 0x7f602810a1b0 2024-04-27T00:11:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.064+0000 7f6027400640 1 --2- 172.21.15.40:0/1295384002 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 0x562e454b4ff0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6020009ca0 tx=0x7f60200093c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8e324982d65686c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.064+0000 7f6013e00640 1 -- 172.21.15.40:0/1295384002 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6020031650 con 0x562e454b4bd0 2024-04-27T00:11:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.064+0000 7f6013e00640 1 -- 172.21.15.40:0/1295384002 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6020036030 con 0x562e454b4bd0 2024-04-27T00:11:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.064+0000 7f6013e00640 1 -- 172.21.15.40:0/1295384002 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6020004660 con 0x562e454b4bd0 2024-04-27T00:11:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.065+0000 7f603062d740 1 -- 172.21.15.40:0/1295384002 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 msgr2=0x562e454b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.065+0000 7f603062d740 1 --2- 172.21.15.40:0/1295384002 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 0x562e454b4ff0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6020009ca0 tx=0x7f60200093c0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.065+0000 7f603062d740 1 -- 172.21.15.40:0/1295384002 shutdown_connections 2024-04-27T00:11:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.065+0000 7f603062d740 1 --2- 172.21.15.40:0/1295384002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454ba730 0x562e454bcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.065+0000 7f603062d740 1 --2- 172.21.15.40:0/1295384002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b55c0 0x562e454ba0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.065+0000 7f603062d740 1 --2- 172.21.15.40:0/1295384002 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 0x562e454b4ff0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.065+0000 7f603062d740 1 -- 172.21.15.40:0/1295384002 >> 172.21.15.40:0/1295384002 conn(0x562e454b0240 msgr2=0x562e454b0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:43.070 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.065+0000 7f603062d740 1 -- 172.21.15.40:0/1295384002 shutdown_connections 2024-04-27T00:11:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.065+0000 7f603062d740 1 -- 172.21.15.40:0/1295384002 wait complete. 2024-04-27T00:11:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.065+0000 7f601f400640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f600c0a3c00 0x7f600c0a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.066+0000 7f601f400640 1 -- 172.21.15.40:0/137347588 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f60140d0be0 con 0x7f602810a1b0 2024-04-27T00:11:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.066+0000 7f603062d740 1 -- start start 2024-04-27T00:11:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.066+0000 7f6026a00640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f600c0a3c00 0x7f600c0a60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.066+0000 7f603062d740 1 Processor -- start 2024-04-27T00:11:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.066+0000 7f603062d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b4bd0 0x562e454becc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.066+0000 7f603062d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b55c0 0x562e454bf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.066+0000 7f6026a00640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f600c0a3c00 0x7f600c0a60c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x562e454b11b0 tx=0x7f6018005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f603062d740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454ba730 0x562e454c6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f603062d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562e454bd500 con 0x562e454b4bd0 2024-04-27T00:11:43.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f603062d740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x562e453a0c60 con 0x562e454ba730 2024-04-27T00:11:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f603062d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562e454bd680 con 0x562e454b55c0 2024-04-27T00:11:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f603062d740 10 client.0 _ll_register_callbacks cb 0x562e453cce50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:11:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f6027400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454ba730 0x562e454c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f6026a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b4bd0 0x562e454becc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f6027400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454ba730 0x562e454c6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:59458/0 (socket says 172.21.15.40:59458) 2024-04-27T00:11:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f6027400640 1 -- 172.21.15.40:0/3096448608 learned_addr learned my addr 172.21.15.40:0/3096448608 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f6026000640 1 --2- 172.21.15.40:0/3096448608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b55c0 0x562e454bf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f6027400640 1 -- 172.21.15.40:0/3096448608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b55c0 msgr2=0x562e454bf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f6027400640 1 --2- 172.21.15.40:0/3096448608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b55c0 0x562e454bf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f6027400640 1 -- 172.21.15.40:0/3096448608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b4bd0 msgr2=0x562e454becc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f6027400640 1 --2- 172.21.15.40:0/3096448608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b4bd0 0x562e454becc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.067+0000 7f6027400640 1 -- 172.21.15.40:0/3096448608 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e453af510 con 0x562e454ba730 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.068+0000 7f6027400640 1 --2- 172.21.15.40:0/3096448608 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454ba730 0x562e454c6240 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6020006010 tx=0x7f6020004690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe29c527d7fa4c68 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.068+0000 7f6010c00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.068+0000 7f6010c00640 1 -- 172.21.15.40:0/3096448608 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602004a040 con 0x562e454ba730 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.068+0000 7f603062d740 1 -- 172.21.15.40:0/3096448608 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6020036030 con 0x562e454ba730 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.068+0000 7f6010c00640 1 -- 172.21.15.40:0/3096448608 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f6020003b70 con 0x562e454ba730 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.068+0000 7f6010c00640 1 -- 172.21.15.40:0/3096448608 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f602005e050 con 0x562e454ba730 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.068+0000 7f603062d740 10 client.26168 fetch_fsmap learned FSMap version 24 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.068+0000 7f603062d740 1 -- 172.21.15.40:0/3096448608 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562e4512b210 con 0x562e454ba730 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.069+0000 7f6010c00640 1 -- 172.21.15.40:0/3096448608 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6020036030 con 0x562e454ba730 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.069+0000 7f6010c00640 1 -- 172.21.15.40:0/3096448608 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f602004d030 con 0x562e454ba730 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.069+0000 7f603062d740 10 client.26168 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.069+0000 7f603062d740 10 client.26168 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.069+0000 7f603062d740 20 client.26168 populate_metadata read hostname 'smithi040' 2024-04-27T00:11:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.069+0000 7f603062d740 1 --2- 172.21.15.40:0/3096448608 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562e454ca460 0x562e454cc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.069+0000 7f603062d740 4 client.26168 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:11:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.069+0000 7f603062d740 1 -- 172.21.15.40:0/3096448608 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e45426f90 con 0x562e454ca460 2024-04-27T00:11:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.069+0000 7f6026a00640 1 --2- 172.21.15.40:0/3096448608 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562e454ca460 0x562e454cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.070+0000 7f6026a00640 1 --2- 172.21.15.40:0/3096448608 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562e454ca460 0x562e454cc850 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91b9561e76a6e156 server_cookie=52caaaf5a416c4 in_seq=0 out_seq=0 2024-04-27T00:11:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.070+0000 7f6010c00640 10 client.26168 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f6010c00640 1 -- 172.21.15.40:0/3096448608 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e45426f90 con 0x562e454ca460 2024-04-27T00:11:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f6010c00640 10 client.26168 handle_command_reply: tid=0 2024-04-27T00:11:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f603062d740 1 client.26168 shutdown 2024-04-27T00:11:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f603062d740 1 -- 172.21.15.40:0/3096448608 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454ba730 msgr2=0x562e454c6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f603062d740 1 --2- 172.21.15.40:0/3096448608 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454ba730 0x562e454c6240 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6020006010 tx=0x7f6020004690 comp rx=0 tx=0).stop 2024-04-27T00:11:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f603062d740 1 -- 172.21.15.40:0/3096448608 shutdown_connections 2024-04-27T00:11:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f603062d740 1 --2- 172.21.15.40:0/3096448608 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562e454ca460 0x562e454cc850 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f603062d740 1 --2- 172.21.15.40:0/3096448608 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454ba730 0x562e454c6240 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f603062d740 1 --2- 172.21.15.40:0/3096448608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b55c0 0x562e454bf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f603062d740 1 --2- 172.21.15.40:0/3096448608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b4bd0 0x562e454becc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f603062d740 1 -- 172.21.15.40:0/3096448608 >> 172.21.15.40:0/3096448608 conn(0x562e454b0240 msgr2=0x562e454af440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f6010c00640 0 client.26168 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f603062d740 1 -- 172.21.15.40:0/3096448608 shutdown_connections 2024-04-27T00:11:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.071+0000 7f603062d740 1 -- 172.21.15.40:0/3096448608 wait complete. 2024-04-27T00:11:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.072+0000 7f603062d740 20 client.26168 trim_cache size 0 max 16384 2024-04-27T00:11:43.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.088+0000 7f603062d740 1 Processor -- start 2024-04-27T00:11:43.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.088+0000 7f603062d740 1 -- start start 2024-04-27T00:11:43.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.088+0000 7f603062d740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 0x562e454be6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.088+0000 7f603062d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b55c0 0x562e454c6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.088+0000 7f603062d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454ba730 0x562e454b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.088+0000 7f603062d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f60281b41f0 con 0x562e454b55c0 2024-04-27T00:11:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.088+0000 7f603062d740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f60281b1ac0 con 0x562e454b4bd0 2024-04-27T00:11:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.088+0000 7f603062d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60281af910 con 0x562e454ba730 2024-04-27T00:11:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.088+0000 7f6026000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b55c0 0x562e454c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.089+0000 7f6026a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 0x562e454be6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.089+0000 7f6026a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 0x562e454be6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:59460/0 (socket says 172.21.15.40:59460) 2024-04-27T00:11:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.089+0000 7f6026a00640 1 -- 172.21.15.40:0/1806863053 learned_addr learned my addr 172.21.15.40:0/1806863053 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.089+0000 7f6027400640 1 --2- 172.21.15.40:0/1806863053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454ba730 0x562e454b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.089+0000 7f6026a00640 1 -- 172.21.15.40:0/1806863053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454ba730 msgr2=0x562e454b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.089+0000 7f6026a00640 1 --2- 172.21.15.40:0/1806863053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454ba730 0x562e454b31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.089+0000 7f6026a00640 1 -- 172.21.15.40:0/1806863053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b55c0 msgr2=0x562e454c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.089+0000 7f6026a00640 1 --2- 172.21.15.40:0/1806863053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b55c0 0x562e454c6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.089+0000 7f6026a00640 1 -- 172.21.15.40:0/1806863053 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e454ccbc0 con 0x562e454b4bd0 2024-04-27T00:11:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.089+0000 7f6026a00640 1 --2- 172.21.15.40:0/1806863053 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 0x562e454be6a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x562e454b6030 tx=0x7f6018019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48d8972863f6bc87 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.090+0000 7f6013400640 1 -- 172.21.15.40:0/1806863053 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601801dca0 con 0x562e454b4bd0 2024-04-27T00:11:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.090+0000 7f6013400640 1 -- 172.21.15.40:0/1806863053 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6018033070 con 0x562e454b4bd0 2024-04-27T00:11:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.090+0000 7f6013400640 1 -- 172.21.15.40:0/1806863053 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601802a820 con 0x562e454b4bd0 2024-04-27T00:11:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.090+0000 7f603062d740 1 -- 172.21.15.40:0/1806863053 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 msgr2=0x562e454be6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.090+0000 7f603062d740 1 --2- 172.21.15.40:0/1806863053 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 0x562e454be6a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x562e454b6030 tx=0x7f6018019000 comp rx=0 tx=0).stop 2024-04-27T00:11:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.090+0000 7f603062d740 1 -- 172.21.15.40:0/1806863053 shutdown_connections 2024-04-27T00:11:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.090+0000 7f603062d740 1 --2- 172.21.15.40:0/1806863053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454ba730 0x562e454b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.090+0000 7f603062d740 1 --2- 172.21.15.40:0/1806863053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b55c0 0x562e454c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.090+0000 7f603062d740 1 --2- 172.21.15.40:0/1806863053 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454b4bd0 0x562e454be6a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.090+0000 7f603062d740 1 -- 172.21.15.40:0/1806863053 >> 172.21.15.40:0/1806863053 conn(0x562e454b0240 msgr2=0x562e454bdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.090+0000 7f603062d740 1 -- 172.21.15.40:0/1806863053 shutdown_connections 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.091+0000 7f603062d740 1 -- 172.21.15.40:0/1806863053 wait complete. 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.091+0000 7f603062d740 1 -- start start 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.091+0000 7f603062d740 1 Processor -- start 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.091+0000 7f603062d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b4bd0 0x562e454c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.091+0000 7f603062d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b55c0 0x562e454c4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.091+0000 7f603062d740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454ba730 0x562e454c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f603062d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f60281aff80 con 0x562e454b55c0 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f603062d740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f60281b13f0 con 0x562e454ba730 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f603062d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60281b0650 con 0x562e454b4bd0 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f603062d740 10 client.0 _ll_register_callbacks cb 0x7f6020009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f6026000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b55c0 0x562e454c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f6026000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b55c0 0x562e454c4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38622/0 (socket says 172.21.15.40:38622) 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f6026000640 1 -- 172.21.15.40:0/123588612 learned_addr learned my addr 172.21.15.40:0/123588612 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f6026a00640 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b4bd0 0x562e454c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f6027400640 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454ba730 0x562e454c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f6026000640 1 -- 172.21.15.40:0/123588612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b4bd0 msgr2=0x562e454c3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f6026000640 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b4bd0 0x562e454c3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f6026000640 1 -- 172.21.15.40:0/123588612 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454ba730 msgr2=0x562e454c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f6026000640 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454ba730 0x562e454c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f6026000640 1 -- 172.21.15.40:0/123588612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e451f7620 con 0x562e454b55c0 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f6026000640 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b55c0 0x562e454c4490 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f601409ca00 tx=0x7f601400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e6f9ed12ac416a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.092+0000 7f6010c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.093+0000 7f6010c00640 1 -- 172.21.15.40:0/123588612 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60140a5630 con 0x562e454b55c0 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.093+0000 7f6010c00640 1 -- 172.21.15.40:0/123588612 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f60140bbb80 con 0x562e454b55c0 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.093+0000 7f603062d740 1 -- 172.21.15.40:0/123588612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6018033070 con 0x562e454b55c0 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.093+0000 7f6010c00640 1 -- 172.21.15.40:0/123588612 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f60140bbda0 con 0x562e454b55c0 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.093+0000 7f603062d740 10 client.16062 fetch_fsmap learned FSMap version 24 2024-04-27T00:11:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.093+0000 7f603062d740 1 -- 172.21.15.40:0/123588612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562e453d3cb0 con 0x562e454b55c0 2024-04-27T00:11:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.093+0000 7f6010c00640 1 -- 172.21.15.40:0/123588612 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6018033070 con 0x562e454b55c0 2024-04-27T00:11:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.094+0000 7f6010c00640 1 -- 172.21.15.40:0/123588612 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f60140987c0 con 0x562e454b55c0 2024-04-27T00:11:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.094+0000 7f603062d740 10 client.16062 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:11:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.094+0000 7f603062d740 10 client.16062 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:11:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.094+0000 7f603062d740 20 client.16062 populate_metadata read hostname 'smithi040' 2024-04-27T00:11:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.094+0000 7f603062d740 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562e45531670 0x562e45533a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.094+0000 7f603062d740 4 client.16062 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:11:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.094+0000 7f603062d740 1 -- 172.21.15.40:0/123588612 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e45420e00 con 0x562e45531670 2024-04-27T00:11:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.094+0000 7f6026a00640 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562e45531670 0x562e45533a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.095+0000 7f6026a00640 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562e45531670 0x562e45533a90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2852d6458b982d40 server_cookie=e6d43db9d4f38307 in_seq=0 out_seq=0 2024-04-27T00:11:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.095+0000 7f6010c00640 10 client.16062 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f6010c00640 1 -- 172.21.15.40:0/123588612 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e45420e00 con 0x562e45531670 2024-04-27T00:11:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f6010c00640 10 client.16062 handle_command_reply: tid=0 2024-04-27T00:11:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f603062d740 1 client.16062 shutdown 2024-04-27T00:11:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f603062d740 1 -- 172.21.15.40:0/123588612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b55c0 msgr2=0x562e454c4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f603062d740 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b55c0 0x562e454c4490 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f601409ca00 tx=0x7f601400cf20 comp rx=0 tx=0).stop 2024-04-27T00:11:43.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f603062d740 1 -- 172.21.15.40:0/123588612 shutdown_connections 2024-04-27T00:11:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f603062d740 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x562e45531670 0x562e45533a90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f603062d740 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x562e454ba730 0x562e454c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f6010c00640 0 client.16062 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f603062d740 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562e454b55c0 0x562e454c4490 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f603062d740 1 --2- 172.21.15.40:0/123588612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562e454b4bd0 0x562e454c3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f603062d740 1 -- 172.21.15.40:0/123588612 >> 172.21.15.40:0/123588612 conn(0x562e454b0240 msgr2=0x562e454b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.110+0000 7f603062d740 1 -- 172.21.15.40:0/123588612 shutdown_connections 2024-04-27T00:11:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.111+0000 7f603062d740 1 -- 172.21.15.40:0/123588612 wait complete. 2024-04-27T00:11:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.111+0000 7f603062d740 20 client.16062 trim_cache size 0 max 16384 2024-04-27T00:11:43.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:11:43.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:11:43.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:11:43.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:11:43.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.111+0000 7f602d800640 1 -- 172.21.15.40:0/137347588 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f600c0a3c00 msgr2=0x7f600c0a60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:43.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.111+0000 7f602d800640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f600c0a3c00 0x7f600c0a60c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x562e454b11b0 tx=0x7f6018005f90 comp rx=0 tx=0).stop 2024-04-27T00:11:43.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.111+0000 7f602d800640 1 -- 172.21.15.40:0/137347588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602810a1b0 msgr2=0x7f602819f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.111+0000 7f602d800640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602810a1b0 0x7f602819f5e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f601400e9f0 tx=0x7f601400eeb0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.112+0000 7f602d800640 1 -- 172.21.15.40:0/137347588 shutdown_connections 2024-04-27T00:11:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.112+0000 7f602d800640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f602810ab70 0x7f60281a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.112+0000 7f602d800640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f600c0a3c00 0x7f600c0a60c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.112+0000 7f602d800640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f602810a1b0 0x7f602819f5e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.112+0000 7f602d800640 1 --2- 172.21.15.40:0/137347588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6028108fc0 0x7f602819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.112+0000 7f602d800640 1 -- 172.21.15.40:0/137347588 >> 172.21.15.40:0/137347588 conn(0x7f6028077f80 msgr2=0x7f602810d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.112+0000 7f602d800640 1 -- 172.21.15.40:0/137347588 shutdown_connections 2024-04-27T00:11:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:43.112+0000 7f602d800640 1 -- 172.21.15.40:0/137347588 wait complete. 2024-04-27T00:11:44.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:44 smithi040 ceph-mon[28095]: pgmap v713: 129 pgs: 129 active+clean; 2.8 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:11:44.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:44 smithi080 ceph-mon[32362]: pgmap v713: 129 pgs: 129 active+clean; 2.8 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:11:44.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:44 smithi103 ceph-mon[32952]: pgmap v713: 129 pgs: 129 active+clean; 2.8 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:11:46.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:46 smithi040 ceph-mon[28095]: pgmap v714: 129 pgs: 129 active+clean; 2.8 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-27T00:11:46.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:46 smithi080 ceph-mon[32362]: pgmap v714: 129 pgs: 129 active+clean; 2.8 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-27T00:11:46.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:46 smithi103 ceph-mon[32952]: pgmap v714: 129 pgs: 129 active+clean; 2.8 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-27T00:11:48.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:48 smithi040 ceph-mon[28095]: pgmap v715: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:11:48.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:48 smithi080 ceph-mon[32362]: pgmap v715: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:11:48.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:48 smithi103 ceph-mon[32952]: pgmap v715: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:11:50.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:50 smithi040 ceph-mon[28095]: pgmap v716: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:50.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:50 smithi080 ceph-mon[32362]: pgmap v716: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:50.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:50 smithi103 ceph-mon[32952]: pgmap v716: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:52.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:52 smithi040 ceph-mon[28095]: pgmap v717: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:11:52.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:52 smithi080 ceph-mon[32362]: pgmap v717: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:11:52.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:52 smithi103 ceph-mon[32952]: pgmap v717: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:11:54.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:54 smithi040 ceph-mon[28095]: pgmap v718: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:11:54.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:54 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:11:54.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:54 smithi080 ceph-mon[32362]: pgmap v718: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:11:54.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:54 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:11:54.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:54 smithi103 ceph-mon[32952]: pgmap v718: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:11:54.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:54 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:11:55.440 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.440 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.440 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.441 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.441 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.441 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:55 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:55 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:55.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:55 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:56.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.250+0000 7f928a200640 1 -- 172.21.15.40:0/674695857 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9284108530 msgr2=0x7f928410a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:56.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.250+0000 7f928a200640 1 --2- 172.21.15.40:0/674695857 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9284108530 0x7f928410a920 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f927c009ca0 tx=0x7f927c02a8c0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.250+0000 7f928a200640 1 -- 172.21.15.40:0/674695857 shutdown_connections 2024-04-27T00:11:56.251 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.250+0000 7f928a200640 1 --2- 172.21.15.40:0/674695857 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9284108530 0x7f928410a920 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.250+0000 7f928a200640 1 --2- 172.21.15.40:0/674695857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9284105c00 0x7f9284107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.250+0000 7f928a200640 1 --2- 172.21.15.40:0/674695857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92841032d0 0x7f92841056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.250+0000 7f928a200640 1 -- 172.21.15.40:0/674695857 >> 172.21.15.40:0/674695857 conn(0x7f92840fcc20 msgr2=0x7f92840ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.251+0000 7f928a200640 1 -- 172.21.15.40:0/674695857 shutdown_connections 2024-04-27T00:11:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.251+0000 7f928a200640 1 -- 172.21.15.40:0/674695857 wait complete. 2024-04-27T00:11:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f928a200640 1 Processor -- start 2024-04-27T00:11:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f928a200640 1 -- start start 2024-04-27T00:11:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f928a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92841032d0 0x7f928419abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f928a200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9284105c00 0x7f928419b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f928a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9284108530 0x7f92841a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f928a200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f928410e350 con 0x7f92841032d0 2024-04-27T00:11:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f928a200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f928410e1d0 con 0x7f9284105c00 2024-04-27T00:11:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f928a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f928410e4d0 con 0x7f9284108530 2024-04-27T00:11:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f9283e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9284105c00 0x7f928419b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.254 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f9289800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9284108530 0x7f92841a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f9283e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9284105c00 0x7f928419b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:32832/0 (socket says 172.21.15.40:32832) 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f9283e00640 1 -- 172.21.15.40:0/4034063352 learned_addr learned my addr 172.21.15.40:0/4034063352 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.252+0000 7f9288e00640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92841032d0 0x7f928419abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.253+0000 7f9283e00640 1 -- 172.21.15.40:0/4034063352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9284108530 msgr2=0x7f92841a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.253+0000 7f9283e00640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9284108530 0x7f92841a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.253+0000 7f9283e00640 1 -- 172.21.15.40:0/4034063352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92841032d0 msgr2=0x7f928419abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.253+0000 7f9283e00640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92841032d0 0x7f928419abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.253+0000 7f9283e00640 1 -- 172.21.15.40:0/4034063352 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92841a2800 con 0x7f9284105c00 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.253+0000 7f9283e00640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9284105c00 0x7f928419b120 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f927000e9f0 tx=0x7f927000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.253+0000 7f9281600640 1 -- 172.21.15.40:0/4034063352 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927000ccc0 con 0x7f9284105c00 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.253+0000 7f9281600640 1 -- 172.21.15.40:0/4034063352 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92700054d0 con 0x7f9284105c00 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.253+0000 7f9281600640 1 -- 172.21.15.40:0/4034063352 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9270010430 con 0x7f9284105c00 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.253+0000 7f928a200640 1 -- 172.21.15.40:0/4034063352 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92841a2af0 con 0x7f9284105c00 2024-04-27T00:11:56.255 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.254+0000 7f928a200640 1 -- 172.21.15.40:0/4034063352 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92841a2ff0 con 0x7f9284105c00 2024-04-27T00:11:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.257+0000 7f928e61f740 1 Processor -- start 2024-04-27T00:11:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f928e61f740 1 -- start start 2024-04-27T00:11:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f9281600640 1 -- 172.21.15.40:0/4034063352 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f92700046c0 con 0x7f9284105c00 2024-04-27T00:11:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f928e61f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb934a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f928e61f740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb934f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f928e61f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb934f940 0x563eb9351dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f928e61f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563eb92ae380 con 0x563eb934f940 2024-04-27T00:11:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f928e61f740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x563eb924ac30 con 0x563eb934a7d0 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f928e61f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563eb92aa340 con 0x563eb9349de0 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f9288e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb934f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f9289800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb934a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f9288e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb934f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:32836/0 (socket says 172.21.15.40:32836) 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f9288e00640 1 -- 172.21.15.40:0/2022136605 learned_addr learned my addr 172.21.15.40:0/2022136605 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f9289800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb934a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50566/0 (socket says 172.21.15.40:50566) 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.258+0000 7f9283e00640 1 --2- 172.21.15.40:0/2022136605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb934f940 0x563eb9351dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.259+0000 7f9288e00640 1 -- 172.21.15.40:0/2022136605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 msgr2=0x563eb934a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.259+0000 7f9288e00640 1 --2- 172.21.15.40:0/2022136605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb934a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.259+0000 7f9288e00640 1 -- 172.21.15.40:0/2022136605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb934f940 msgr2=0x563eb9351dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.259+0000 7f9288e00640 1 --2- 172.21.15.40:0/2022136605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb934f940 0x563eb9351dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.259+0000 7f9288e00640 1 -- 172.21.15.40:0/2022136605 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563eb926c940 con 0x563eb934a7d0 2024-04-27T00:11:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.259+0000 7f9289800640 1 --2- 172.21.15.40:0/2022136605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb934a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:11:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.259+0000 7f9288e00640 1 --2- 172.21.15.40:0/2022136605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb934f2d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9278009c50 tx=0x7f9278010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33ddb6f672b7bf41 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.259+0000 7f9276000640 1 -- 172.21.15.40:0/2022136605 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278004020 con 0x563eb934a7d0 2024-04-27T00:11:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.259+0000 7f9276000640 1 -- 172.21.15.40:0/2022136605 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92780041c0 con 0x563eb934a7d0 2024-04-27T00:11:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.260+0000 7f9276000640 1 -- 172.21.15.40:0/2022136605 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92780126b0 con 0x563eb934a7d0 2024-04-27T00:11:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.260+0000 7f9281600640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f926c0a3cb0 0x7f926c0a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.260+0000 7f9281600640 1 -- 172.21.15.40:0/4034063352 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f92700d0cd0 con 0x7f9284105c00 2024-04-27T00:11:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.260+0000 7f9289800640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f926c0a3cb0 0x7f926c0a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.260+0000 7f928e61f740 1 -- 172.21.15.40:0/2022136605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 msgr2=0x563eb934f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.260+0000 7f928e61f740 1 --2- 172.21.15.40:0/2022136605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb934f2d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9278009c50 tx=0x7f9278010740 comp rx=0 tx=0).stop 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.260+0000 7f928e61f740 1 -- 172.21.15.40:0/2022136605 shutdown_connections 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.260+0000 7f928e61f740 1 --2- 172.21.15.40:0/2022136605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb934f940 0x563eb9351dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.260+0000 7f928e61f740 1 --2- 172.21.15.40:0/2022136605 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb934f2d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.260+0000 7f928e61f740 1 --2- 172.21.15.40:0/2022136605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb934a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.260+0000 7f928e61f740 1 -- 172.21.15.40:0/2022136605 >> 172.21.15.40:0/2022136605 conn(0x563eb9345450 msgr2=0x563eb9345850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.261+0000 7f928e61f740 1 -- 172.21.15.40:0/2022136605 shutdown_connections 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.261+0000 7f928e61f740 1 -- 172.21.15.40:0/2022136605 wait complete. 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.261+0000 7f9289800640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f926c0a3cb0 0x7f926c0a6170 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f927c005e30 tx=0x7f927c031af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.261+0000 7f928e61f740 1 -- start start 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.261+0000 7f928e61f740 1 Processor -- start 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f928e61f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb9354080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f928e61f740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb93545c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f928e61f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb934f940 0x563eb935b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f928e61f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563eb93528f0 con 0x563eb934f940 2024-04-27T00:11:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f928e61f740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x563eb9235c60 con 0x563eb934a7d0 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f928e61f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563eb9352a70 con 0x563eb9349de0 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f928e61f740 10 client.0 _ll_register_callbacks cb 0x563eb9261e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f9288e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb9354080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f9288e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb9354080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50570/0 (socket says 172.21.15.40:50570) 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f9283e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb93545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f9288e00640 1 -- 172.21.15.40:0/783662166 learned_addr learned my addr 172.21.15.40:0/783662166 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f9289800640 1 --2- 172.21.15.40:0/783662166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb934f940 0x563eb935b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f9288e00640 1 -- 172.21.15.40:0/783662166 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 msgr2=0x563eb93545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f9288e00640 1 --2- 172.21.15.40:0/783662166 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb93545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f9288e00640 1 -- 172.21.15.40:0/783662166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb934f940 msgr2=0x563eb935b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.262+0000 7f9288e00640 1 --2- 172.21.15.40:0/783662166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb934f940 0x563eb935b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.263+0000 7f9288e00640 1 -- 172.21.15.40:0/783662166 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563eb9244510 con 0x563eb9349de0 2024-04-27T00:11:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.263+0000 7f9288e00640 1 --2- 172.21.15.40:0/783662166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb9354080 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f92780097d0 tx=0x7f9278005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b50582b0257a9dbf server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.263+0000 7f926aa00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:11:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.263+0000 7f926aa00640 1 -- 172.21.15.40:0/783662166 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927802a060 con 0x563eb9349de0 2024-04-27T00:11:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.263+0000 7f928e61f740 1 -- 172.21.15.40:0/783662166 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92780041c0 con 0x563eb9349de0 2024-04-27T00:11:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.263+0000 7f926aa00640 1 -- 172.21.15.40:0/783662166 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f9278005050 con 0x563eb9349de0 2024-04-27T00:11:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.263+0000 7f926aa00640 1 -- 172.21.15.40:0/783662166 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9278031050 con 0x563eb9349de0 2024-04-27T00:11:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.263+0000 7f928e61f740 10 client.26218 fetch_fsmap learned FSMap version 24 2024-04-27T00:11:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.263+0000 7f928e61f740 1 -- 172.21.15.40:0/783662166 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563eb8fc0210 con 0x563eb9349de0 2024-04-27T00:11:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.264+0000 7f926aa00640 1 -- 172.21.15.40:0/783662166 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92780041c0 con 0x563eb9349de0 2024-04-27T00:11:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.264+0000 7f926aa00640 1 -- 172.21.15.40:0/783662166 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f927800f040 con 0x563eb9349de0 2024-04-27T00:11:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.264+0000 7f928e61f740 10 client.26218 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:11:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.264+0000 7f928e61f740 10 client.26218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:11:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.264+0000 7f928e61f740 20 client.26218 populate_metadata read hostname 'smithi040' 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.264+0000 7f928e61f740 1 --2- 172.21.15.40:0/783662166 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563eb935f7f0 0x563eb9361be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.264+0000 7f928e61f740 4 client.26218 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.264+0000 7f928e61f740 1 -- 172.21.15.40:0/783662166 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563eb92bbf90 con 0x563eb935f7f0 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.264+0000 7f9283e00640 1 --2- 172.21.15.40:0/783662166 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563eb935f7f0 0x563eb9361be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.265+0000 7f9283e00640 1 --2- 172.21.15.40:0/783662166 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563eb935f7f0 0x563eb9361be0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e71b9bdfeed5265 server_cookie=75581458fde12d56 in_seq=0 out_seq=0 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.265+0000 7f926aa00640 10 client.26218 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.265+0000 7f926aa00640 1 -- 172.21.15.40:0/783662166 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563eb92bbf90 con 0x563eb935f7f0 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.265+0000 7f926aa00640 10 client.26218 handle_command_reply: tid=0 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.265+0000 7f928e61f740 1 client.26218 shutdown 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.266+0000 7f928e61f740 1 -- 172.21.15.40:0/783662166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 msgr2=0x563eb9354080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.266+0000 7f928e61f740 1 --2- 172.21.15.40:0/783662166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb9354080 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f92780097d0 tx=0x7f9278005d30 comp rx=0 tx=0).stop 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.266+0000 7f928e61f740 1 -- 172.21.15.40:0/783662166 shutdown_connections 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.266+0000 7f928e61f740 1 --2- 172.21.15.40:0/783662166 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563eb935f7f0 0x563eb9361be0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.266+0000 7f928e61f740 1 --2- 172.21.15.40:0/783662166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb934f940 0x563eb935b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.266+0000 7f928e61f740 1 --2- 172.21.15.40:0/783662166 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb93545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.266+0000 7f926aa00640 0 client.26218 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.266+0000 7f928e61f740 1 --2- 172.21.15.40:0/783662166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb9354080 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.266+0000 7f928e61f740 1 -- 172.21.15.40:0/783662166 >> 172.21.15.40:0/783662166 conn(0x563eb9345450 msgr2=0x563eb9344650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.266+0000 7f928e61f740 1 -- 172.21.15.40:0/783662166 shutdown_connections 2024-04-27T00:11:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.266+0000 7f928e61f740 1 -- 172.21.15.40:0/783662166 wait complete. 2024-04-27T00:11:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.266+0000 7f928e61f740 20 client.26218 trim_cache size 0 max 16384 2024-04-27T00:11:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.282+0000 7f928e61f740 1 Processor -- start 2024-04-27T00:11:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.282+0000 7f928e61f740 1 -- start start 2024-04-27T00:11:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f928e61f740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb9345450 0x563eb9348bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f928e61f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb9349de0 0x563eb935b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f928e61f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb934a7d0 0x563eb93536a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f928e61f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f92841afe30 con 0x563eb9349de0 2024-04-27T00:11:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f928e61f740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f92841ad700 con 0x563eb9345450 2024-04-27T00:11:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f928e61f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f92841ab550 con 0x563eb934a7d0 2024-04-27T00:11:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f9288e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb9345450 0x563eb9348bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f9288e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb9345450 0x563eb9348bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:32856/0 (socket says 172.21.15.40:32856) 2024-04-27T00:11:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f9288e00640 1 -- 172.21.15.40:0/3456629252 learned_addr learned my addr 172.21.15.40:0/3456629252 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f9289800640 1 --2- 172.21.15.40:0/3456629252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb934a7d0 0x563eb93536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f9283e00640 1 --2- 172.21.15.40:0/3456629252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb9349de0 0x563eb935b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f9288e00640 1 -- 172.21.15.40:0/3456629252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb934a7d0 msgr2=0x563eb93536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f9288e00640 1 --2- 172.21.15.40:0/3456629252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb934a7d0 0x563eb93536a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f9288e00640 1 -- 172.21.15.40:0/3456629252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb9349de0 msgr2=0x563eb935b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f9288e00640 1 --2- 172.21.15.40:0/3456629252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb9349de0 0x563eb935b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f9288e00640 1 -- 172.21.15.40:0/3456629252 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563eb908c620 con 0x563eb9345450 2024-04-27T00:11:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.283+0000 7f9283e00640 1 --2- 172.21.15.40:0/3456629252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb9349de0 0x563eb935b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:11:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.284+0000 7f9288e00640 1 --2- 172.21.15.40:0/3456629252 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb9345450 0x563eb9348bd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9278004c10 tx=0x7f9278005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45764adf9c8350f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.284+0000 7f9275600640 1 -- 172.21.15.40:0/3456629252 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927802a040 con 0x563eb9345450 2024-04-27T00:11:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.284+0000 7f9275600640 1 -- 172.21.15.40:0/3456629252 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9278009d40 con 0x563eb9345450 2024-04-27T00:11:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.284+0000 7f9275600640 1 -- 172.21.15.40:0/3456629252 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278003d50 con 0x563eb9345450 2024-04-27T00:11:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.284+0000 7f928e61f740 1 -- 172.21.15.40:0/3456629252 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb9345450 msgr2=0x563eb9348bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.284+0000 7f928e61f740 1 --2- 172.21.15.40:0/3456629252 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb9345450 0x563eb9348bd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9278004c10 tx=0x7f9278005f70 comp rx=0 tx=0).stop 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.285+0000 7f928e61f740 1 -- 172.21.15.40:0/3456629252 shutdown_connections 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.285+0000 7f928e61f740 1 --2- 172.21.15.40:0/3456629252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb934a7d0 0x563eb93536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.285+0000 7f928e61f740 1 --2- 172.21.15.40:0/3456629252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb9349de0 0x563eb935b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.285+0000 7f928e61f740 1 --2- 172.21.15.40:0/3456629252 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb9345450 0x563eb9348bd0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.285+0000 7f928e61f740 1 -- 172.21.15.40:0/3456629252 >> 172.21.15.40:0/3456629252 conn(0x563eb934f410 msgr2=0x563eb93484c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.285+0000 7f928e61f740 1 -- 172.21.15.40:0/3456629252 shutdown_connections 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.285+0000 7f928e61f740 1 -- 172.21.15.40:0/3456629252 wait complete. 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.285+0000 7f928e61f740 1 -- start start 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.285+0000 7f928e61f740 1 Processor -- start 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f928e61f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb9345450 0x563eb9358450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f928e61f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb9358990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f928e61f740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb93cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f928e61f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f92841abbc0 con 0x563eb9345450 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f928e61f740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f92841ad030 con 0x563eb934a7d0 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f928e61f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f92841ac290 con 0x563eb9349de0 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f928e61f740 10 client.0 _ll_register_callbacks cb 0x7f9270005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f9283e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb9358990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f9289800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb93cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f9283e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb9358990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50586/0 (socket says 172.21.15.40:50586) 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f9288e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb9345450 0x563eb9358450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f9283e00640 1 -- 172.21.15.40:0/809600162 learned_addr learned my addr 172.21.15.40:0/809600162 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f9283e00640 1 -- 172.21.15.40:0/809600162 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 msgr2=0x563eb93cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f9283e00640 1 --2- 172.21.15.40:0/809600162 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb93cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f9283e00640 1 -- 172.21.15.40:0/809600162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb9345450 msgr2=0x563eb9358450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f9283e00640 1 --2- 172.21.15.40:0/809600162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb9345450 0x563eb9358450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.286+0000 7f9283e00640 1 -- 172.21.15.40:0/809600162 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563eb9268cb0 con 0x563eb9349de0 2024-04-27T00:11:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f9283e00640 1 --2- 172.21.15.40:0/809600162 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb9358990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f927009c7b0 tx=0x7f9270004a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10348d63938b1b89 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:11:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f926aa00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:11:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f926aa00640 1 -- 172.21.15.40:0/809600162 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92700a0e60 con 0x563eb9349de0 2024-04-27T00:11:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f928e61f740 1 -- 172.21.15.40:0/809600162 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9278009d40 con 0x563eb9349de0 2024-04-27T00:11:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f926aa00640 1 -- 172.21.15.40:0/809600162 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f92700ce530 con 0x563eb9349de0 2024-04-27T00:11:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f926aa00640 1 -- 172.21.15.40:0/809600162 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f92700a5720 con 0x563eb9349de0 2024-04-27T00:11:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f928e61f740 10 client.26224 fetch_fsmap learned FSMap version 24 2024-04-27T00:11:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f928e61f740 1 -- 172.21.15.40:0/809600162 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563eb93cdc80 con 0x563eb9349de0 2024-04-27T00:11:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f926aa00640 1 -- 172.21.15.40:0/809600162 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9278009d40 con 0x563eb9349de0 2024-04-27T00:11:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f926aa00640 1 -- 172.21.15.40:0/809600162 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9270004fe0 con 0x563eb9349de0 2024-04-27T00:11:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f928e61f740 10 client.26224 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:11:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f928e61f740 10 client.26224 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:11:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f928e61f740 20 client.26224 populate_metadata read hostname 'smithi040' 2024-04-27T00:11:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f928e61f740 1 --2- 172.21.15.40:0/809600162 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563eb934f940 0x563eb93d0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:11:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f928e61f740 4 client.26224 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:11:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.287+0000 7f928e61f740 1 -- 172.21.15.40:0/809600162 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563eb92b5e00 con 0x563eb934f940 2024-04-27T00:11:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.288+0000 7f9288e00640 1 --2- 172.21.15.40:0/809600162 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563eb934f940 0x563eb93d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:11:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.288+0000 7f9288e00640 1 --2- 172.21.15.40:0/809600162 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563eb934f940 0x563eb93d0620 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9a30750b5e1ae16 server_cookie=2deaa78b59918339 in_seq=0 out_seq=0 2024-04-27T00:11:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.288+0000 7f926aa00640 10 client.26224 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.302+0000 7f926aa00640 1 -- 172.21.15.40:0/809600162 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563eb92b5e00 con 0x563eb934f940 2024-04-27T00:11:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.302+0000 7f926aa00640 10 client.26224 handle_command_reply: tid=0 2024-04-27T00:11:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.302+0000 7f928e61f740 1 client.26224 shutdown 2024-04-27T00:11:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.302+0000 7f928e61f740 1 -- 172.21.15.40:0/809600162 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 msgr2=0x563eb9358990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.302+0000 7f928e61f740 1 --2- 172.21.15.40:0/809600162 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb9358990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f927009c7b0 tx=0x7f9270004a60 comp rx=0 tx=0).stop 2024-04-27T00:11:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.302+0000 7f928e61f740 1 -- 172.21.15.40:0/809600162 shutdown_connections 2024-04-27T00:11:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.302+0000 7f928e61f740 1 --2- 172.21.15.40:0/809600162 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x563eb934f940 0x563eb93d0620 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.302+0000 7f928e61f740 1 --2- 172.21.15.40:0/809600162 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x563eb934a7d0 0x563eb93cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.302+0000 7f928e61f740 1 --2- 172.21.15.40:0/809600162 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eb9349de0 0x563eb9358990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.302+0000 7f926aa00640 0 client.26224 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:11:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.302+0000 7f928e61f740 1 --2- 172.21.15.40:0/809600162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eb9345450 0x563eb9358450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.302+0000 7f928e61f740 1 -- 172.21.15.40:0/809600162 >> 172.21.15.40:0/809600162 conn(0x563eb934f410 msgr2=0x563eb93484c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.303+0000 7f928e61f740 1 -- 172.21.15.40:0/809600162 shutdown_connections 2024-04-27T00:11:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.303+0000 7f928e61f740 1 -- 172.21.15.40:0/809600162 wait complete. 2024-04-27T00:11:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.303+0000 7f928e61f740 20 client.26224 trim_cache size 0 max 16384 2024-04-27T00:11:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:11:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:11:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:11:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:11:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.304+0000 7f928a200640 1 -- 172.21.15.40:0/4034063352 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f926c0a3cb0 msgr2=0x7f926c0a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.304+0000 7f928a200640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f926c0a3cb0 0x7f926c0a6170 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f927c005e30 tx=0x7f927c031af0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.304+0000 7f928a200640 1 -- 172.21.15.40:0/4034063352 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9284105c00 msgr2=0x7f928419b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:11:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.304+0000 7f928a200640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9284105c00 0x7f928419b120 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f927000e9f0 tx=0x7f927000eeb0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.304+0000 7f928a200640 1 -- 172.21.15.40:0/4034063352 shutdown_connections 2024-04-27T00:11:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.304+0000 7f928a200640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f926c0a3cb0 0x7f926c0a6170 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.304+0000 7f928a200640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9284108530 0x7f92841a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.304+0000 7f928a200640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9284105c00 0x7f928419b120 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.304+0000 7f928a200640 1 --2- 172.21.15.40:0/4034063352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f92841032d0 0x7f928419abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:11:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.304+0000 7f928a200640 1 -- 172.21.15.40:0/4034063352 >> 172.21.15.40:0/4034063352 conn(0x7f92840fcc20 msgr2=0x7f9284102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:11:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.304+0000 7f928a200640 1 -- 172.21.15.40:0/4034063352 shutdown_connections 2024-04-27T00:11:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:11:56.304+0000 7f928a200640 1 -- 172.21.15.40:0/4034063352 wait complete. 2024-04-27T00:11:57.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:56 smithi080 ceph-mon[32362]: pgmap v719: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:57.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:57.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:57.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:57.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:56 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:57.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:56 smithi103 ceph-mon[32952]: pgmap v719: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:57.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:57.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:57.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:57.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:56 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:57.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:56 smithi040 ceph-mon[28095]: pgmap v719: 129 pgs: 129 active+clean; 2.9 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:11:57.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:57.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:57.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:57.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:56 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:58.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:58 smithi040 ceph-mon[28095]: pgmap v720: 129 pgs: 129 active+clean; 3.0 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:11:58.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:58 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:58.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:58 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:59.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:58 smithi080 ceph-mon[32362]: pgmap v720: 129 pgs: 129 active+clean; 3.0 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:11:59.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:58 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:59.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:58 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:59.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:58 smithi103 ceph-mon[32952]: pgmap v720: 129 pgs: 129 active+clean; 3.0 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:11:59.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:58 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:59.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:58 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:11:59.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:59 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:11:59.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:59 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:11:59.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:11:59 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:12:00.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:59 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:12:00.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:59 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:12:00.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:11:59 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:12:00.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:59 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:12:00.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:59 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:12:00.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:11:59 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:12:00.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:00 smithi040 ceph-mon[28095]: pgmap v721: 129 pgs: 129 active+clean; 3.0 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:12:00.998 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:00 smithi103 ceph-mon[32952]: pgmap v721: 129 pgs: 129 active+clean; 3.0 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:12:01.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:00 smithi080 ceph-mon[32362]: pgmap v721: 129 pgs: 129 active+clean; 3.0 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-27T00:12:02.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:02 smithi040 ceph-mon[28095]: pgmap v722: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:12:03.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:02 smithi080 ceph-mon[32362]: pgmap v722: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:12:03.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:02 smithi103 ceph-mon[32952]: pgmap v722: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-27T00:12:04.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:04 smithi040 ceph-mon[28095]: pgmap v723: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:12:05.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:04 smithi080 ceph-mon[32362]: pgmap v723: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:12:05.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:04 smithi103 ceph-mon[32952]: pgmap v723: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:12:06.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:06 smithi040 ceph-mon[28095]: pgmap v724: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:07.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:06 smithi080 ceph-mon[32362]: pgmap v724: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:07.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:06 smithi103 ceph-mon[32952]: pgmap v724: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:08.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:08 smithi040 ceph-mon[28095]: pgmap v725: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:12:09.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:08 smithi080 ceph-mon[32362]: pgmap v725: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:12:09.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:08 smithi103 ceph-mon[32952]: pgmap v725: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:12:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.450+0000 7f3bff200640 1 -- 172.21.15.40:0/1862512054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c0010a990 msgr2=0x7f3c00111220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.450+0000 7f3bff200640 1 --2- 172.21.15.40:0/1862512054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c0010a990 0x7f3c00111220 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8009ca0 tx=0x7f3bf802a8b0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.450+0000 7f3bff200640 1 -- 172.21.15.40:0/1862512054 shutdown_connections 2024-04-27T00:12:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.450+0000 7f3bff200640 1 --2- 172.21.15.40:0/1862512054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c0010a990 0x7f3c00111220 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.450+0000 7f3bff200640 1 --2- 172.21.15.40:0/1862512054 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c00109fd0 0x7f3c0010a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.450+0000 7f3bff200640 1 --2- 172.21.15.40:0/1862512054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c00108de0 0x7f3c00109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.451 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.450+0000 7f3bff200640 1 -- 172.21.15.40:0/1862512054 >> 172.21.15.40:0/1862512054 conn(0x7f3c00077dc0 msgr2=0x7f3c0007a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.450+0000 7f3bff200640 1 -- 172.21.15.40:0/1862512054 shutdown_connections 2024-04-27T00:12:09.452 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.450+0000 7f3bff200640 1 -- 172.21.15.40:0/1862512054 wait complete. 2024-04-27T00:12:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.451+0000 7f3bff200640 1 Processor -- start 2024-04-27T00:12:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.451+0000 7f3bff200640 1 -- start start 2024-04-27T00:12:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.452+0000 7f3bff200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c00108de0 0x7f3c0019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.452+0000 7f3bff200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c00109fd0 0x7f3c0019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.452+0000 7f3bff200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c0010a990 0x7f3c001a6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.452+0000 7f3bff200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c00114940 con 0x7f3c00108de0 2024-04-27T00:12:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.452+0000 7f3bff200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f3c001147c0 con 0x7f3c0010a990 2024-04-27T00:12:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.452+0000 7f3bff200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c00114ac0 con 0x7f3c00109fd0 2024-04-27T00:12:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.452+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c00108de0 0x7f3c0019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.452+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c00108de0 0x7f3c0019ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42480/0 (socket says 172.21.15.40:42480) 2024-04-27T00:12:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.452+0000 7f3bfde00640 1 -- 172.21.15.40:0/833196082 learned_addr learned my addr 172.21.15.40:0/833196082 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.452+0000 7f3bfd400640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c00109fd0 0x7f3c0019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.452+0000 7f3bfe800640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c0010a990 0x7f3c001a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.453+0000 7f3bfd400640 1 -- 172.21.15.40:0/833196082 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c0010a990 msgr2=0x7f3c001a6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.453+0000 7f3bfd400640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c0010a990 0x7f3c001a6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.453+0000 7f3bfd400640 1 -- 172.21.15.40:0/833196082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c00108de0 msgr2=0x7f3c0019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.453+0000 7f3bfd400640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c00108de0 0x7f3c0019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.453+0000 7f3bfd400640 1 -- 172.21.15.40:0/833196082 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c001a6b80 con 0x7f3c00109fd0 2024-04-27T00:12:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.453+0000 7f3bfde00640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c00108de0 0x7f3c0019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:12:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.453+0000 7f3bfd400640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c00109fd0 0x7f3c0019f4a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00e9f0 tx=0x7f3bec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.453+0000 7f3bf6a00640 1 -- 172.21.15.40:0/833196082 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec00cd80 con 0x7f3c00109fd0 2024-04-27T00:12:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.453+0000 7f3bf6a00640 1 -- 172.21.15.40:0/833196082 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bec0035f0 con 0x7f3c00109fd0 2024-04-27T00:12:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.453+0000 7f3bff200640 1 -- 172.21.15.40:0/833196082 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c001a6e70 con 0x7f3c00109fd0 2024-04-27T00:12:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.453+0000 7f3bf6a00640 1 -- 172.21.15.40:0/833196082 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec010690 con 0x7f3c00109fd0 2024-04-27T00:12:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.453+0000 7f3bff200640 1 -- 172.21.15.40:0/833196082 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c001a7330 con 0x7f3c00109fd0 2024-04-27T00:12:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.456+0000 7f3c0786e740 1 Processor -- start 2024-04-27T00:12:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3c0786e740 1 -- start start 2024-04-27T00:12:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3c0786e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 0x559de3783200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3c0786e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de37882d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3c0786e740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3788940 0x559de378add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3c0786e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559de36e7380 con 0x559de3782de0 2024-04-27T00:12:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3c0786e740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x559de3683c30 con 0x559de3788940 2024-04-27T00:12:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3c0786e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559de36e3340 con 0x559de37837d0 2024-04-27T00:12:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3bf6a00640 1 -- 172.21.15.40:0/833196082 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3bec0037a0 con 0x7f3c00109fd0 2024-04-27T00:12:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3bfe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 0x559de3783200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3bfe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 0x559de3783200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42490/0 (socket says 172.21.15.40:42490) 2024-04-27T00:12:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3bfe800640 1 -- 172.21.15.40:0/4058763052 learned_addr learned my addr 172.21.15.40:0/4058763052 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3bfd400640 1 --2- 172.21.15.40:0/4058763052 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3788940 0x559de378add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.457+0000 7f3bfde00640 1 --2- 172.21.15.40:0/4058763052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de37882d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.458+0000 7f3bfe800640 1 -- 172.21.15.40:0/4058763052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 msgr2=0x559de37882d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.458+0000 7f3bfe800640 1 --2- 172.21.15.40:0/4058763052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de37882d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.458+0000 7f3bfe800640 1 -- 172.21.15.40:0/4058763052 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3788940 msgr2=0x559de378add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.458+0000 7f3bfe800640 1 --2- 172.21.15.40:0/4058763052 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3788940 0x559de378add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.458+0000 7f3bfe800640 1 -- 172.21.15.40:0/4058763052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559de36a5940 con 0x559de3782de0 2024-04-27T00:12:09.461 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.458+0000 7f3bfe800640 1 --2- 172.21.15.40:0/4058763052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 0x559de3783200 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8035000 tx=0x7f3bf8009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a04f8b030b3a7de server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.458+0000 7f3beb000640 1 -- 172.21.15.40:0/4058763052 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8030900 con 0x559de3782de0 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.458+0000 7f3beb000640 1 -- 172.21.15.40:0/4058763052 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bf8036030 con 0x559de3782de0 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.458+0000 7f3beb000640 1 -- 172.21.15.40:0/4058763052 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8004440 con 0x559de3782de0 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.459+0000 7f3bf6a00640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3be40a3c30 0x7f3be40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.459+0000 7f3bf6a00640 1 -- 172.21.15.40:0/833196082 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f3bec0d0c10 con 0x7f3c00109fd0 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.459+0000 7f3bfde00640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3be40a3c30 0x7f3be40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.459+0000 7f3c0786e740 1 -- 172.21.15.40:0/4058763052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 msgr2=0x559de3783200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.459+0000 7f3c0786e740 1 --2- 172.21.15.40:0/4058763052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 0x559de3783200 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8035000 tx=0x7f3bf8009a20 comp rx=0 tx=0).stop 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.459+0000 7f3c0786e740 1 -- 172.21.15.40:0/4058763052 shutdown_connections 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.459+0000 7f3c0786e740 1 --2- 172.21.15.40:0/4058763052 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3788940 0x559de378add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.459+0000 7f3c0786e740 1 --2- 172.21.15.40:0/4058763052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de37882d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.459+0000 7f3c0786e740 1 --2- 172.21.15.40:0/4058763052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 0x559de3783200 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.462 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.459+0000 7f3c0786e740 1 -- 172.21.15.40:0/4058763052 >> 172.21.15.40:0/4058763052 conn(0x559de377e450 msgr2=0x559de377e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.460+0000 7f3c0786e740 1 -- 172.21.15.40:0/4058763052 shutdown_connections 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.460+0000 7f3c0786e740 1 -- 172.21.15.40:0/4058763052 wait complete. 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.460+0000 7f3bfde00640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3be40a3c30 0x7f3be40a60f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x559de3783bd0 tx=0x7f3bf0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.460+0000 7f3c0786e740 1 -- start start 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.460+0000 7f3c0786e740 1 Processor -- start 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.461+0000 7f3c0786e740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3782de0 0x559de378cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.461+0000 7f3c0786e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de378d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.461+0000 7f3c0786e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 0x559de37944e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.461+0000 7f3c0786e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559de378b890 con 0x559de3788940 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.461+0000 7f3c0786e740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x559de366ec60 con 0x559de3782de0 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.461+0000 7f3c0786e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559de378ba10 con 0x559de37837d0 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.461+0000 7f3c0786e740 10 client.0 _ll_register_callbacks cb 0x559de369ae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.461+0000 7f3bfe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 0x559de37944e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.461+0000 7f3bfe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 0x559de37944e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42506/0 (socket says 172.21.15.40:42506) 2024-04-27T00:12:09.463 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.461+0000 7f3bfe800640 1 -- 172.21.15.40:0/88652188 learned_addr learned my addr 172.21.15.40:0/88652188 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:09.464 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.461+0000 7f3bfde00640 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3782de0 0x559de378cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.464 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.461+0000 7f3bfd400640 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de378d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.462+0000 7f3bfe800640 1 -- 172.21.15.40:0/88652188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 msgr2=0x559de378d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:09.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.462+0000 7f3bfe800640 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de378d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.462+0000 7f3bfe800640 1 -- 172.21.15.40:0/88652188 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3782de0 msgr2=0x559de378cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:09.469 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.462+0000 7f3bfe800640 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3782de0 0x559de378cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.462+0000 7f3bfe800640 1 -- 172.21.15.40:0/88652188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559de367d510 con 0x559de3788940 2024-04-27T00:12:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.462+0000 7f3bfd400640 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de378d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:12:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.462+0000 7f3bfde00640 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3782de0 0x559de378cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:12:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.462+0000 7f3bfe800640 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 0x559de37944e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8006010 tx=0x7f3bf80095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8110104886fc1e63 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.462+0000 7f3bdf400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:12:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.462+0000 7f3bdf400640 1 -- 172.21.15.40:0/88652188 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8053020 con 0x559de3788940 2024-04-27T00:12:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.462+0000 7f3bdf400640 1 -- 172.21.15.40:0/88652188 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f3bf8003970 con 0x559de3788940 2024-04-27T00:12:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.462+0000 7f3c0786e740 1 -- 172.21.15.40:0/88652188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bf8036030 con 0x559de3788940 2024-04-27T00:12:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.463+0000 7f3bdf400640 1 -- 172.21.15.40:0/88652188 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3bf8003bb0 con 0x559de3788940 2024-04-27T00:12:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.463+0000 7f3c0786e740 10 client.16089 fetch_fsmap learned FSMap version 24 2024-04-27T00:12:09.470 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.463+0000 7f3c0786e740 1 -- 172.21.15.40:0/88652188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559de33f9210 con 0x559de3788940 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.463+0000 7f3bdf400640 1 -- 172.21.15.40:0/88652188 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bf8036030 con 0x559de3788940 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.463+0000 7f3bdf400640 1 -- 172.21.15.40:0/88652188 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3bf8043070 con 0x559de3788940 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.463+0000 7f3c0786e740 10 client.16089 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.463+0000 7f3c0786e740 10 client.16089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.463+0000 7f3c0786e740 20 client.16089 populate_metadata read hostname 'smithi040' 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.463+0000 7f3c0786e740 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559de3798690 0x559de379aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.463+0000 7f3c0786e740 4 client.16089 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.463+0000 7f3c0786e740 1 -- 172.21.15.40:0/88652188 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559de36f4f90 con 0x559de3798690 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.464+0000 7f3bfde00640 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559de3798690 0x559de379aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.464+0000 7f3bfde00640 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559de3798690 0x559de379aa80 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd06eb382e60b6eb server_cookie=2c9bcd3d9f107d55 in_seq=0 out_seq=0 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.464+0000 7f3bdf400640 10 client.16089 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.465+0000 7f3bdf400640 1 -- 172.21.15.40:0/88652188 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559de36f4f90 con 0x559de3798690 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.465+0000 7f3bdf400640 10 client.16089 handle_command_reply: tid=0 2024-04-27T00:12:09.471 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.465+0000 7f3c0786e740 1 client.16089 shutdown 2024-04-27T00:12:09.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.465+0000 7f3c0786e740 1 -- 172.21.15.40:0/88652188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 msgr2=0x559de37944e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:09.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.465+0000 7f3c0786e740 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 0x559de37944e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8006010 tx=0x7f3bf80095e0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.466+0000 7f3c0786e740 1 -- 172.21.15.40:0/88652188 shutdown_connections 2024-04-27T00:12:09.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.466+0000 7f3c0786e740 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559de3798690 0x559de379aa80 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.466+0000 7f3c0786e740 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 0x559de37944e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.466+0000 7f3c0786e740 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de378d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.466+0000 7f3bdf400640 0 client.16089 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:09.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.466+0000 7f3c0786e740 1 --2- 172.21.15.40:0/88652188 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3782de0 0x559de378cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.466+0000 7f3c0786e740 1 -- 172.21.15.40:0/88652188 >> 172.21.15.40:0/88652188 conn(0x559de377e450 msgr2=0x559de377d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:09.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.466+0000 7f3c0786e740 1 -- 172.21.15.40:0/88652188 shutdown_connections 2024-04-27T00:12:09.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.466+0000 7f3c0786e740 1 -- 172.21.15.40:0/88652188 wait complete. 2024-04-27T00:12:09.472 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.466+0000 7f3c0786e740 20 client.16089 trim_cache size 0 max 16384 2024-04-27T00:12:09.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.482+0000 7f3c0786e740 1 Processor -- start 2024-04-27T00:12:09.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.482+0000 7f3c0786e740 1 -- start start 2024-04-27T00:12:09.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3c0786e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 0x559de378cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3c0786e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de37944a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3c0786e740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3788940 0x559de378d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3c0786e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c001b40a0 con 0x559de3782de0 2024-04-27T00:12:09.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3c0786e740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f3c001b1970 con 0x559de3788940 2024-04-27T00:12:09.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3c0786e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c001af7c0 con 0x559de37837d0 2024-04-27T00:12:09.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 0x559de378cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 0x559de378cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42514/0 (socket says 172.21.15.40:42514) 2024-04-27T00:12:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3bfde00640 1 -- 172.21.15.40:0/2789598546 learned_addr learned my addr 172.21.15.40:0/2789598546 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3bfd400640 1 --2- 172.21.15.40:0/2789598546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de37944a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3bfe800640 1 --2- 172.21.15.40:0/2789598546 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3788940 0x559de378d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3bfde00640 1 -- 172.21.15.40:0/2789598546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 msgr2=0x559de37944a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3bfde00640 1 --2- 172.21.15.40:0/2789598546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de37944a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3bfde00640 1 -- 172.21.15.40:0/2789598546 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3788940 msgr2=0x559de378d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3bfde00640 1 --2- 172.21.15.40:0/2789598546 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3788940 0x559de378d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3bfde00640 1 -- 172.21.15.40:0/2789598546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559de34c5620 con 0x559de3782de0 2024-04-27T00:12:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.483+0000 7f3bfe800640 1 --2- 172.21.15.40:0/2789598546 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3788940 0x559de378d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T00:12:09.485 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.484+0000 7f3bfde00640 1 --2- 172.21.15.40:0/2789598546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 0x559de378cf60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x559de3784240 tx=0x7f3bf0022040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc87ef1cceaa6815 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:09.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.484+0000 7f3bea600640 1 -- 172.21.15.40:0/2789598546 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf000fe50 con 0x559de3782de0 2024-04-27T00:12:09.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.484+0000 7f3bea600640 1 -- 172.21.15.40:0/2789598546 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bf0030070 con 0x559de3782de0 2024-04-27T00:12:09.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.484+0000 7f3bea600640 1 -- 172.21.15.40:0/2789598546 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf00116a0 con 0x559de3782de0 2024-04-27T00:12:09.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.485+0000 7f3c0786e740 1 -- 172.21.15.40:0/2789598546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 msgr2=0x559de378cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.485+0000 7f3c0786e740 1 --2- 172.21.15.40:0/2789598546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 0x559de378cf60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x559de3784240 tx=0x7f3bf0022040 comp rx=0 tx=0).stop 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.485+0000 7f3c0786e740 1 -- 172.21.15.40:0/2789598546 shutdown_connections 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.485+0000 7f3c0786e740 1 --2- 172.21.15.40:0/2789598546 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de3788940 0x559de378d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.485+0000 7f3c0786e740 1 --2- 172.21.15.40:0/2789598546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de37837d0 0x559de37944a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.485+0000 7f3c0786e740 1 --2- 172.21.15.40:0/2789598546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3782de0 0x559de378cf60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.485+0000 7f3c0786e740 1 -- 172.21.15.40:0/2789598546 >> 172.21.15.40:0/2789598546 conn(0x559de377e450 msgr2=0x559de3784950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.485+0000 7f3c0786e740 1 -- 172.21.15.40:0/2789598546 shutdown_connections 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.485+0000 7f3c0786e740 1 -- 172.21.15.40:0/2789598546 wait complete. 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.485+0000 7f3c0786e740 1 -- start start 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3c0786e740 1 Processor -- start 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3c0786e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de3782de0 0x559de3794520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3c0786e740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de37837d0 0x559de3791030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3c0786e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 0x559de3791640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.489 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3c0786e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3c001afe30 con 0x559de3788940 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3c0786e740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f3c001b12a0 con 0x559de37837d0 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3c0786e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c001b0500 con 0x559de3782de0 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3c0786e740 10 client.0 _ll_register_callbacks cb 0x7f3bec0d4590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3bfe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 0x559de3791640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3bfd400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de37837d0 0x559de3791030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3bfe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 0x559de3791640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42522/0 (socket says 172.21.15.40:42522) 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3bfe800640 1 -- 172.21.15.40:0/3985491204 learned_addr learned my addr 172.21.15.40:0/3985491204 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.486+0000 7f3bfde00640 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de3782de0 0x559de3794520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.487+0000 7f3bfd400640 1 -- 172.21.15.40:0/3985491204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de3782de0 msgr2=0x559de3794520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.487+0000 7f3bfd400640 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de3782de0 0x559de3794520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.487+0000 7f3bfd400640 1 -- 172.21.15.40:0/3985491204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 msgr2=0x559de3791640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.487+0000 7f3bfd400640 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 0x559de3791640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.487+0000 7f3bfd400640 1 -- 172.21.15.40:0/3985491204 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559de36a1cb0 con 0x559de37837d0 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.487+0000 7f3bfe800640 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 0x559de3791640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.487+0000 7f3bfd400640 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de37837d0 0x559de3791030 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0ce1c0 tx=0x7f3bec0bd140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bb5616e334bcea5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.487+0000 7f3bdf400640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.487+0000 7f3bdf400640 1 -- 172.21.15.40:0/3985491204 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec09c2f0 con 0x559de37837d0 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.487+0000 7f3bdf400640 1 -- 172.21.15.40:0/3985491204 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f3bec0ce410 con 0x559de37837d0 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.487+0000 7f3c0786e740 1 -- 172.21.15.40:0/3985491204 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bf0030070 con 0x559de37837d0 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.488+0000 7f3bdf400640 1 -- 172.21.15.40:0/3985491204 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3bec09d090 con 0x559de37837d0 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.488+0000 7f3c0786e740 10 client.26207 fetch_fsmap learned FSMap version 24 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.488+0000 7f3c0786e740 1 -- 172.21.15.40:0/3985491204 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559de37fdd00 con 0x559de37837d0 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.488+0000 7f3bdf400640 1 -- 172.21.15.40:0/3985491204 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bf0030070 con 0x559de37837d0 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.488+0000 7f3bdf400640 1 -- 172.21.15.40:0/3985491204 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3bec0d9070 con 0x559de37837d0 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.488+0000 7f3c0786e740 10 client.26207 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:12:09.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.488+0000 7f3c0786e740 10 client.26207 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:12:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.488+0000 7f3c0786e740 20 client.26207 populate_metadata read hostname 'smithi040' 2024-04-27T00:12:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.488+0000 7f3c0786e740 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559de37fe660 0x559de3800a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.488+0000 7f3c0786e740 4 client.26207 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:12:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.488+0000 7f3c0786e740 1 -- 172.21.15.40:0/3985491204 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559de36eee00 con 0x559de37fe660 2024-04-27T00:12:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.488+0000 7f3bfde00640 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559de37fe660 0x559de3800a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.489+0000 7f3bfde00640 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559de37fe660 0x559de3800a80 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceeb5228bb13ddf1 server_cookie=75954e5d6444407 in_seq=0 out_seq=0 2024-04-27T00:12:09.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.489+0000 7f3bdf400640 10 client.26207 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.505+0000 7f3bdf400640 1 -- 172.21.15.40:0/3985491204 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559de36eee00 con 0x559de37fe660 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.505+0000 7f3bdf400640 10 client.26207 handle_command_reply: tid=0 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.505+0000 7f3c0786e740 1 client.26207 shutdown 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.506+0000 7f3c0786e740 1 -- 172.21.15.40:0/3985491204 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de37837d0 msgr2=0x559de3791030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.506+0000 7f3c0786e740 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de37837d0 0x559de3791030 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0ce1c0 tx=0x7f3bec0bd140 comp rx=0 tx=0).stop 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.506+0000 7f3c0786e740 1 -- 172.21.15.40:0/3985491204 shutdown_connections 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.506+0000 7f3c0786e740 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x559de37fe660 0x559de3800a80 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.506+0000 7f3c0786e740 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559de3788940 0x559de3791640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.506+0000 7f3c0786e740 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x559de37837d0 0x559de3791030 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.506+0000 7f3c0786e740 1 --2- 172.21.15.40:0/3985491204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559de3782de0 0x559de3794520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.506+0000 7f3bdf400640 0 client.26207 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.506+0000 7f3c0786e740 1 -- 172.21.15.40:0/3985491204 >> 172.21.15.40:0/3985491204 conn(0x559de377e450 msgr2=0x559de3784950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:09.507 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.506+0000 7f3c0786e740 1 -- 172.21.15.40:0/3985491204 shutdown_connections 2024-04-27T00:12:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.506+0000 7f3c0786e740 1 -- 172.21.15.40:0/3985491204 wait complete. 2024-04-27T00:12:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.506+0000 7f3c0786e740 20 client.26207 trim_cache size 0 max 16384 2024-04-27T00:12:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:12:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:12:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:12:09.508 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:12:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.507+0000 7f3bff200640 1 -- 172.21.15.40:0/833196082 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3be40a3c30 msgr2=0x7f3be40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.507+0000 7f3bff200640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3be40a3c30 0x7f3be40a60f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x559de3783bd0 tx=0x7f3bf0005f90 comp rx=0 tx=0).stop 2024-04-27T00:12:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.507+0000 7f3bff200640 1 -- 172.21.15.40:0/833196082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c00109fd0 msgr2=0x7f3c0019f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.507+0000 7f3bff200640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c00109fd0 0x7f3c0019f4a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00e9f0 tx=0x7f3bec00eeb0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.507+0000 7f3bff200640 1 -- 172.21.15.40:0/833196082 shutdown_connections 2024-04-27T00:12:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.507+0000 7f3bff200640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f3c0010a990 0x7f3c001a6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.507+0000 7f3bff200640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f3be40a3c30 0x7f3be40a60f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.507+0000 7f3bff200640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c00109fd0 0x7f3c0019f4a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.507+0000 7f3bff200640 1 --2- 172.21.15.40:0/833196082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c00108de0 0x7f3c0019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:09.509 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.507+0000 7f3bff200640 1 -- 172.21.15.40:0/833196082 >> 172.21.15.40:0/833196082 conn(0x7f3c00077dc0 msgr2=0x7f3c0010d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:09.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.507+0000 7f3bff200640 1 -- 172.21.15.40:0/833196082 shutdown_connections 2024-04-27T00:12:09.510 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:09.507+0000 7f3bff200640 1 -- 172.21.15.40:0/833196082 wait complete. 2024-04-27T00:12:10.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:10 smithi040 ceph-mon[28095]: pgmap v726: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:11.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:10 smithi080 ceph-mon[32362]: pgmap v726: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:11.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:10 smithi103 ceph-mon[32952]: pgmap v726: 129 pgs: 129 active+clean; 3.1 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:12.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:12 smithi040 ceph-mon[28095]: pgmap v727: 129 pgs: 129 active+clean; 3.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-27T00:12:13.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:12 smithi080 ceph-mon[32362]: pgmap v727: 129 pgs: 129 active+clean; 3.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-27T00:12:13.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:12 smithi103 ceph-mon[32952]: pgmap v727: 129 pgs: 129 active+clean; 3.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-27T00:12:14.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:14 smithi040 ceph-mon[28095]: pgmap v728: 129 pgs: 129 active+clean; 3.2 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-27T00:12:15.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:14 smithi080 ceph-mon[32362]: pgmap v728: 129 pgs: 129 active+clean; 3.2 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-27T00:12:15.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:14 smithi103 ceph-mon[32952]: pgmap v728: 129 pgs: 129 active+clean; 3.2 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-27T00:12:16.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:16 smithi040 ceph-mon[28095]: pgmap v729: 129 pgs: 129 active+clean; 3.2 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:17.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:16 smithi080 ceph-mon[32362]: pgmap v729: 129 pgs: 129 active+clean; 3.2 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:17.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:16 smithi103 ceph-mon[32952]: pgmap v729: 129 pgs: 129 active+clean; 3.2 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:18.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:18 smithi040 ceph-mon[28095]: pgmap v730: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:12:19.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:18 smithi080 ceph-mon[32362]: pgmap v730: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:12:19.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:18 smithi103 ceph-mon[32952]: pgmap v730: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:12:20.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:20 smithi040 ceph-mon[28095]: pgmap v731: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:21.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:20 smithi080 ceph-mon[32362]: pgmap v731: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:21.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:20 smithi103 ceph-mon[32952]: pgmap v731: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:22.658 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.657+0000 7f746ce00640 1 -- 172.21.15.40:0/1251321552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7468105d40 msgr2=0x7f7468108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.657+0000 7f746ce00640 1 --2- 172.21.15.40:0/1251321552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7468105d40 0x7f7468108130 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7450009900 tx=0x7f745002a410 comp rx=0 tx=0).stop 2024-04-27T00:12:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.658+0000 7f746ce00640 1 -- 172.21.15.40:0/1251321552 shutdown_connections 2024-04-27T00:12:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.658+0000 7f746ce00640 1 --2- 172.21.15.40:0/1251321552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7468108670 0x7f746810aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.658+0000 7f746ce00640 1 --2- 172.21.15.40:0/1251321552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7468105d40 0x7f7468108130 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.658+0000 7f746ce00640 1 --2- 172.21.15.40:0/1251321552 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7468103410 0x7f7468105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.658+0000 7f746ce00640 1 -- 172.21.15.40:0/1251321552 >> 172.21.15.40:0/1251321552 conn(0x7f74680fce00 msgr2=0x7f74680ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.658+0000 7f746ce00640 1 -- 172.21.15.40:0/1251321552 shutdown_connections 2024-04-27T00:12:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.658+0000 7f746ce00640 1 -- 172.21.15.40:0/1251321552 wait complete. 2024-04-27T00:12:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.659+0000 7f746ce00640 1 Processor -- start 2024-04-27T00:12:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.659+0000 7f746ce00640 1 -- start start 2024-04-27T00:12:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.659+0000 7f746ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7468103410 0x7f746819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.659+0000 7f746ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7468105d40 0x7f746819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.659+0000 7f746ce00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7468108670 0x7f74681a2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.659+0000 7f746ce00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f746810e500 con 0x7f7468105d40 2024-04-27T00:12:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.659+0000 7f746ce00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f746810e380 con 0x7f7468108670 2024-04-27T00:12:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.659+0000 7f746ce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f746810e680 con 0x7f7468103410 2024-04-27T00:12:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.660+0000 7f745fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7468105d40 0x7f746819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.660+0000 7f7465e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7468103410 0x7f746819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.660+0000 7f745fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7468105d40 0x7f746819b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40044/0 (socket says 172.21.15.40:40044) 2024-04-27T00:12:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.660+0000 7f745fe00640 1 -- 172.21.15.40:0/3678518604 learned_addr learned my addr 172.21.15.40:0/3678518604 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.660+0000 7f7465e00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7468103410 0x7f746819ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48676/0 (socket says 172.21.15.40:48676) 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.660+0000 7f7466800640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7468108670 0x7f74681a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.660+0000 7f7465e00640 1 -- 172.21.15.40:0/3678518604 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7468108670 msgr2=0x7f74681a2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.660+0000 7f7465e00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7468108670 0x7f74681a2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.660+0000 7f7465e00640 1 -- 172.21.15.40:0/3678518604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7468105d40 msgr2=0x7f746819b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.660+0000 7f7465e00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7468105d40 0x7f746819b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.660+0000 7f7465e00640 1 -- 172.21.15.40:0/3678518604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74681a2970 con 0x7f7468103410 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.660+0000 7f745fe00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7468105d40 0x7f746819b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.661+0000 7f7465e00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7468103410 0x7f746819ad50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f745800e9f0 tx=0x7f745800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.661+0000 7f745ea00640 1 -- 172.21.15.40:0/3678518604 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745800ccc0 con 0x7f7468103410 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.661+0000 7f745ea00640 1 -- 172.21.15.40:0/3678518604 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74580108e0 con 0x7f7468103410 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.661+0000 7f746ce00640 1 -- 172.21.15.40:0/3678518604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74681a2c60 con 0x7f7468103410 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.661+0000 7f745ea00640 1 -- 172.21.15.40:0/3678518604 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458018880 con 0x7f7468103410 2024-04-27T00:12:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.661+0000 7f746ce00640 1 -- 172.21.15.40:0/3678518604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74681a31d0 con 0x7f7468103410 2024-04-27T00:12:22.664 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.664+0000 7f746fc4d740 1 Processor -- start 2024-04-27T00:12:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.664+0000 7f746fc4d740 1 -- start start 2024-04-27T00:12:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.665+0000 7f745ea00640 1 -- 172.21.15.40:0/3678518604 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7458010430 con 0x7f7468103410 2024-04-27T00:12:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.665+0000 7f746fc4d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 0x55f22d8c0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.665+0000 7f746fc4d740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c0dd0 0x55f22d8c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.665+0000 7f746fc4d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c5f40 0x55f22d8c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.665+0000 7f746fc4d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f22d81c380 con 0x55f22d8c03e0 2024-04-27T00:12:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.665+0000 7f746fc4d740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55f22d7b8c30 con 0x55f22d8c0dd0 2024-04-27T00:12:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.665+0000 7f746fc4d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f22d818340 con 0x55f22d8c5f40 2024-04-27T00:12:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.665+0000 7f7466800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 0x55f22d8c0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.665+0000 7f7466800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 0x55f22d8c0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40046/0 (socket says 172.21.15.40:40046) 2024-04-27T00:12:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.665+0000 7f7466800640 1 -- 172.21.15.40:0/3049070410 learned_addr learned my addr 172.21.15.40:0/3049070410 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.665+0000 7f745fe00640 1 --2- 172.21.15.40:0/3049070410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c5f40 0x55f22d8c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.665+0000 7f7465e00640 1 --2- 172.21.15.40:0/3049070410 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c0dd0 0x55f22d8c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.666+0000 7f7466800640 1 -- 172.21.15.40:0/3049070410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c5f40 msgr2=0x55f22d8c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.666+0000 7f7466800640 1 --2- 172.21.15.40:0/3049070410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c5f40 0x55f22d8c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.666+0000 7f7466800640 1 -- 172.21.15.40:0/3049070410 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c0dd0 msgr2=0x55f22d8c58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.666+0000 7f7466800640 1 --2- 172.21.15.40:0/3049070410 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c0dd0 0x55f22d8c58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.666+0000 7f7466800640 1 -- 172.21.15.40:0/3049070410 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f22d7da940 con 0x55f22d8c03e0 2024-04-27T00:12:22.667 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.666+0000 7f7465e00640 1 --2- 172.21.15.40:0/3049070410 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c0dd0 0x55f22d8c58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:12:22.670 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.666+0000 7f7466800640 1 --2- 172.21.15.40:0/3049070410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 0x55f22d8c0800 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7460009fd0 tx=0x7f74600118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a77e3378024108 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:22.670 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.666+0000 7f7457400640 1 -- 172.21.15.40:0/3049070410 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460004d20 con 0x55f22d8c03e0 2024-04-27T00:12:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.666+0000 7f7457400640 1 -- 172.21.15.40:0/3049070410 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7460004050 con 0x55f22d8c03e0 2024-04-27T00:12:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.666+0000 7f7457400640 1 -- 172.21.15.40:0/3049070410 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460005280 con 0x55f22d8c03e0 2024-04-27T00:12:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.666+0000 7f745ea00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f744c0a3d70 0x7f744c0a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.667+0000 7f745ea00640 1 -- 172.21.15.40:0/3678518604 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f74580d0c10 con 0x7f7468103410 2024-04-27T00:12:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.667+0000 7f7465e00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f744c0a3d70 0x7f744c0a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.667+0000 7f746fc4d740 1 -- 172.21.15.40:0/3049070410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 msgr2=0x55f22d8c0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.667+0000 7f746fc4d740 1 --2- 172.21.15.40:0/3049070410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 0x55f22d8c0800 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7460009fd0 tx=0x7f74600118d0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.667+0000 7f746fc4d740 1 -- 172.21.15.40:0/3049070410 shutdown_connections 2024-04-27T00:12:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.667+0000 7f746fc4d740 1 --2- 172.21.15.40:0/3049070410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c5f40 0x55f22d8c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.667+0000 7f746fc4d740 1 --2- 172.21.15.40:0/3049070410 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c0dd0 0x55f22d8c58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.667+0000 7f746fc4d740 1 --2- 172.21.15.40:0/3049070410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 0x55f22d8c0800 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.667+0000 7f746fc4d740 1 -- 172.21.15.40:0/3049070410 >> 172.21.15.40:0/3049070410 conn(0x55f22d8b3240 msgr2=0x55f22d8b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.667+0000 7f746fc4d740 1 -- 172.21.15.40:0/3049070410 shutdown_connections 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.667+0000 7f746fc4d740 1 -- 172.21.15.40:0/3049070410 wait complete. 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.668+0000 7f7465e00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f744c0a3d70 0x7f744c0a6230 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55f22d8c11d0 tx=0x7f7458009700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.668+0000 7f746fc4d740 1 -- start start 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.668+0000 7f746fc4d740 1 Processor -- start 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.669+0000 7f746fc4d740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c03e0 0x55f22d8ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.669+0000 7f746fc4d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c0dd0 0x55f22d8cab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.669+0000 7f746fc4d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c5f40 0x55f22d8d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.669+0000 7f746fc4d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f22d8c8ef0 con 0x55f22d8c0dd0 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.669+0000 7f746fc4d740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55f22d7a3c60 con 0x55f22d8c03e0 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.669+0000 7f746fc4d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f22d8c9070 con 0x55f22d8c5f40 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.669+0000 7f746fc4d740 10 client.0 _ll_register_callbacks cb 0x55f22d7cfe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.669+0000 7f745fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c0dd0 0x55f22d8cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.669+0000 7f7466800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c03e0 0x55f22d8ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.669+0000 7f745fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c0dd0 0x55f22d8cab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40048/0 (socket says 172.21.15.40:40048) 2024-04-27T00:12:22.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.669+0000 7f745fe00640 1 -- 172.21.15.40:0/497643954 learned_addr learned my addr 172.21.15.40:0/497643954 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:22.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.669+0000 7f7466800640 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c5f40 0x55f22d8d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.670+0000 7f745fe00640 1 -- 172.21.15.40:0/497643954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c5f40 msgr2=0x55f22d8d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:22.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.670+0000 7f745fe00640 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c5f40 0x55f22d8d1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.670+0000 7f745fe00640 1 -- 172.21.15.40:0/497643954 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c03e0 msgr2=0x55f22d8ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:22.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.670+0000 7f745fe00640 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c03e0 0x55f22d8ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.670+0000 7f745fe00640 1 -- 172.21.15.40:0/497643954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f22d7b2510 con 0x55f22d8c0dd0 2024-04-27T00:12:22.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.670+0000 7f7466800640 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c5f40 0x55f22d8d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:12:22.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.670+0000 7f745fe00640 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c0dd0 0x55f22d8cab10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f745002aa00 tx=0x7f7450003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e848488ca2089e83 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:22.673 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.670+0000 7f744be00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:12:22.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.673+0000 7f744be00640 1 -- 172.21.15.40:0/497643954 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450035070 con 0x55f22d8c0dd0 2024-04-27T00:12:22.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.674+0000 7f744be00640 1 -- 172.21.15.40:0/497643954 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f7450003ba0 con 0x55f22d8c0dd0 2024-04-27T00:12:22.674 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.674+0000 7f746fc4d740 1 -- 172.21.15.40:0/497643954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7460004050 con 0x55f22d8c0dd0 2024-04-27T00:12:22.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.674+0000 7f744be00640 1 -- 172.21.15.40:0/497643954 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f745004f050 con 0x55f22d8c0dd0 2024-04-27T00:12:22.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.674+0000 7f746fc4d740 10 client.16107 fetch_fsmap learned FSMap version 24 2024-04-27T00:12:22.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.674+0000 7f746fc4d740 1 -- 172.21.15.40:0/497643954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f22d52e210 con 0x55f22d8c0dd0 2024-04-27T00:12:22.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.674+0000 7f744be00640 1 -- 172.21.15.40:0/497643954 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7460004050 con 0x55f22d8c0dd0 2024-04-27T00:12:22.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.674+0000 7f744be00640 1 -- 172.21.15.40:0/497643954 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7450046460 con 0x55f22d8c0dd0 2024-04-27T00:12:22.675 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.675+0000 7f746fc4d740 10 client.16107 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:12:22.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.675+0000 7f746fc4d740 10 client.16107 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:12:22.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.675+0000 7f746fc4d740 20 client.16107 populate_metadata read hostname 'smithi040' 2024-04-27T00:12:22.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.675+0000 7f746fc4d740 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55f22d8d60a0 0x55f22d8d84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.675+0000 7f746fc4d740 4 client.16107 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:12:22.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.675+0000 7f746fc4d740 1 -- 172.21.15.40:0/497643954 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f22d829f90 con 0x55f22d8d60a0 2024-04-27T00:12:22.676 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.675+0000 7f7466800640 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55f22d8d60a0 0x55f22d8d84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.675+0000 7f7466800640 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55f22d8d60a0 0x55f22d8d84f0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14348fb2bcba791f server_cookie=35bb0a44be53015c in_seq=0 out_seq=0 2024-04-27T00:12:22.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.675+0000 7f744be00640 10 client.16107 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:22.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.676+0000 7f744be00640 1 -- 172.21.15.40:0/497643954 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f22d829f90 con 0x55f22d8d60a0 2024-04-27T00:12:22.677 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.676+0000 7f744be00640 10 client.16107 handle_command_reply: tid=0 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.676+0000 7f746fc4d740 1 client.16107 shutdown 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.677+0000 7f746fc4d740 1 -- 172.21.15.40:0/497643954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c0dd0 msgr2=0x55f22d8cab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.677+0000 7f746fc4d740 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c0dd0 0x55f22d8cab10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f745002aa00 tx=0x7f7450003820 comp rx=0 tx=0).stop 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.677+0000 7f746fc4d740 1 -- 172.21.15.40:0/497643954 shutdown_connections 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.677+0000 7f746fc4d740 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55f22d8d60a0 0x55f22d8d84f0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.677+0000 7f746fc4d740 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c5f40 0x55f22d8d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.677+0000 7f746fc4d740 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c0dd0 0x55f22d8cab10 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.677+0000 7f746fc4d740 1 --2- 172.21.15.40:0/497643954 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c03e0 0x55f22d8ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.677+0000 7f744be00640 0 client.16107 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.677+0000 7f746fc4d740 1 -- 172.21.15.40:0/497643954 >> 172.21.15.40:0/497643954 conn(0x55f22d8b3240 msgr2=0x55f22d8b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.677+0000 7f746fc4d740 1 -- 172.21.15.40:0/497643954 shutdown_connections 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.677+0000 7f746fc4d740 1 -- 172.21.15.40:0/497643954 wait complete. 2024-04-27T00:12:22.678 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.677+0000 7f746fc4d740 20 client.16107 trim_cache size 0 max 16384 2024-04-27T00:12:22.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.693+0000 7f746fc4d740 1 Processor -- start 2024-04-27T00:12:22.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.693+0000 7f746fc4d740 1 -- start start 2024-04-27T00:12:22.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.694+0000 7f746fc4d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 0x55f22d8c1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.694+0000 7f746fc4d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 0x55f22d8c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.694+0000 7f746fc4d740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c5f40 0x55f22d8d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.694+0000 7f746fc4d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f74681abbf0 con 0x55f22d8c03e0 2024-04-27T00:12:22.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.694+0000 7f746fc4d740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f74681a94c0 con 0x55f22d8c5f40 2024-04-27T00:12:22.694 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.694+0000 7f746fc4d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f74681a7310 con 0x55f22d8c0dd0 2024-04-27T00:12:22.695 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.694+0000 7f7466800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 0x55f22d8c1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.694+0000 7f745fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 0x55f22d8c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.694+0000 7f745fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 0x55f22d8c24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48692/0 (socket says 172.21.15.40:48692) 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.694+0000 7f745fe00640 1 -- 172.21.15.40:0/3483445654 learned_addr learned my addr 172.21.15.40:0/3483445654 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.694+0000 7f7466800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 0x55f22d8c1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40052/0 (socket says 172.21.15.40:40052) 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.694+0000 7f7466800640 1 --2- 172.21.15.40:0/3483445654 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c5f40 0x55f22d8d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.695+0000 7f745fe00640 1 -- 172.21.15.40:0/3483445654 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c5f40 msgr2=0x55f22d8d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.695+0000 7f745fe00640 1 --2- 172.21.15.40:0/3483445654 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c5f40 0x55f22d8d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.695+0000 7f745fe00640 1 -- 172.21.15.40:0/3483445654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 msgr2=0x55f22d8c1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.695+0000 7f745fe00640 1 --2- 172.21.15.40:0/3483445654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 0x55f22d8c1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.695+0000 7f745fe00640 1 -- 172.21.15.40:0/3483445654 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f22d5fa620 con 0x55f22d8c0dd0 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.695+0000 7f745fe00640 1 --2- 172.21.15.40:0/3483445654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 0x55f22d8c24f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f745002a930 tx=0x7f745002a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b384f3c8ed31186f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.695+0000 7f7456a00640 1 -- 172.21.15.40:0/3483445654 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450000e60 con 0x55f22d8c0dd0 2024-04-27T00:12:22.696 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.695+0000 7f7456a00640 1 -- 172.21.15.40:0/3483445654 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74500054f0 con 0x55f22d8c0dd0 2024-04-27T00:12:22.697 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.695+0000 7f7456a00640 1 -- 172.21.15.40:0/3483445654 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450035040 con 0x55f22d8c0dd0 2024-04-27T00:12:22.698 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.696+0000 7f746fc4d740 1 -- 172.21.15.40:0/3483445654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 msgr2=0x55f22d8c24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.696+0000 7f746fc4d740 1 --2- 172.21.15.40:0/3483445654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 0x55f22d8c24f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f745002a930 tx=0x7f745002a410 comp rx=0 tx=0).stop 2024-04-27T00:12:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.696+0000 7f746fc4d740 1 -- 172.21.15.40:0/3483445654 shutdown_connections 2024-04-27T00:12:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.696+0000 7f746fc4d740 1 --2- 172.21.15.40:0/3483445654 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c5f40 0x55f22d8d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.696+0000 7f746fc4d740 1 --2- 172.21.15.40:0/3483445654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 0x55f22d8c24f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.696+0000 7f746fc4d740 1 --2- 172.21.15.40:0/3483445654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c03e0 0x55f22d8c1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.696+0000 7f746fc4d740 1 -- 172.21.15.40:0/3483445654 >> 172.21.15.40:0/3483445654 conn(0x55f22d8b3240 msgr2=0x55f22d8c5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.696+0000 7f746fc4d740 1 -- 172.21.15.40:0/3483445654 shutdown_connections 2024-04-27T00:12:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.696+0000 7f746fc4d740 1 -- 172.21.15.40:0/3483445654 wait complete. 2024-04-27T00:12:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.697+0000 7f746fc4d740 1 -- start start 2024-04-27T00:12:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.697+0000 7f746fc4d740 1 Processor -- start 2024-04-27T00:12:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.697+0000 7f746fc4d740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c03e0 0x55f22d8cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.699 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.697+0000 7f746fc4d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 0x55f22d8cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.697+0000 7f746fc4d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c5f40 0x55f22d8d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.697+0000 7f746fc4d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f74681a7980 con 0x55f22d8c5f40 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.697+0000 7f746fc4d740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f74681a8df0 con 0x55f22d8c03e0 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.697+0000 7f746fc4d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f74681a8050 con 0x55f22d8c0dd0 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.697+0000 7f746fc4d740 10 client.0 _ll_register_callbacks cb 0x7f7450009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.697+0000 7f7466800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c03e0 0x55f22d8cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.697+0000 7f745fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 0x55f22d8cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.698+0000 7f7466800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c5f40 0x55f22d8d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.698+0000 7f745fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 0x55f22d8cf4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48704/0 (socket says 172.21.15.40:48704) 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.698+0000 7f745fe00640 1 -- 172.21.15.40:0/2541557670 learned_addr learned my addr 172.21.15.40:0/2541557670 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.698+0000 7f745fe00640 1 -- 172.21.15.40:0/2541557670 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c03e0 msgr2=0x55f22d8cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.698+0000 7f745fe00640 1 --2- 172.21.15.40:0/2541557670 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c03e0 0x55f22d8cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.698+0000 7f745fe00640 1 -- 172.21.15.40:0/2541557670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c5f40 msgr2=0x55f22d8d04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:22.700 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.698+0000 7f745fe00640 1 --2- 172.21.15.40:0/2541557670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c5f40 0x55f22d8d04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.698+0000 7f745fe00640 1 -- 172.21.15.40:0/2541557670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f22d7d6cb0 con 0x55f22d8c0dd0 2024-04-27T00:12:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.698+0000 7f745fe00640 1 --2- 172.21.15.40:0/2541557670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 0x55f22d8cf4b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7450005ec0 tx=0x7f7450005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c197bc891b59f1ab server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:22.701 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.698+0000 7f744be00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:12:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.699+0000 7f744be00640 1 -- 172.21.15.40:0/2541557670 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450035040 con 0x55f22d8c0dd0 2024-04-27T00:12:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.699+0000 7f746fc4d740 1 -- 172.21.15.40:0/2541557670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74500054f0 con 0x55f22d8c0dd0 2024-04-27T00:12:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.699+0000 7f744be00640 1 -- 172.21.15.40:0/2541557670 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f745004c040 con 0x55f22d8c0dd0 2024-04-27T00:12:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.699+0000 7f744be00640 1 -- 172.21.15.40:0/2541557670 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7450065050 con 0x55f22d8c0dd0 2024-04-27T00:12:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.699+0000 7f746fc4d740 10 client.26260 fetch_fsmap learned FSMap version 24 2024-04-27T00:12:22.702 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.699+0000 7f746fc4d740 1 -- 172.21.15.40:0/2541557670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f22d8d1bf0 con 0x55f22d8c0dd0 2024-04-27T00:12:22.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.700+0000 7f744be00640 1 -- 172.21.15.40:0/2541557670 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74500054f0 con 0x55f22d8c0dd0 2024-04-27T00:12:22.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.700+0000 7f744be00640 1 -- 172.21.15.40:0/2541557670 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7450031ae0 con 0x55f22d8c0dd0 2024-04-27T00:12:22.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.700+0000 7f746fc4d740 10 client.26260 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:12:22.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.700+0000 7f746fc4d740 10 client.26260 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:12:22.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.700+0000 7f746fc4d740 20 client.26260 populate_metadata read hostname 'smithi040' 2024-04-27T00:12:22.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.700+0000 7f746fc4d740 1 --2- 172.21.15.40:0/2541557670 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55f22d93cad0 0x55f22d93eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:22.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.700+0000 7f746fc4d740 4 client.26260 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:12:22.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.700+0000 7f746fc4d740 1 -- 172.21.15.40:0/2541557670 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f22d823e00 con 0x55f22d93cad0 2024-04-27T00:12:22.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.700+0000 7f7466800640 1 --2- 172.21.15.40:0/2541557670 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55f22d93cad0 0x55f22d93eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:22.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.701+0000 7f7466800640 1 --2- 172.21.15.40:0/2541557670 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55f22d93cad0 0x55f22d93eef0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=963ccabf8954ff81 server_cookie=f7161efa3e42f37f in_seq=0 out_seq=0 2024-04-27T00:12:22.703 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.701+0000 7f744be00640 10 client.26260 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:22.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.716+0000 7f744be00640 1 -- 172.21.15.40:0/2541557670 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f22d823e00 con 0x55f22d93cad0 2024-04-27T00:12:22.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.716+0000 7f744be00640 10 client.26260 handle_command_reply: tid=0 2024-04-27T00:12:22.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.716+0000 7f746fc4d740 1 client.26260 shutdown 2024-04-27T00:12:22.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.716+0000 7f746fc4d740 1 -- 172.21.15.40:0/2541557670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 msgr2=0x55f22d8cf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:22.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.716+0000 7f746fc4d740 1 --2- 172.21.15.40:0/2541557670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 0x55f22d8cf4b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7450005ec0 tx=0x7f7450005c90 comp rx=0 tx=0).stop 2024-04-27T00:12:22.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.717+0000 7f746fc4d740 1 -- 172.21.15.40:0/2541557670 shutdown_connections 2024-04-27T00:12:22.717 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.717+0000 7f746fc4d740 1 --2- 172.21.15.40:0/2541557670 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55f22d93cad0 0x55f22d93eef0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.717+0000 7f746fc4d740 1 --2- 172.21.15.40:0/2541557670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f22d8c5f40 0x55f22d8d04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.717+0000 7f746fc4d740 1 --2- 172.21.15.40:0/2541557670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f22d8c0dd0 0x55f22d8cf4b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.717+0000 7f746fc4d740 1 --2- 172.21.15.40:0/2541557670 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55f22d8c03e0 0x55f22d8cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.717+0000 7f744be00640 0 client.26260 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:22.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.717+0000 7f746fc4d740 1 -- 172.21.15.40:0/2541557670 >> 172.21.15.40:0/2541557670 conn(0x55f22d8b3240 msgr2=0x55f22d8ca5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:22.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.717+0000 7f746fc4d740 1 -- 172.21.15.40:0/2541557670 shutdown_connections 2024-04-27T00:12:22.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.717+0000 7f746fc4d740 1 -- 172.21.15.40:0/2541557670 wait complete. 2024-04-27T00:12:22.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.717+0000 7f746fc4d740 20 client.26260 trim_cache size 0 max 16384 2024-04-27T00:12:22.718 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:12:22.719 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:12:22.719 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:12:22.719 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:12:22.719 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.718+0000 7f746ce00640 1 -- 172.21.15.40:0/3678518604 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f744c0a3d70 msgr2=0x7f744c0a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:22.719 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.718+0000 7f746ce00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f744c0a3d70 0x7f744c0a6230 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55f22d8c11d0 tx=0x7f7458009700 comp rx=0 tx=0).stop 2024-04-27T00:12:22.719 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.718+0000 7f746ce00640 1 -- 172.21.15.40:0/3678518604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7468103410 msgr2=0x7f746819ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:22.720 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.718+0000 7f746ce00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7468103410 0x7f746819ad50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f745800e9f0 tx=0x7f745800eeb0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.720 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.718+0000 7f746ce00640 1 -- 172.21.15.40:0/3678518604 shutdown_connections 2024-04-27T00:12:22.720 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.718+0000 7f746ce00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7468108670 0x7f74681a2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.720 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.718+0000 7f746ce00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7468105d40 0x7f746819b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.720 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.718+0000 7f746ce00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f744c0a3d70 0x7f744c0a6230 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.720 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.718+0000 7f746ce00640 1 --2- 172.21.15.40:0/3678518604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7468103410 0x7f746819ad50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:22.720 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.718+0000 7f746ce00640 1 -- 172.21.15.40:0/3678518604 >> 172.21.15.40:0/3678518604 conn(0x7f74680fce00 msgr2=0x7f74681029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:22.720 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.718+0000 7f746ce00640 1 -- 172.21.15.40:0/3678518604 shutdown_connections 2024-04-27T00:12:22.720 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:22.718+0000 7f746ce00640 1 -- 172.21.15.40:0/3678518604 wait complete. 2024-04-27T00:12:22.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:22 smithi040 ceph-mon[28095]: pgmap v732: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:12:23.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:22 smithi080 ceph-mon[32362]: pgmap v732: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:12:23.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:22 smithi103 ceph-mon[32952]: pgmap v732: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:12:24.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:24 smithi040 ceph-mon[28095]: pgmap v733: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:12:25.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:24 smithi080 ceph-mon[32362]: pgmap v733: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:12:25.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:24 smithi103 ceph-mon[32952]: pgmap v733: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:12:26.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:26 smithi040 ceph-mon[28095]: pgmap v734: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:12:27.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:26 smithi080 ceph-mon[32362]: pgmap v734: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:12:27.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:26 smithi103 ceph-mon[32952]: pgmap v734: 129 pgs: 129 active+clean; 3.3 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:12:28.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:28 smithi040 ceph-mon[28095]: pgmap v735: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:12:29.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:28 smithi080 ceph-mon[32362]: pgmap v735: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:12:29.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:28 smithi103 ceph-mon[32952]: pgmap v735: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 17 op/s 2024-04-27T00:12:30.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:30 smithi040 ceph-mon[28095]: pgmap v736: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:12:31.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:30 smithi080 ceph-mon[32362]: pgmap v736: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:12:31.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:30 smithi103 ceph-mon[32952]: pgmap v736: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 12 op/s 2024-04-27T00:12:32.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:32 smithi040 ceph-mon[28095]: pgmap v737: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:12:33.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:32 smithi080 ceph-mon[32362]: pgmap v737: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:12:33.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:32 smithi103 ceph-mon[32952]: pgmap v737: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:12:34.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:34 smithi040 ceph-mon[28095]: pgmap v738: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:12:35.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:34 smithi080 ceph-mon[32362]: pgmap v738: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:12:35.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:34 smithi103 ceph-mon[32952]: pgmap v738: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:12:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.866+0000 7fe211800640 1 -- 172.21.15.40:0/2609866243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe20c10a990 msgr2=0x7fe20c111220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.866+0000 7fe211800640 1 --2- 172.21.15.40:0/2609866243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe20c10a990 0x7fe20c111220 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe204009ca0 tx=0x7fe20402a8b0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.866+0000 7fe211800640 1 -- 172.21.15.40:0/2609866243 shutdown_connections 2024-04-27T00:12:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.866+0000 7fe211800640 1 --2- 172.21.15.40:0/2609866243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe20c10a990 0x7fe20c111220 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.866+0000 7fe211800640 1 --2- 172.21.15.40:0/2609866243 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe20c109fd0 0x7fe20c10a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.866+0000 7fe211800640 1 --2- 172.21.15.40:0/2609866243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe20c108de0 0x7fe20c109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.866+0000 7fe211800640 1 -- 172.21.15.40:0/2609866243 >> 172.21.15.40:0/2609866243 conn(0x7fe20c077dc0 msgr2=0x7fe20c07a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.866+0000 7fe211800640 1 -- 172.21.15.40:0/2609866243 shutdown_connections 2024-04-27T00:12:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.867+0000 7fe211800640 1 -- 172.21.15.40:0/2609866243 wait complete. 2024-04-27T00:12:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.867+0000 7fe211800640 1 Processor -- start 2024-04-27T00:12:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.868+0000 7fe211800640 1 -- start start 2024-04-27T00:12:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.868+0000 7fe211800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe20c108de0 0x7fe20c19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.868+0000 7fe211800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe20c109fd0 0x7fe20c19f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.868+0000 7fe211800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe20c10a990 0x7fe20c1a6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.868+0000 7fe211800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c114940 con 0x7fe20c108de0 2024-04-27T00:12:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.868+0000 7fe211800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c1147c0 con 0x7fe20c10a990 2024-04-27T00:12:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.868+0000 7fe211800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c114ac0 con 0x7fe20c109fd0 2024-04-27T00:12:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.868+0000 7fe20be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe20c108de0 0x7fe20c19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.868+0000 7fe20b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe20c109fd0 0x7fe20c19f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.868+0000 7fe20b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe20c109fd0 0x7fe20c19f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:42904/0 (socket says 172.21.15.40:42904) 2024-04-27T00:12:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.868+0000 7fe20b400640 1 -- 172.21.15.40:0/249746645 learned_addr learned my addr 172.21.15.40:0/249746645 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.868+0000 7fe210e00640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe20c10a990 0x7fe20c1a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.869+0000 7fe20b400640 1 -- 172.21.15.40:0/249746645 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe20c10a990 msgr2=0x7fe20c1a6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.869+0000 7fe20b400640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe20c10a990 0x7fe20c1a6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.869+0000 7fe20b400640 1 -- 172.21.15.40:0/249746645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe20c108de0 msgr2=0x7fe20c19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.869+0000 7fe20b400640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe20c108de0 0x7fe20c19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.869+0000 7fe20b400640 1 -- 172.21.15.40:0/249746645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20c1a6b80 con 0x7fe20c109fd0 2024-04-27T00:12:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.869+0000 7fe20b400640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe20c109fd0 0x7fe20c19f4a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800e9f0 tx=0x7fe1f800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.869+0000 7fe208c00640 1 -- 172.21.15.40:0/249746645 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f800cd80 con 0x7fe20c109fd0 2024-04-27T00:12:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.869+0000 7fe208c00640 1 -- 172.21.15.40:0/249746645 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1f80108e0 con 0x7fe20c109fd0 2024-04-27T00:12:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.870+0000 7fe211800640 1 -- 172.21.15.40:0/249746645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe20c1a6e70 con 0x7fe20c109fd0 2024-04-27T00:12:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.870+0000 7fe208c00640 1 -- 172.21.15.40:0/249746645 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8018980 con 0x7fe20c109fd0 2024-04-27T00:12:35.872 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.870+0000 7fe211800640 1 -- 172.21.15.40:0/249746645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe20c1a7330 con 0x7fe20c109fd0 2024-04-27T00:12:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.873+0000 7fe215c4e740 1 Processor -- start 2024-04-27T00:12:35.873 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.873+0000 7fe215c4e740 1 -- start start 2024-04-27T00:12:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.873+0000 7fe215c4e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0ca200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.873+0000 7fe215c4e740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.873+0000 7fe215c4e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 0x5608fe0d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.873+0000 7fe215c4e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5608fe02e380 con 0x5608fe0c9de0 2024-04-27T00:12:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.873+0000 7fe215c4e740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5608fdfcac30 con 0x5608fe0ca7d0 2024-04-27T00:12:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.873+0000 7fe215c4e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5608fe02a340 con 0x5608fe0cf940 2024-04-27T00:12:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.873+0000 7fe20be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.874+0000 7fe210e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.874+0000 7fe20be00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0cf2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:48340/0 (socket says 172.21.15.40:48340) 2024-04-27T00:12:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.874+0000 7fe20be00640 1 -- 172.21.15.40:0/2241393751 learned_addr learned my addr 172.21.15.40:0/2241393751 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.874+0000 7fe208c00640 1 -- 172.21.15.40:0/249746645 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe1f8010430 con 0x7fe20c109fd0 2024-04-27T00:12:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.874+0000 7fe20be00640 1 -- 172.21.15.40:0/2241393751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 msgr2=0x5608fe0d1dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-27T00:12:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.874+0000 7fe20be00640 1 --2- 172.21.15.40:0/2241393751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 0x5608fe0d1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.874+0000 7fe20be00640 1 -- 172.21.15.40:0/2241393751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 msgr2=0x5608fe0ca200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.874+0000 7fe20be00640 1 --2- 172.21.15.40:0/2241393751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0ca200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.874+0000 7fe20be00640 1 -- 172.21.15.40:0/2241393751 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608fdfec940 con 0x5608fe0ca7d0 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.874+0000 7fe20be00640 1 --2- 172.21.15.40:0/2241393751 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0cf2d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009ca0 tx=0x7fe1fc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a542b03ae577971d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.875+0000 7fe201600640 1 -- 172.21.15.40:0/2241393751 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc004220 con 0x5608fe0ca7d0 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.875+0000 7fe201600640 1 -- 172.21.15.40:0/2241393751 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1fc0043c0 con 0x5608fe0ca7d0 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.875+0000 7fe201600640 1 -- 172.21.15.40:0/2241393751 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc019810 con 0x5608fe0ca7d0 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.875+0000 7fe208c00640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe1f00a3c30 0x7fe1f00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.875+0000 7fe215c4e740 1 -- 172.21.15.40:0/2241393751 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 msgr2=0x5608fe0cf2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.875+0000 7fe215c4e740 1 --2- 172.21.15.40:0/2241393751 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0cf2d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009ca0 tx=0x7fe1fc011810 comp rx=0 tx=0).stop 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.875+0000 7fe208c00640 1 -- 172.21.15.40:0/249746645 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fe1f80d0c40 con 0x7fe20c109fd0 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.876+0000 7fe215c4e740 1 -- 172.21.15.40:0/2241393751 shutdown_connections 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.876+0000 7fe210e00640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe1f00a3c30 0x7fe1f00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.876+0000 7fe215c4e740 1 --2- 172.21.15.40:0/2241393751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 0x5608fe0d1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.876+0000 7fe215c4e740 1 --2- 172.21.15.40:0/2241393751 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0cf2d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.876+0000 7fe215c4e740 1 --2- 172.21.15.40:0/2241393751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.876+0000 7fe215c4e740 1 -- 172.21.15.40:0/2241393751 >> 172.21.15.40:0/2241393751 conn(0x5608fe0c5450 msgr2=0x5608fe0c5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.876+0000 7fe215c4e740 1 -- 172.21.15.40:0/2241393751 shutdown_connections 2024-04-27T00:12:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.876+0000 7fe215c4e740 1 -- 172.21.15.40:0/2241393751 wait complete. 2024-04-27T00:12:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.876+0000 7fe215c4e740 1 -- start start 2024-04-27T00:12:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.876+0000 7fe210e00640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe1f00a3c30 0x7fe1f00a60f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x5608fe0cb030 tx=0x7fe204030800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:35.878 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe215c4e740 1 Processor -- start 2024-04-27T00:12:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe215c4e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0d4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe215c4e740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0d4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe215c4e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 0x5608fe0db580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe215c4e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5608fe0d28f0 con 0x5608fe0c9de0 2024-04-27T00:12:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe215c4e740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5608fdfb5c60 con 0x5608fe0ca7d0 2024-04-27T00:12:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe215c4e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5608fe0d2a70 con 0x5608fe0cf940 2024-04-27T00:12:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe215c4e740 10 client.0 _ll_register_callbacks cb 0x5608fdfe1e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:12:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe20be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0d4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe20b400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0d4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe20b400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0d4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:48348/0 (socket says 172.21.15.40:48348) 2024-04-27T00:12:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe20be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0d4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37194/0 (socket says 172.21.15.40:37194) 2024-04-27T00:12:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe20b400640 1 -- 172.21.15.40:0/1887806573 learned_addr learned my addr 172.21.15.40:0/1887806573 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.877+0000 7fe210e00640 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 0x5608fe0db580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.878+0000 7fe20b400640 1 -- 172.21.15.40:0/1887806573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 msgr2=0x5608fe0db580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.878+0000 7fe20b400640 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 0x5608fe0db580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.878+0000 7fe20b400640 1 -- 172.21.15.40:0/1887806573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 msgr2=0x5608fe0d4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.878+0000 7fe20b400640 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0d4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.878+0000 7fe20b400640 1 -- 172.21.15.40:0/1887806573 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608fdfc4510 con 0x5608fe0ca7d0 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.878+0000 7fe210e00640 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 0x5608fe0db580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.878+0000 7fe20b400640 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0d4540 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe1f809c770 tx=0x7fe1f809af80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c12006d336e41ba server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.878+0000 7fe1f6000640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.878+0000 7fe1f6000640 1 -- 172.21.15.40:0/1887806573 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f80a0d70 con 0x5608fe0ca7d0 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.878+0000 7fe1f6000640 1 -- 172.21.15.40:0/1887806573 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fe1f80bbb80 con 0x5608fe0ca7d0 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.878+0000 7fe215c4e740 1 -- 172.21.15.40:0/1887806573 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1fc0043c0 con 0x5608fe0ca7d0 2024-04-27T00:12:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.879+0000 7fe1f6000640 1 -- 172.21.15.40:0/1887806573 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe1f80a5630 con 0x5608fe0ca7d0 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.879+0000 7fe215c4e740 10 client.26237 fetch_fsmap learned FSMap version 24 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.879+0000 7fe215c4e740 1 -- 172.21.15.40:0/1887806573 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5608fdd40210 con 0x5608fe0ca7d0 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.879+0000 7fe1f6000640 1 -- 172.21.15.40:0/1887806573 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1fc0043c0 con 0x5608fe0ca7d0 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.879+0000 7fe1f6000640 1 -- 172.21.15.40:0/1887806573 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe1f80987c0 con 0x5608fe0ca7d0 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.879+0000 7fe215c4e740 10 client.26237 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.879+0000 7fe215c4e740 10 client.26237 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.879+0000 7fe215c4e740 20 client.26237 populate_metadata read hostname 'smithi040' 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.879+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5608fe0dfac0 0x5608fe0e1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.879+0000 7fe215c4e740 4 client.26237 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.879+0000 7fe215c4e740 1 -- 172.21.15.40:0/1887806573 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608fe03bf90 con 0x5608fe0dfac0 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.880+0000 7fe20be00640 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5608fe0dfac0 0x5608fe0e1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.880+0000 7fe20be00640 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5608fe0dfac0 0x5608fe0e1eb0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d559f9b4b6f227 server_cookie=f2476d8315b37b85 in_seq=0 out_seq=0 2024-04-27T00:12:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.880+0000 7fe1f6000640 10 client.26237 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe1f6000640 1 -- 172.21.15.40:0/1887806573 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5608fe03bf90 con 0x5608fe0dfac0 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe1f6000640 10 client.26237 handle_command_reply: tid=0 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe215c4e740 1 client.26237 shutdown 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe215c4e740 1 -- 172.21.15.40:0/1887806573 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 msgr2=0x5608fe0d4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0d4540 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe1f809c770 tx=0x7fe1f809af80 comp rx=0 tx=0).stop 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe215c4e740 1 -- 172.21.15.40:0/1887806573 shutdown_connections 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5608fe0dfac0 0x5608fe0e1eb0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 0x5608fe0db580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0d4540 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe1f6000640 0 client.26237 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1887806573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0d4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe215c4e740 1 -- 172.21.15.40:0/1887806573 >> 172.21.15.40:0/1887806573 conn(0x5608fe0c5450 msgr2=0x5608fe0c4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe215c4e740 1 -- 172.21.15.40:0/1887806573 shutdown_connections 2024-04-27T00:12:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.881+0000 7fe215c4e740 1 -- 172.21.15.40:0/1887806573 wait complete. 2024-04-27T00:12:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.882+0000 7fe215c4e740 20 client.26237 trim_cache size 0 max 16384 2024-04-27T00:12:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.898+0000 7fe215c4e740 1 Processor -- start 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.898+0000 7fe215c4e740 1 -- start start 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.898+0000 7fe215c4e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.898+0000 7fe215c4e740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0db540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.898+0000 7fe215c4e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 0x5608fe0cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.898+0000 7fe215c4e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c1b4160 con 0x5608fe0c9de0 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.898+0000 7fe215c4e740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c1b1a30 con 0x5608fe0ca7d0 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.898+0000 7fe215c4e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c1af880 con 0x5608fe0cf940 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.899+0000 7fe20be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.899+0000 7fe20be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0cb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37206/0 (socket says 172.21.15.40:37206) 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.899+0000 7fe20be00640 1 -- 172.21.15.40:0/1120788841 learned_addr learned my addr 172.21.15.40:0/1120788841 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.899+0000 7fe210e00640 1 --2- 172.21.15.40:0/1120788841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 0x5608fe0cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.899+0000 7fe20b400640 1 --2- 172.21.15.40:0/1120788841 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0db540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.899+0000 7fe20be00640 1 -- 172.21.15.40:0/1120788841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 msgr2=0x5608fe0cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.899+0000 7fe20be00640 1 --2- 172.21.15.40:0/1120788841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 0x5608fe0cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.899+0000 7fe20be00640 1 -- 172.21.15.40:0/1120788841 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 msgr2=0x5608fe0db540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.899+0000 7fe20be00640 1 --2- 172.21.15.40:0/1120788841 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0db540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.899+0000 7fe20be00640 1 -- 172.21.15.40:0/1120788841 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608fde0c620 con 0x5608fe0c9de0 2024-04-27T00:12:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.899+0000 7fe20b400640 1 --2- 172.21.15.40:0/1120788841 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0db540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:12:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.900+0000 7fe20be00640 1 --2- 172.21.15.40:0/1120788841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0cb9b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc011d10 tx=0x7fe1fc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518ff41e3d898694 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.900+0000 7fe200c00640 1 -- 172.21.15.40:0/1120788841 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc004330 con 0x5608fe0c9de0 2024-04-27T00:12:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.900+0000 7fe200c00640 1 -- 172.21.15.40:0/1120788841 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1fc02c860 con 0x5608fe0c9de0 2024-04-27T00:12:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.900+0000 7fe200c00640 1 -- 172.21.15.40:0/1120788841 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc02ce60 con 0x5608fe0c9de0 2024-04-27T00:12:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.900+0000 7fe215c4e740 1 -- 172.21.15.40:0/1120788841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 msgr2=0x5608fe0cb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.900+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1120788841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0cb9b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc011d10 tx=0x7fe1fc005f90 comp rx=0 tx=0).stop 2024-04-27T00:12:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.901+0000 7fe215c4e740 1 -- 172.21.15.40:0/1120788841 shutdown_connections 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.901+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1120788841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0cf940 0x5608fe0cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.901+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1120788841 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0ca7d0 0x5608fe0db540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.901+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1120788841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0c9de0 0x5608fe0cb9b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.901+0000 7fe215c4e740 1 -- 172.21.15.40:0/1120788841 >> 172.21.15.40:0/1120788841 conn(0x5608fe0c5450 msgr2=0x5608fe0d4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.901+0000 7fe215c4e740 1 -- 172.21.15.40:0/1120788841 shutdown_connections 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.901+0000 7fe215c4e740 1 -- 172.21.15.40:0/1120788841 wait complete. 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.901+0000 7fe215c4e740 1 -- start start 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.901+0000 7fe215c4e740 1 Processor -- start 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe215c4e740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0c9de0 0x5608fe0d8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe215c4e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0ca7d0 0x5608fe0d8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe215c4e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0cf940 0x5608fe0d9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe215c4e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c1afef0 con 0x5608fe0cf940 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe215c4e740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c1b1360 con 0x5608fe0c9de0 2024-04-27T00:12:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe215c4e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c1b05c0 con 0x5608fe0ca7d0 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe215c4e740 10 client.0 _ll_register_callbacks cb 0x7fe1f8009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe210e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0cf940 0x5608fe0d9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe20b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0ca7d0 0x5608fe0d8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe210e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0cf940 0x5608fe0d9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37212/0 (socket says 172.21.15.40:37212) 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe210e00640 1 -- 172.21.15.40:0/1632838439 learned_addr learned my addr 172.21.15.40:0/1632838439 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe20be00640 1 --2- 172.21.15.40:0/1632838439 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0c9de0 0x5608fe0d8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe20b400640 1 -- 172.21.15.40:0/1632838439 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0c9de0 msgr2=0x5608fe0d8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe20b400640 1 --2- 172.21.15.40:0/1632838439 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0c9de0 0x5608fe0d8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe20b400640 1 -- 172.21.15.40:0/1632838439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0cf940 msgr2=0x5608fe0d9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe20b400640 1 --2- 172.21.15.40:0/1632838439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0cf940 0x5608fe0d9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.902+0000 7fe20b400640 1 -- 172.21.15.40:0/1632838439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608fdfe8cb0 con 0x5608fe0ca7d0 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.903+0000 7fe20b400640 1 --2- 172.21.15.40:0/1632838439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0ca7d0 0x5608fe0d8e50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8098e80 tx=0x7fe1f809bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbba7b1c50ea4b8e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.903+0000 7fe1f6000640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.903+0000 7fe1f6000640 1 -- 172.21.15.40:0/1632838439 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f80a0d70 con 0x5608fe0ca7d0 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.903+0000 7fe1f6000640 1 -- 172.21.15.40:0/1632838439 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fe1f8018b20 con 0x5608fe0ca7d0 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.903+0000 7fe215c4e740 1 -- 172.21.15.40:0/1632838439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1fc02c860 con 0x5608fe0ca7d0 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.903+0000 7fe1f6000640 1 -- 172.21.15.40:0/1632838439 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe1f8018d20 con 0x5608fe0ca7d0 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.903+0000 7fe215c4e740 10 client.26275 fetch_fsmap learned FSMap version 24 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.903+0000 7fe215c4e740 1 -- 172.21.15.40:0/1632838439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5608fe0db600 con 0x5608fe0ca7d0 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.903+0000 7fe1f6000640 1 -- 172.21.15.40:0/1632838439 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1fc02c860 con 0x5608fe0ca7d0 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.904+0000 7fe1f6000640 1 -- 172.21.15.40:0/1632838439 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe1f80a1f00 con 0x5608fe0ca7d0 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.904+0000 7fe215c4e740 10 client.26275 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.904+0000 7fe215c4e740 10 client.26275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.904+0000 7fe215c4e740 20 client.26275 populate_metadata read hostname 'smithi040' 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.904+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1632838439 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5608fe146400 0x5608fe148820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.904+0000 7fe215c4e740 4 client.26275 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:12:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.904+0000 7fe215c4e740 1 -- 172.21.15.40:0/1632838439 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5608fe035e00 con 0x5608fe146400 2024-04-27T00:12:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.904+0000 7fe20be00640 1 --2- 172.21.15.40:0/1632838439 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5608fe146400 0x5608fe148820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.904+0000 7fe20be00640 1 --2- 172.21.15.40:0/1632838439 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5608fe146400 0x5608fe148820 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6f941ee50cc9142 server_cookie=d1c6bfabd78bd363 in_seq=0 out_seq=0 2024-04-27T00:12:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.904+0000 7fe1f6000640 10 client.26275 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.920+0000 7fe1f6000640 1 -- 172.21.15.40:0/1632838439 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5608fe035e00 con 0x5608fe146400 2024-04-27T00:12:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.920+0000 7fe1f6000640 10 client.26275 handle_command_reply: tid=0 2024-04-27T00:12:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.920+0000 7fe215c4e740 1 client.26275 shutdown 2024-04-27T00:12:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.920+0000 7fe215c4e740 1 -- 172.21.15.40:0/1632838439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0ca7d0 msgr2=0x5608fe0d8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.920+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1632838439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0ca7d0 0x5608fe0d8e50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8098e80 tx=0x7fe1f809bda0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.921+0000 7fe215c4e740 1 -- 172.21.15.40:0/1632838439 shutdown_connections 2024-04-27T00:12:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.921+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1632838439 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5608fe146400 0x5608fe148820 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.921+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1632838439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5608fe0cf940 0x5608fe0d9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.921+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1632838439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5608fe0ca7d0 0x5608fe0d8e50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.921+0000 7fe215c4e740 1 --2- 172.21.15.40:0/1632838439 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5608fe0c9de0 0x5608fe0d8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.921+0000 7fe1f6000640 0 client.26275 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.921+0000 7fe215c4e740 1 -- 172.21.15.40:0/1632838439 >> 172.21.15.40:0/1632838439 conn(0x5608fe0c5450 msgr2=0x5608fe0d3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.921+0000 7fe215c4e740 1 -- 172.21.15.40:0/1632838439 shutdown_connections 2024-04-27T00:12:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.921+0000 7fe215c4e740 1 -- 172.21.15.40:0/1632838439 wait complete. 2024-04-27T00:12:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.921+0000 7fe215c4e740 20 client.26275 trim_cache size 0 max 16384 2024-04-27T00:12:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:12:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:12:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:12:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:12:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.922+0000 7fe211800640 1 -- 172.21.15.40:0/249746645 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe1f00a3c30 msgr2=0x7fe1f00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.922+0000 7fe211800640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe1f00a3c30 0x7fe1f00a60f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x5608fe0cb030 tx=0x7fe204030800 comp rx=0 tx=0).stop 2024-04-27T00:12:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.922+0000 7fe211800640 1 -- 172.21.15.40:0/249746645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe20c109fd0 msgr2=0x7fe20c19f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.922+0000 7fe211800640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe20c109fd0 0x7fe20c19f4a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800e9f0 tx=0x7fe1f800eeb0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.922+0000 7fe211800640 1 -- 172.21.15.40:0/249746645 shutdown_connections 2024-04-27T00:12:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.922+0000 7fe211800640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe20c10a990 0x7fe20c1a6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.922+0000 7fe211800640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe1f00a3c30 0x7fe1f00a60f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.922+0000 7fe211800640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe20c109fd0 0x7fe20c19f4a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.922+0000 7fe211800640 1 --2- 172.21.15.40:0/249746645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe20c108de0 0x7fe20c19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.922+0000 7fe211800640 1 -- 172.21.15.40:0/249746645 >> 172.21.15.40:0/249746645 conn(0x7fe20c077dc0 msgr2=0x7fe20c10d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.922+0000 7fe211800640 1 -- 172.21.15.40:0/249746645 shutdown_connections 2024-04-27T00:12:35.925 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:35.922+0000 7fe211800640 1 -- 172.21.15.40:0/249746645 wait complete. 2024-04-27T00:12:36.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:36 smithi040 ceph-mon[28095]: pgmap v739: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:12:37.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:36 smithi080 ceph-mon[32362]: pgmap v739: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:12:37.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:36 smithi103 ceph-mon[32952]: pgmap v739: 129 pgs: 129 active+clean; 3.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:12:38.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:38 smithi040 ceph-mon[28095]: pgmap v740: 129 pgs: 129 active+clean; 3.5 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:12:39.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:38 smithi080 ceph-mon[32362]: pgmap v740: 129 pgs: 129 active+clean; 3.5 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:12:39.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:38 smithi103 ceph-mon[32952]: pgmap v740: 129 pgs: 129 active+clean; 3.5 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-04-27T00:12:41.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:40 smithi080 ceph-mon[32362]: pgmap v741: 129 pgs: 129 active+clean; 3.5 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:12:41.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:40 smithi103 ceph-mon[32952]: pgmap v741: 129 pgs: 129 active+clean; 3.5 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:12:41.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:40 smithi040 ceph-mon[28095]: pgmap v741: 129 pgs: 129 active+clean; 3.5 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:12:43.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:42 smithi080 ceph-mon[32362]: pgmap v742: 129 pgs: 129 active+clean; 3.6 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:12:43.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:42 smithi103 ceph-mon[32952]: pgmap v742: 129 pgs: 129 active+clean; 3.6 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:12:43.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:42 smithi040 ceph-mon[28095]: pgmap v742: 129 pgs: 129 active+clean; 3.6 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:12:44.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:43 smithi080 ceph-mon[32362]: pgmap v743: 129 pgs: 129 active+clean; 3.6 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:12:44.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:43 smithi103 ceph-mon[32952]: pgmap v743: 129 pgs: 129 active+clean; 3.6 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:12:44.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:43 smithi040 ceph-mon[28095]: pgmap v743: 129 pgs: 129 active+clean; 3.6 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:12:45.246 INFO:tasks.workunit.client.0.smithi040.stdout:499/600: open fd = 3 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:500/600: open fd = 3 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:501/600: open fd = 3 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:502/600: open fd = 3 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.247 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:503/600: open fd = 3 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:504/600: open fd = 3 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:505/600: open fd = 3 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.248 INFO:tasks.workunit.client.0.smithi040.stdout:506/600: open fd = 3 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:507/600: open fd = 3 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:508/600: open fd = 3 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:509/600: open fd = 3 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.249 INFO:tasks.workunit.client.0.smithi040.stdout:510/600: open fd = 3 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:511/600: open fd = 3 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:512/600: open fd = 3 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:513/600: open fd = 3 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.250 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:514/600: open fd = 3 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:515/600: open fd = 3 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:516/600: open fd = 3 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.251 INFO:tasks.workunit.client.0.smithi040.stdout:517/600: open fd = 3 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:518/600: open fd = 3 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:519/600: open fd = 3 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:520/600: open fd = 3 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.252 INFO:tasks.workunit.client.0.smithi040.stdout:521/600: open fd = 3 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:522/600: open fd = 3 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:523/600: open fd = 3 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:524/600: open fd = 3 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.253 INFO:tasks.workunit.client.0.smithi040.stdout:525/600: open fd = 3 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:526/600: open fd = 3 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:527/600: open fd = 3 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:528/600: open fd = 3 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.254 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:529/600: open fd = 3 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:530/600: open fd = 3 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:531/600: open fd = 3 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:532/600: open fd = 3 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.255 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:533/600: open fd = 3 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:534/600: open fd = 3 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:535/600: open fd = 3 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.256 INFO:tasks.workunit.client.0.smithi040.stdout:536/600: open fd = 3 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:537/600: open fd = 3 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:538/600: open fd = 3 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:539/600: open fd = 3 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.257 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:540/600: open fd = 3 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:541/600: open fd = 3 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:542/600: open fd = 3 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:543/600: open fd = 3 2024-04-27T00:12:45.258 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:544/600: open fd = 3 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:545/600: open fd = 3 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:546/600: open fd = 3 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.259 INFO:tasks.workunit.client.0.smithi040.stdout:547/600: open fd = 3 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:548/600: open fd = 3 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:549/600: open fd = 3 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:550/600: open fd = 3 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.260 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:551/600: open fd = 3 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:552/600: open fd = 3 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:553/600: open fd = 3 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:554/600: open fd = 3 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.261 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:555/600: open fd = 3 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:556/600: open fd = 3 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:557/600: open fd = 3 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.262 INFO:tasks.workunit.client.0.smithi040.stdout:558/600: open fd = 3 2024-04-27T00:12:45.263 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.263 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.263 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.263 INFO:tasks.workunit.client.0.smithi040.stdout:559/600: open fd = 3 2024-04-27T00:12:45.263 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.263 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.263 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:45.263 INFO:tasks.workunit.client.0.smithi040.stdout:560/600: open fd = 3 2024-04-27T00:12:45.263 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.263 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:12:45.263 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:12:46.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:46 smithi040 ceph-mon[28095]: pgmap v744: 129 pgs: 129 active+clean; 3.6 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:46.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:46 smithi080 ceph-mon[32362]: pgmap v744: 129 pgs: 129 active+clean; 3.6 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:46.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:46 smithi103 ceph-mon[32952]: pgmap v744: 129 pgs: 129 active+clean; 3.6 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:48.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:48 smithi040 ceph-mon[28095]: pgmap v745: 129 pgs: 129 active+clean; 3.6 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:12:48.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:48 smithi080 ceph-mon[32362]: pgmap v745: 129 pgs: 129 active+clean; 3.6 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:12:48.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:48 smithi103 ceph-mon[32952]: pgmap v745: 129 pgs: 129 active+clean; 3.6 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:12:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.070+0000 7f0e77200640 1 -- 172.21.15.40:0/1840193648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e781019e0 msgr2=0x7f0e78101e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.070+0000 7f0e77200640 1 --2- 172.21.15.40:0/1840193648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e781019e0 0x7f0e78101e40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009900 tx=0x7f0e6402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.071+0000 7f0e77200640 1 -- 172.21.15.40:0/1840193648 shutdown_connections 2024-04-27T00:12:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.071+0000 7f0e77200640 1 --2- 172.21.15.40:0/1840193648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e78102380 0x7f0e78102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.071+0000 7f0e77200640 1 --2- 172.21.15.40:0/1840193648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e781019e0 0x7f0e78101e40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.071+0000 7f0e77200640 1 --2- 172.21.15.40:0/1840193648 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e78101080 0x7f0e781014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.071+0000 7f0e77200640 1 -- 172.21.15.40:0/1840193648 >> 172.21.15.40:0/1840193648 conn(0x7f0e780fcc20 msgr2=0x7f0e780ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.071+0000 7f0e77200640 1 -- 172.21.15.40:0/1840193648 shutdown_connections 2024-04-27T00:12:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.071+0000 7f0e77200640 1 -- 172.21.15.40:0/1840193648 wait complete. 2024-04-27T00:12:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.072+0000 7f0e77200640 1 Processor -- start 2024-04-27T00:12:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.072+0000 7f0e77200640 1 -- start start 2024-04-27T00:12:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.073+0000 7f0e77200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e78101080 0x7f0e7819ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.073+0000 7f0e77200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e781019e0 0x7f0e7819b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.073+0000 7f0e77200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e78102380 0x7f0e781a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.073+0000 7f0e77200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781104d0 con 0x7f0e78102380 2024-04-27T00:12:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.073+0000 7f0e77200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0e78110350 con 0x7f0e78101080 2024-04-27T00:12:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.073+0000 7f0e77200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0e78110650 con 0x7f0e781019e0 2024-04-27T00:12:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.073+0000 7f0e75e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e78101080 0x7f0e7819ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.073+0000 7f0e75400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e781019e0 0x7f0e7819b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.073+0000 7f0e75e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e78101080 0x7f0e7819ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:37102/0 (socket says 172.21.15.40:37102) 2024-04-27T00:12:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.073+0000 7f0e75e00640 1 -- 172.21.15.40:0/2215081279 learned_addr learned my addr 172.21.15.40:0/2215081279 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.073+0000 7f0e76800640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e78102380 0x7f0e781a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.074+0000 7f0e75e00640 1 -- 172.21.15.40:0/2215081279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e781019e0 msgr2=0x7f0e7819b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.074+0000 7f0e75e00640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e781019e0 0x7f0e7819b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.074+0000 7f0e75e00640 1 -- 172.21.15.40:0/2215081279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e78102380 msgr2=0x7f0e781a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.074+0000 7f0e75e00640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e78102380 0x7f0e781a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.074+0000 7f0e75e00640 1 -- 172.21.15.40:0/2215081279 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e781a2830 con 0x7f0e78101080 2024-04-27T00:12:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.074+0000 7f0e76800640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e78102380 0x7f0e781a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:12:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.074+0000 7f0e75e00640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e78101080 0x7f0e7819ac20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800e9f0 tx=0x7f0e6800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.074+0000 7f0e6ea00640 1 -- 172.21.15.40:0/2215081279 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6800cd80 con 0x7f0e78101080 2024-04-27T00:12:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.074+0000 7f0e6ea00640 1 -- 172.21.15.40:0/2215081279 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e680108e0 con 0x7f0e78101080 2024-04-27T00:12:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.074+0000 7f0e77200640 1 -- 172.21.15.40:0/2215081279 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e781a2b20 con 0x7f0e78101080 2024-04-27T00:12:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.075+0000 7f0e6ea00640 1 -- 172.21.15.40:0/2215081279 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e68018980 con 0x7f0e78101080 2024-04-27T00:12:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.075+0000 7f0e77200640 1 -- 172.21.15.40:0/2215081279 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e781a30c0 con 0x7f0e78101080 2024-04-27T00:12:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.078+0000 7f0e6ea00640 1 -- 172.21.15.40:0/2215081279 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0e68010430 con 0x7f0e78101080 2024-04-27T00:12:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.078+0000 7f0e7ffee740 1 Processor -- start 2024-04-27T00:12:49.079 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.078+0000 7f0e7ffee740 1 -- start start 2024-04-27T00:12:49.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e7ffee740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcede0 0x5570e4bcf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e7ffee740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcf7d0 0x5570e4bd42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e7ffee740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bd4940 0x5570e4bd6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e7ffee740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5570e4b33380 con 0x5570e4bd4940 2024-04-27T00:12:49.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e7ffee740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5570e4acfc30 con 0x5570e4bcede0 2024-04-27T00:12:49.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e7ffee740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5570e4b2f340 con 0x5570e4bcf7d0 2024-04-27T00:12:49.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e76800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcede0 0x5570e4bcf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e76800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcede0 0x5570e4bcf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:37106/0 (socket says 172.21.15.40:37106) 2024-04-27T00:12:49.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e76800640 1 -- 172.21.15.40:0/1790180218 learned_addr learned my addr 172.21.15.40:0/1790180218 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:49.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e75e00640 1 --2- 172.21.15.40:0/1790180218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcf7d0 0x5570e4bd42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.080 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e75400640 1 --2- 172.21.15.40:0/1790180218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bd4940 0x5570e4bd6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e76800640 1 -- 172.21.15.40:0/1790180218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcf7d0 msgr2=0x5570e4bd42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e76800640 1 --2- 172.21.15.40:0/1790180218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcf7d0 0x5570e4bd42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e76800640 1 -- 172.21.15.40:0/1790180218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bd4940 msgr2=0x5570e4bd6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:49.082 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e76800640 1 --2- 172.21.15.40:0/1790180218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bd4940 0x5570e4bd6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.079+0000 7f0e76800640 1 -- 172.21.15.40:0/1790180218 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570e4af1940 con 0x5570e4bcede0 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.080+0000 7f0e76800640 1 --2- 172.21.15.40:0/1790180218 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcede0 0x5570e4bcf200 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009fd0 tx=0x7f0e700108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8ca74f3846133a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.080+0000 7f0e63400640 1 -- 172.21.15.40:0/1790180218 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e70011760 con 0x5570e4bcede0 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.080+0000 7f0e63400640 1 -- 172.21.15.40:0/1790180218 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e70011db0 con 0x5570e4bcede0 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.080+0000 7f0e63400640 1 -- 172.21.15.40:0/1790180218 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7000e3c0 con 0x5570e4bcede0 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.080+0000 7f0e6ea00640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0e5c0a3cb0 0x7f0e5c0a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.081+0000 7f0e75e00640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0e5c0a3cb0 0x7f0e5c0a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.081+0000 7f0e6ea00640 1 -- 172.21.15.40:0/2215081279 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f0e680d0d30 con 0x7f0e78101080 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.081+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1790180218 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcede0 msgr2=0x5570e4bcf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.081+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1790180218 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcede0 0x5570e4bcf200 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009fd0 tx=0x7f0e700108e0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.081+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1790180218 shutdown_connections 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.081+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1790180218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bd4940 0x5570e4bd6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.081+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1790180218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcf7d0 0x5570e4bd42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.083 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.081+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1790180218 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcede0 0x5570e4bcf200 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.081+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1790180218 >> 172.21.15.40:0/1790180218 conn(0x5570e4bca450 msgr2=0x5570e4bca850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.081+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1790180218 shutdown_connections 2024-04-27T00:12:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.081+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1790180218 wait complete. 2024-04-27T00:12:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.081+0000 7f0e75e00640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0e5c0a3cb0 0x7f0e5c0a6170 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x5570e4bcfbd0 tx=0x7f0e680d47e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.082+0000 7f0e7ffee740 1 -- start start 2024-04-27T00:12:49.084 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.082+0000 7f0e7ffee740 1 Processor -- start 2024-04-27T00:12:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e7ffee740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcede0 0x5570e4bd9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e7ffee740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcf7d0 0x5570e4bd9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e7ffee740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bd4940 0x5570e4be0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e7ffee740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5570e4bd78f0 con 0x5570e4bcf7d0 2024-04-27T00:12:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e7ffee740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5570e4abac60 con 0x5570e4bd4940 2024-04-27T00:12:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e7ffee740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5570e4bd7a70 con 0x5570e4bcede0 2024-04-27T00:12:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e7ffee740 10 client.0 _ll_register_callbacks cb 0x5570e4ae6e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:12:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e75400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcf7d0 0x5570e4bd9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e75400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcf7d0 0x5570e4bd9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46530/0 (socket says 172.21.15.40:46530) 2024-04-27T00:12:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e75400640 1 -- 172.21.15.40:0/1928127171 learned_addr learned my addr 172.21.15.40:0/1928127171 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e76800640 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcede0 0x5570e4bd9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e76800640 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bd4940 0x5570e4be0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e75400640 1 -- 172.21.15.40:0/1928127171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcede0 msgr2=0x5570e4bd9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e75400640 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcede0 0x5570e4bd9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e75400640 1 -- 172.21.15.40:0/1928127171 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bd4940 msgr2=0x5570e4be0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e75400640 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bd4940 0x5570e4be0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e75400640 1 -- 172.21.15.40:0/1928127171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570e4ac9510 con 0x5570e4bcf7d0 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.083+0000 7f0e76800640 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcede0 0x5570e4bd9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.084+0000 7f0e75400640 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcf7d0 0x5570e4bd9540 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009900 tx=0x7f0e64031db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37de2bda56aa3944 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.084+0000 7f0e57e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.084+0000 7f0e57e00640 1 -- 172.21.15.40:0/1928127171 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64035040 con 0x5570e4bcf7d0 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.084+0000 7f0e57e00640 1 -- 172.21.15.40:0/1928127171 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f0e64004040 con 0x5570e4bcf7d0 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.084+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1928127171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e70011db0 con 0x5570e4bcf7d0 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.084+0000 7f0e57e00640 1 -- 172.21.15.40:0/1928127171 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0e6404e050 con 0x5570e4bcf7d0 2024-04-27T00:12:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.084+0000 7f0e7ffee740 10 client.16143 fetch_fsmap learned FSMap version 24 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.084+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1928127171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5570e4845210 con 0x5570e4bcf7d0 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.085+0000 7f0e57e00640 1 -- 172.21.15.40:0/1928127171 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e70011db0 con 0x5570e4bcf7d0 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.085+0000 7f0e57e00640 1 -- 172.21.15.40:0/1928127171 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f0e6403c440 con 0x5570e4bcf7d0 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.085+0000 7f0e7ffee740 10 client.16143 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.085+0000 7f0e7ffee740 10 client.16143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.085+0000 7f0e7ffee740 20 client.16143 populate_metadata read hostname 'smithi040' 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.085+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5570e4be4c00 0x5570e4be7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.085+0000 7f0e7ffee740 4 client.16143 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.085+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1928127171 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570e4b40f90 con 0x5570e4be4c00 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.085+0000 7f0e76800640 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5570e4be4c00 0x5570e4be7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.086+0000 7f0e76800640 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5570e4be4c00 0x5570e4be7010 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2c0f27bc7df04ed server_cookie=e04c7e34fab510e2 in_seq=0 out_seq=0 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.086+0000 7f0e57e00640 10 client.16143 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e57e00640 1 -- 172.21.15.40:0/1928127171 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5570e4b40f90 con 0x5570e4be4c00 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e57e00640 10 client.16143 handle_command_reply: tid=0 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e7ffee740 1 client.16143 shutdown 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1928127171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcf7d0 msgr2=0x5570e4bd9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcf7d0 0x5570e4bd9540 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009900 tx=0x7f0e64031db0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1928127171 shutdown_connections 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5570e4be4c00 0x5570e4be7010 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bd4940 0x5570e4be0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcf7d0 0x5570e4bd9540 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e57e00640 0 client.16143 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1928127171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcede0 0x5570e4bd9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1928127171 >> 172.21.15.40:0/1928127171 conn(0x5570e4bca450 msgr2=0x5570e4bc9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1928127171 shutdown_connections 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.087+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1928127171 wait complete. 2024-04-27T00:12:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.088+0000 7f0e7ffee740 20 client.16143 trim_cache size 0 max 16384 2024-04-27T00:12:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.104+0000 7f0e7ffee740 1 Processor -- start 2024-04-27T00:12:49.105 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.104+0000 7f0e7ffee740 1 -- start start 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.104+0000 7f0e7ffee740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcede0 0x5570e4bd4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.104+0000 7f0e7ffee740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 0x5570e4be0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.104+0000 7f0e7ffee740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bd4940 0x5570e4bcd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.104+0000 7f0e7ffee740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781afd30 con 0x5570e4bd4940 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.104+0000 7f0e7ffee740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781ad600 con 0x5570e4bcf7d0 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.104+0000 7f0e7ffee740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781ab450 con 0x5570e4bcede0 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.105+0000 7f0e75400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 0x5570e4be0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.105+0000 7f0e75400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 0x5570e4be0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:37112/0 (socket says 172.21.15.40:37112) 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.105+0000 7f0e75400640 1 -- 172.21.15.40:0/1022213013 learned_addr learned my addr 172.21.15.40:0/1022213013 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.105+0000 7f0e76800640 1 --2- 172.21.15.40:0/1022213013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcede0 0x5570e4bd4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.105+0000 7f0e76800640 1 --2- 172.21.15.40:0/1022213013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bd4940 0x5570e4bcd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.105+0000 7f0e75400640 1 -- 172.21.15.40:0/1022213013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcede0 msgr2=0x5570e4bd4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.105+0000 7f0e75400640 1 --2- 172.21.15.40:0/1022213013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcede0 0x5570e4bd4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.106 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.105+0000 7f0e75400640 1 -- 172.21.15.40:0/1022213013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bd4940 msgr2=0x5570e4bcd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.105+0000 7f0e75400640 1 --2- 172.21.15.40:0/1022213013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bd4940 0x5570e4bcd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.105+0000 7f0e75400640 1 -- 172.21.15.40:0/1022213013 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570e4911620 con 0x5570e4bcf7d0 2024-04-27T00:12:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.105+0000 7f0e76800640 1 --2- 172.21.15.40:0/1022213013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bd4940 0x5570e4bcd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:12:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.105+0000 7f0e75400640 1 --2- 172.21.15.40:0/1022213013 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 0x5570e4be0540 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0e6403c860 tx=0x7f0e64032fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c4bc5fa94371aa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.106+0000 7f0e62a00640 1 -- 172.21.15.40:0/1022213013 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64057070 con 0x5570e4bcf7d0 2024-04-27T00:12:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.106+0000 7f0e62a00640 1 -- 172.21.15.40:0/1022213013 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e64005850 con 0x5570e4bcf7d0 2024-04-27T00:12:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.106+0000 7f0e62a00640 1 -- 172.21.15.40:0/1022213013 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64035040 con 0x5570e4bcf7d0 2024-04-27T00:12:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.106+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1022213013 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 msgr2=0x5570e4be0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.106+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1022213013 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 0x5570e4be0540 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0e6403c860 tx=0x7f0e64032fc0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.106+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1022213013 shutdown_connections 2024-04-27T00:12:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.106+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1022213013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bd4940 0x5570e4bcd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.106+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1022213013 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 0x5570e4be0540 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.106+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/1022213013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bcede0 0x5570e4bd4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.106+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1022213013 >> 172.21.15.40:0/1022213013 conn(0x5570e4bca450 msgr2=0x5570e4bd09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.106+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1022213013 shutdown_connections 2024-04-27T00:12:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.107+0000 7f0e7ffee740 1 -- 172.21.15.40:0/1022213013 wait complete. 2024-04-27T00:12:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.107+0000 7f0e7ffee740 1 -- start start 2024-04-27T00:12:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.107+0000 7f0e7ffee740 1 Processor -- start 2024-04-27T00:12:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e7ffee740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcede0 0x5570e4bddc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e7ffee740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 0x5570e4bde160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e7ffee740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bd4940 0x5570e4bdf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e7ffee740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781abac0 con 0x5570e4bcede0 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e7ffee740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781acf30 con 0x5570e4bcf7d0 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e7ffee740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781ac190 con 0x5570e4bd4940 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e7ffee740 10 client.0 _ll_register_callbacks cb 0x7f0e64009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e76800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcede0 0x5570e4bddc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e76800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcede0 0x5570e4bddc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46546/0 (socket says 172.21.15.40:46546) 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e76800640 1 -- 172.21.15.40:0/3282665079 learned_addr learned my addr 172.21.15.40:0/3282665079 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e75400640 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 0x5570e4bde160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e76800640 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bd4940 0x5570e4bdf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e75400640 1 -- 172.21.15.40:0/3282665079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bd4940 msgr2=0x5570e4bdf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.108+0000 7f0e75400640 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bd4940 0x5570e4bdf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.109+0000 7f0e75400640 1 -- 172.21.15.40:0/3282665079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcede0 msgr2=0x5570e4bddc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.109+0000 7f0e75400640 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcede0 0x5570e4bddc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.109+0000 7f0e75400640 1 -- 172.21.15.40:0/3282665079 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570e4aedcb0 con 0x5570e4bcf7d0 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.109+0000 7f0e76800640 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcede0 0x5570e4bddc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:12:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.109+0000 7f0e75400640 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 0x5570e4bde160 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x5570e4bcfee0 tx=0x7f0e64005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d22ff30ccb3f50d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:12:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.109+0000 7f0e57e00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:12:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.109+0000 7f0e57e00640 1 -- 172.21.15.40:0/3282665079 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64060060 con 0x5570e4bcf7d0 2024-04-27T00:12:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.109+0000 7f0e57e00640 1 -- 172.21.15.40:0/3282665079 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f0e6405b430 con 0x5570e4bcf7d0 2024-04-27T00:12:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.109+0000 7f0e7ffee740 1 -- 172.21.15.40:0/3282665079 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e64005850 con 0x5570e4bcf7d0 2024-04-27T00:12:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.110+0000 7f0e57e00640 1 -- 172.21.15.40:0/3282665079 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0e64067050 con 0x5570e4bcf7d0 2024-04-27T00:12:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.110+0000 7f0e7ffee740 10 client.26261 fetch_fsmap learned FSMap version 24 2024-04-27T00:12:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.110+0000 7f0e7ffee740 1 -- 172.21.15.40:0/3282665079 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5570e4be08a0 con 0x5570e4bcf7d0 2024-04-27T00:12:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.110+0000 7f0e57e00640 1 -- 172.21.15.40:0/3282665079 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e64005850 con 0x5570e4bcf7d0 2024-04-27T00:12:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.110+0000 7f0e57e00640 1 -- 172.21.15.40:0/3282665079 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f0e6405a070 con 0x5570e4bcf7d0 2024-04-27T00:12:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.110+0000 7f0e7ffee740 10 client.26261 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:12:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.110+0000 7f0e7ffee740 10 client.26261 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:12:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.110+0000 7f0e7ffee740 20 client.26261 populate_metadata read hostname 'smithi040' 2024-04-27T00:12:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.110+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5570e4c4b8b0 0x5570e4c4dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:12:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.110+0000 7f0e7ffee740 4 client.26261 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:12:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.110+0000 7f0e7ffee740 1 -- 172.21.15.40:0/3282665079 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5570e4b3ae00 con 0x5570e4c4b8b0 2024-04-27T00:12:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.111+0000 7f0e76800640 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5570e4c4b8b0 0x5570e4c4dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:12:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.111+0000 7f0e76800640 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5570e4c4b8b0 0x5570e4c4dcd0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=580d50c644304dcc server_cookie=ddad515644bc3a61 in_seq=0 out_seq=0 2024-04-27T00:12:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.111+0000 7f0e57e00640 10 client.26261 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e57e00640 1 -- 172.21.15.40:0/3282665079 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5570e4b3ae00 con 0x5570e4c4b8b0 2024-04-27T00:12:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e57e00640 10 client.26261 handle_command_reply: tid=0 2024-04-27T00:12:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e7ffee740 1 client.26261 shutdown 2024-04-27T00:12:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e7ffee740 1 -- 172.21.15.40:0/3282665079 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 msgr2=0x5570e4bde160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 0x5570e4bde160 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x5570e4bcfee0 tx=0x7f0e64005e50 comp rx=0 tx=0).stop 2024-04-27T00:12:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e7ffee740 1 -- 172.21.15.40:0/3282665079 shutdown_connections 2024-04-27T00:12:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5570e4c4b8b0 0x5570e4c4dcd0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570e4bd4940 0x5570e4bdf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5570e4bcf7d0 0x5570e4bde160 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e7ffee740 1 --2- 172.21.15.40:0/3282665079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570e4bcede0 0x5570e4bddc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e57e00640 0 client.26261 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:12:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e7ffee740 1 -- 172.21.15.40:0/3282665079 >> 172.21.15.40:0/3282665079 conn(0x5570e4bca450 msgr2=0x5570e4bcd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e7ffee740 1 -- 172.21.15.40:0/3282665079 shutdown_connections 2024-04-27T00:12:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e7ffee740 1 -- 172.21.15.40:0/3282665079 wait complete. 2024-04-27T00:12:49.130 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.129+0000 7f0e7ffee740 20 client.26261 trim_cache size 0 max 16384 2024-04-27T00:12:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:12:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:12:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:12:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:12:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.130+0000 7f0e77200640 1 -- 172.21.15.40:0/2215081279 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0e5c0a3cb0 msgr2=0x7f0e5c0a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:49.131 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.130+0000 7f0e77200640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0e5c0a3cb0 0x7f0e5c0a6170 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x5570e4bcfbd0 tx=0x7f0e680d47e0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.130+0000 7f0e77200640 1 -- 172.21.15.40:0/2215081279 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e78101080 msgr2=0x7f0e7819ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:12:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.130+0000 7f0e77200640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e78101080 0x7f0e7819ac20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800e9f0 tx=0x7f0e6800eeb0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.130+0000 7f0e77200640 1 -- 172.21.15.40:0/2215081279 shutdown_connections 2024-04-27T00:12:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.130+0000 7f0e77200640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e78102380 0x7f0e781a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.130+0000 7f0e77200640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0e5c0a3cb0 0x7f0e5c0a6170 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.130+0000 7f0e77200640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e781019e0 0x7f0e7819b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.130+0000 7f0e77200640 1 --2- 172.21.15.40:0/2215081279 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0e78101080 0x7f0e7819ac20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:12:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.130+0000 7f0e77200640 1 -- 172.21.15.40:0/2215081279 >> 172.21.15.40:0/2215081279 conn(0x7f0e780fcc20 msgr2=0x7f0e780fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:12:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.130+0000 7f0e77200640 1 -- 172.21.15.40:0/2215081279 shutdown_connections 2024-04-27T00:12:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:12:49.130+0000 7f0e77200640 1 -- 172.21.15.40:0/2215081279 wait complete. 2024-04-27T00:12:50.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:50 smithi040 ceph-mon[28095]: pgmap v746: 129 pgs: 129 active+clean; 3.6 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:12:50.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:50 smithi080 ceph-mon[32362]: pgmap v746: 129 pgs: 129 active+clean; 3.6 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:12:50.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:50 smithi103 ceph-mon[32952]: pgmap v746: 129 pgs: 129 active+clean; 3.6 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:12:52.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:52 smithi040 ceph-mon[28095]: pgmap v747: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-27T00:12:52.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:52 smithi080 ceph-mon[32362]: pgmap v747: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-27T00:12:52.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:52 smithi103 ceph-mon[32952]: pgmap v747: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-27T00:12:54.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:54 smithi040 ceph-mon[28095]: pgmap v748: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:12:54.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:54 smithi080 ceph-mon[32362]: pgmap v748: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:12:54.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:54 smithi103 ceph-mon[32952]: pgmap v748: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:12:56.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:56 smithi040 ceph-mon[28095]: pgmap v749: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:56.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:56 smithi080 ceph-mon[32362]: pgmap v749: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:56.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:56 smithi103 ceph-mon[32952]: pgmap v749: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:12:58.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:12:58 smithi080 ceph-mon[32362]: pgmap v750: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:12:58.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:12:58 smithi103 ceph-mon[32952]: pgmap v750: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:12:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:12:58 smithi040 ceph-mon[28095]: pgmap v750: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:13:00.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:00 smithi080 ceph-mon[32362]: pgmap v751: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:13:00.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:00 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:13:00.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:00 smithi103 ceph-mon[32952]: pgmap v751: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:13:00.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:00 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:13:00.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:00 smithi040 ceph-mon[28095]: pgmap v751: 129 pgs: 129 active+clean; 3.7 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:13:00.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:00 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:13:01.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:13:01.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:13:01.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:13:01.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:13:01.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:13:01.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:13:01.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:13:01.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:13:01.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:13:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.277+0000 7fdeb7e00640 1 -- 172.21.15.40:0/422915735 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdeb81027d0 msgr2=0x7fdeb8102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.277+0000 7fdeb7e00640 1 --2- 172.21.15.40:0/422915735 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdeb81027d0 0x7fdeb8102bf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009860 tx=0x7fdea802a340 comp rx=0 tx=0).stop 2024-04-27T00:13:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.277+0000 7fdeb7e00640 1 -- 172.21.15.40:0/422915735 shutdown_connections 2024-04-27T00:13:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.277+0000 7fdeb7e00640 1 --2- 172.21.15.40:0/422915735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeb8104380 0x7fdeb810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.277+0000 7fdeb7e00640 1 --2- 172.21.15.40:0/422915735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeb81039c0 0x7fdeb8103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.277+0000 7fdeb7e00640 1 --2- 172.21.15.40:0/422915735 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdeb81027d0 0x7fdeb8102bf0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.277+0000 7fdeb7e00640 1 -- 172.21.15.40:0/422915735 >> 172.21.15.40:0/422915735 conn(0x7fdeb80fdd40 msgr2=0x7fdeb8100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.277+0000 7fdeb7e00640 1 -- 172.21.15.40:0/422915735 shutdown_connections 2024-04-27T00:13:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.277+0000 7fdeb7e00640 1 -- 172.21.15.40:0/422915735 wait complete. 2024-04-27T00:13:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.278+0000 7fdeb7e00640 1 Processor -- start 2024-04-27T00:13:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.278+0000 7fdeb7e00640 1 -- start start 2024-04-27T00:13:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.279+0000 7fdeb7e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdeb81027d0 0x7fdeb819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.279+0000 7fdeb7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeb81039c0 0x7fdeb819b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.279+0000 7fdeb7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeb8104380 0x7fdeb81a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.279+0000 7fdeb7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb810e2d0 con 0x7fdeb81039c0 2024-04-27T00:13:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.279+0000 7fdeb7e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb810e150 con 0x7fdeb81027d0 2024-04-27T00:13:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.279+0000 7fdeb7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb810e450 con 0x7fdeb8104380 2024-04-27T00:13:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.279+0000 7fdeb7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeb8104380 0x7fdeb81a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.279+0000 7fdeb7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeb8104380 0x7fdeb81a21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52562/0 (socket says 172.21.15.40:52562) 2024-04-27T00:13:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.279+0000 7fdeb7400640 1 -- 172.21.15.40:0/3639584597 learned_addr learned my addr 172.21.15.40:0/3639584597 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.279+0000 7fdeb6a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdeb81027d0 0x7fdeb819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.279+0000 7fdeb6000640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeb81039c0 0x7fdeb819b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.280+0000 7fdeb7400640 1 -- 172.21.15.40:0/3639584597 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdeb81027d0 msgr2=0x7fdeb819ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.280+0000 7fdeb7400640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdeb81027d0 0x7fdeb819ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.280+0000 7fdeb7400640 1 -- 172.21.15.40:0/3639584597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeb81039c0 msgr2=0x7fdeb819b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.280+0000 7fdeb7400640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeb81039c0 0x7fdeb819b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.280+0000 7fdeb7400640 1 -- 172.21.15.40:0/3639584597 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb81a2880 con 0x7fdeb8104380 2024-04-27T00:13:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.280+0000 7fdeb6a00640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdeb81027d0 0x7fdeb819ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:13:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.280+0000 7fdeb7400640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeb8104380 0x7fdeb81a21e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000eb10 tx=0x7fdeb000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.280+0000 7fdeaf400640 1 -- 172.21.15.40:0/3639584597 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb000ce20 con 0x7fdeb8104380 2024-04-27T00:13:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.280+0000 7fdeb7e00640 1 -- 172.21.15.40:0/3639584597 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb81a2b70 con 0x7fdeb8104380 2024-04-27T00:13:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.280+0000 7fdeaf400640 1 -- 172.21.15.40:0/3639584597 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdeb0005020 con 0x7fdeb8104380 2024-04-27T00:13:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.280+0000 7fdeaf400640 1 -- 172.21.15.40:0/3639584597 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0010690 con 0x7fdeb8104380 2024-04-27T00:13:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.281+0000 7fdeb7e00640 1 -- 172.21.15.40:0/3639584597 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb81a3070 con 0x7fdeb8104380 2024-04-27T00:13:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.284+0000 7fdec07e9740 1 Processor -- start 2024-04-27T00:13:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.284+0000 7fdeaf400640 1 -- 172.21.15.40:0/3639584597 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdeb0005510 con 0x7fdeb8104380 2024-04-27T00:13:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.284+0000 7fdec07e9740 1 -- start start 2024-04-27T00:13:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.284+0000 7fdec07e9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e978392de0 0x55e978393200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.284+0000 7fdec07e9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9783937d0 0x55e9783982d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.284+0000 7fdec07e9740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978398940 0x55e97839add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.284+0000 7fdec07e9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e9782f7380 con 0x55e978392de0 2024-04-27T00:13:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.284+0000 7fdec07e9740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55e978293c30 con 0x55e978398940 2024-04-27T00:13:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.284+0000 7fdec07e9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e9782f3340 con 0x55e9783937d0 2024-04-27T00:13:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.285+0000 7fdeb6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e978392de0 0x55e978393200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.285+0000 7fdeb6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e978392de0 0x55e978393200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36986/0 (socket says 172.21.15.40:36986) 2024-04-27T00:13:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.285+0000 7fdeb6000640 1 -- 172.21.15.40:0/2344299475 learned_addr learned my addr 172.21.15.40:0/2344299475 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.285+0000 7fdeb6a00640 1 --2- 172.21.15.40:0/2344299475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9783937d0 0x55e9783982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.285+0000 7fdeb6000640 1 --2- 172.21.15.40:0/2344299475 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978398940 0x55e97839add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.285+0000 7fdeb6000640 1 -- 172.21.15.40:0/2344299475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9783937d0 msgr2=0x55e9783982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.285+0000 7fdeb6000640 1 --2- 172.21.15.40:0/2344299475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9783937d0 0x55e9783982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.285+0000 7fdeb6000640 1 -- 172.21.15.40:0/2344299475 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978398940 msgr2=0x55e97839add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.285+0000 7fdeb6000640 1 --2- 172.21.15.40:0/2344299475 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978398940 0x55e97839add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.285+0000 7fdeb6000640 1 -- 172.21.15.40:0/2344299475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9782b5940 con 0x55e978392de0 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.285+0000 7fdeaf400640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fde9c0a3c30 0x7fde9c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.286+0000 7fdeb6000640 1 --2- 172.21.15.40:0/2344299475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e978392de0 0x55e978393200 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009fd0 tx=0x7fdea4009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b7695b41eeaf4a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.286+0000 7fdeaf400640 1 -- 172.21.15.40:0/3639584597 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fdeb00d0fa0 con 0x7fdeb8104380 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.286+0000 7fdeb6a00640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fde9c0a3c30 0x7fde9c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.286+0000 7fdea3e00640 1 -- 172.21.15.40:0/2344299475 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea4004050 con 0x55e978392de0 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.286+0000 7fdea3e00640 1 -- 172.21.15.40:0/2344299475 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdea40041f0 con 0x55e978392de0 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.286+0000 7fdea3e00640 1 -- 172.21.15.40:0/2344299475 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea400f7e0 con 0x55e978392de0 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.287+0000 7fdec07e9740 1 -- 172.21.15.40:0/2344299475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e978392de0 msgr2=0x55e978393200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.287+0000 7fdeb6a00640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fde9c0a3c30 0x7fde9c0a60f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdeb819c140 tx=0x7fdea802c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.287+0000 7fdec07e9740 1 --2- 172.21.15.40:0/2344299475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e978392de0 0x55e978393200 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009fd0 tx=0x7fdea4009890 comp rx=0 tx=0).stop 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.287+0000 7fdec07e9740 1 -- 172.21.15.40:0/2344299475 shutdown_connections 2024-04-27T00:13:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.287+0000 7fdec07e9740 1 --2- 172.21.15.40:0/2344299475 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978398940 0x55e97839add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.287+0000 7fdec07e9740 1 --2- 172.21.15.40:0/2344299475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9783937d0 0x55e9783982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.287+0000 7fdec07e9740 1 --2- 172.21.15.40:0/2344299475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e978392de0 0x55e978393200 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.287+0000 7fdec07e9740 1 -- 172.21.15.40:0/2344299475 >> 172.21.15.40:0/2344299475 conn(0x55e97838e450 msgr2=0x55e97838e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.287+0000 7fdec07e9740 1 -- 172.21.15.40:0/2344299475 shutdown_connections 2024-04-27T00:13:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.287+0000 7fdec07e9740 1 -- 172.21.15.40:0/2344299475 wait complete. 2024-04-27T00:13:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.287+0000 7fdec07e9740 1 -- start start 2024-04-27T00:13:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdec07e9740 1 Processor -- start 2024-04-27T00:13:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdec07e9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978392de0 0x55e97839cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdec07e9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e97839d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdec07e9740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978398940 0x55e9783a4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdec07e9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e97839b8f0 con 0x55e9783937d0 2024-04-27T00:13:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdec07e9740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x55e97827ec60 con 0x55e978398940 2024-04-27T00:13:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdec07e9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e97839ba70 con 0x55e978392de0 2024-04-27T00:13:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdec07e9740 10 client.0 _ll_register_callbacks cb 0x55e9782aae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:13:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdeb6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e97839d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdeb6a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978392de0 0x55e97839cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdeb6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e97839d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36994/0 (socket says 172.21.15.40:36994) 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdeb6000640 1 -- 172.21.15.40:0/3854609483 learned_addr learned my addr 172.21.15.40:0/3854609483 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.288+0000 7fdeb7400640 1 --2- 172.21.15.40:0/3854609483 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978398940 0x55e9783a4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.289+0000 7fdeb6000640 1 -- 172.21.15.40:0/3854609483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978392de0 msgr2=0x55e97839cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.289+0000 7fdeb6000640 1 --2- 172.21.15.40:0/3854609483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978392de0 0x55e97839cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.289+0000 7fdeb6000640 1 -- 172.21.15.40:0/3854609483 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978398940 msgr2=0x55e9783a4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.289+0000 7fdeb6000640 1 --2- 172.21.15.40:0/3854609483 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978398940 0x55e9783a4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.289+0000 7fdeb6000640 1 -- 172.21.15.40:0/3854609483 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e97828d510 con 0x55e9783937d0 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.289+0000 7fdeb6000640 1 --2- 172.21.15.40:0/3854609483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e97839d4d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdea4005e90 tx=0x7fdea40056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76ccbf4b1d523c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.289+0000 7fdea0c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.290+0000 7fdea0c00640 1 -- 172.21.15.40:0/3854609483 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea402f060 con 0x55e9783937d0 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.290+0000 7fdea0c00640 1 -- 172.21.15.40:0/3854609483 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fdea40043a0 con 0x55e9783937d0 2024-04-27T00:13:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.290+0000 7fdec07e9740 1 -- 172.21.15.40:0/3854609483 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdea40041f0 con 0x55e9783937d0 2024-04-27T00:13:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.290+0000 7fdea0c00640 1 -- 172.21.15.40:0/3854609483 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdea4036050 con 0x55e9783937d0 2024-04-27T00:13:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.290+0000 7fdec07e9740 10 client.16161 fetch_fsmap learned FSMap version 24 2024-04-27T00:13:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.290+0000 7fdec07e9740 1 -- 172.21.15.40:0/3854609483 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e978009210 con 0x55e9783937d0 2024-04-27T00:13:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.290+0000 7fdea0c00640 1 -- 172.21.15.40:0/3854609483 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdea40041f0 con 0x55e9783937d0 2024-04-27T00:13:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.290+0000 7fdea0c00640 1 -- 172.21.15.40:0/3854609483 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdea400fcd0 con 0x55e9783937d0 2024-04-27T00:13:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.291+0000 7fdec07e9740 10 client.16161 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:13:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.291+0000 7fdec07e9740 10 client.16161 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:13:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.291+0000 7fdec07e9740 20 client.16161 populate_metadata read hostname 'smithi040' 2024-04-27T00:13:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.291+0000 7fdec07e9740 1 --2- 172.21.15.40:0/3854609483 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e9783a89a0 0x55e9783aad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.291+0000 7fdec07e9740 4 client.16161 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:13:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.291+0000 7fdec07e9740 1 -- 172.21.15.40:0/3854609483 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e978304f90 con 0x55e9783a89a0 2024-04-27T00:13:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.291+0000 7fdeb6a00640 1 --2- 172.21.15.40:0/3854609483 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e9783a89a0 0x55e9783aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.292+0000 7fdeb6a00640 1 --2- 172.21.15.40:0/3854609483 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e9783a89a0 0x55e9783aad90 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4672d59720aae9c server_cookie=3a7c06b6df4a5530 in_seq=0 out_seq=0 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.292+0000 7fdea0c00640 10 client.16161 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdea0c00640 1 -- 172.21.15.40:0/3854609483 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e978304f90 con 0x55e9783a89a0 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdea0c00640 10 client.16161 handle_command_reply: tid=0 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdec07e9740 1 client.16161 shutdown 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdec07e9740 1 -- 172.21.15.40:0/3854609483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 msgr2=0x55e97839d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdec07e9740 1 --2- 172.21.15.40:0/3854609483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e97839d4d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdea4005e90 tx=0x7fdea40056d0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdec07e9740 1 -- 172.21.15.40:0/3854609483 shutdown_connections 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdec07e9740 1 --2- 172.21.15.40:0/3854609483 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e9783a89a0 0x55e9783aad90 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdec07e9740 1 --2- 172.21.15.40:0/3854609483 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978398940 0x55e9783a4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdec07e9740 1 --2- 172.21.15.40:0/3854609483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e97839d4d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdea0c00640 0 client.16161 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdec07e9740 1 --2- 172.21.15.40:0/3854609483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978392de0 0x55e97839cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdec07e9740 1 -- 172.21.15.40:0/3854609483 >> 172.21.15.40:0/3854609483 conn(0x55e97838e450 msgr2=0x55e97838d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdec07e9740 1 -- 172.21.15.40:0/3854609483 shutdown_connections 2024-04-27T00:13:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdec07e9740 1 -- 172.21.15.40:0/3854609483 wait complete. 2024-04-27T00:13:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.293+0000 7fdec07e9740 20 client.16161 trim_cache size 0 max 16384 2024-04-27T00:13:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.310+0000 7fdec07e9740 1 Processor -- start 2024-04-27T00:13:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.310+0000 7fdec07e9740 1 -- start start 2024-04-27T00:13:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.310+0000 7fdec07e9740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978392de0 0x55e9783913d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.310+0000 7fdec07e9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e9783a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.310+0000 7fdec07e9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978398940 0x55e978391910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.310+0000 7fdec07e9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb81afdc0 con 0x55e9783937d0 2024-04-27T00:13:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.310+0000 7fdec07e9740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb81ad690 con 0x55e978392de0 2024-04-27T00:13:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.310+0000 7fdec07e9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb81ab4e0 con 0x55e978398940 2024-04-27T00:13:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.310+0000 7fdeb6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e9783a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.310+0000 7fdeb6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e9783a44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36998/0 (socket says 172.21.15.40:36998) 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.310+0000 7fdeb6000640 1 -- 172.21.15.40:0/359602603 learned_addr learned my addr 172.21.15.40:0/359602603 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.310+0000 7fdeb6a00640 1 --2- 172.21.15.40:0/359602603 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978392de0 0x55e9783913d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.311+0000 7fdeb7400640 1 --2- 172.21.15.40:0/359602603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978398940 0x55e978391910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.311+0000 7fdeb6000640 1 -- 172.21.15.40:0/359602603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978398940 msgr2=0x55e978391910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.311+0000 7fdeb6000640 1 --2- 172.21.15.40:0/359602603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978398940 0x55e978391910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.311+0000 7fdeb6000640 1 -- 172.21.15.40:0/359602603 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978392de0 msgr2=0x55e9783913d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.311+0000 7fdeb6000640 1 --2- 172.21.15.40:0/359602603 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978392de0 0x55e9783913d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.311+0000 7fdeb6000640 1 -- 172.21.15.40:0/359602603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e97839c3b0 con 0x55e9783937d0 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.311+0000 7fdeb6000640 1 --2- 172.21.15.40:0/359602603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e9783a44d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdea400a010 tx=0x7fdea4003900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16beb953abf7e744 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.311+0000 7fdea3400640 1 -- 172.21.15.40:0/359602603 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea402f070 con 0x55e9783937d0 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.311+0000 7fdea3400640 1 -- 172.21.15.40:0/359602603 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdea4013460 con 0x55e9783937d0 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.311+0000 7fdea3400640 1 -- 172.21.15.40:0/359602603 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea40050c0 con 0x55e9783937d0 2024-04-27T00:13:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.312+0000 7fdec07e9740 1 -- 172.21.15.40:0/359602603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 msgr2=0x55e9783a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.312+0000 7fdec07e9740 1 --2- 172.21.15.40:0/359602603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e9783a44d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdea400a010 tx=0x7fdea4003900 comp rx=0 tx=0).stop 2024-04-27T00:13:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.312+0000 7fdec07e9740 1 -- 172.21.15.40:0/359602603 shutdown_connections 2024-04-27T00:13:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.312+0000 7fdec07e9740 1 --2- 172.21.15.40:0/359602603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978398940 0x55e978391910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.312+0000 7fdec07e9740 1 --2- 172.21.15.40:0/359602603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e9783a44d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.312+0000 7fdec07e9740 1 --2- 172.21.15.40:0/359602603 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978392de0 0x55e9783913d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.312+0000 7fdec07e9740 1 -- 172.21.15.40:0/359602603 >> 172.21.15.40:0/359602603 conn(0x55e97838e450 msgr2=0x55e9783949b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.312+0000 7fdec07e9740 1 -- 172.21.15.40:0/359602603 shutdown_connections 2024-04-27T00:13:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.312+0000 7fdec07e9740 1 -- 172.21.15.40:0/359602603 wait complete. 2024-04-27T00:13:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.312+0000 7fdec07e9740 1 -- start start 2024-04-27T00:13:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.313+0000 7fdec07e9740 1 Processor -- start 2024-04-27T00:13:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.313+0000 7fdec07e9740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978392de0 0x55e9783a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.313+0000 7fdec07e9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e9783a2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.313+0000 7fdec07e9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978398940 0x55e9783a3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.313+0000 7fdec07e9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb81abb50 con 0x55e9783937d0 2024-04-27T00:13:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.313+0000 7fdec07e9740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb81acfc0 con 0x55e978392de0 2024-04-27T00:13:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.313+0000 7fdec07e9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb81ac220 con 0x55e978398940 2024-04-27T00:13:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.313+0000 7fdec07e9740 10 client.0 _ll_register_callbacks cb 0x7fdeb00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.313+0000 7fdeb6a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978392de0 0x55e9783a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.313+0000 7fdeb6a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978392de0 0x55e9783a1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:38606/0 (socket says 172.21.15.40:38606) 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.313+0000 7fdeb6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e9783a2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.313+0000 7fdeb6a00640 1 -- 172.21.15.40:0/2512409011 learned_addr learned my addr 172.21.15.40:0/2512409011 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.314+0000 7fdeb7400640 1 --2- 172.21.15.40:0/2512409011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978398940 0x55e9783a3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.314+0000 7fdeb6a00640 1 -- 172.21.15.40:0/2512409011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978398940 msgr2=0x55e9783a3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.314+0000 7fdeb6a00640 1 --2- 172.21.15.40:0/2512409011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978398940 0x55e9783a3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.314+0000 7fdeb6a00640 1 -- 172.21.15.40:0/2512409011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 msgr2=0x55e9783a2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.314+0000 7fdeb6a00640 1 --2- 172.21.15.40:0/2512409011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e9783a2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.314+0000 7fdeb6a00640 1 -- 172.21.15.40:0/2512409011 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9780d5620 con 0x55e978392de0 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.314+0000 7fdeb6a00640 1 --2- 172.21.15.40:0/2512409011 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978392de0 0x55e9783a1b20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdea80057c0 tx=0x7fdea802a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ba19aedaefff763 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.314+0000 7fdea0c00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:13:02.317 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.314+0000 7fdea0c00640 1 -- 172.21.15.40:0/2512409011 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea8037040 con 0x55e978392de0 2024-04-27T00:13:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.314+0000 7fdea0c00640 1 -- 172.21.15.40:0/2512409011 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fdea80327f0 con 0x55e978392de0 2024-04-27T00:13:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.314+0000 7fdec07e9740 1 -- 172.21.15.40:0/2512409011 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdea4013460 con 0x55e978392de0 2024-04-27T00:13:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.314+0000 7fdea0c00640 1 -- 172.21.15.40:0/2512409011 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdea805d050 con 0x55e978392de0 2024-04-27T00:13:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.315+0000 7fdec07e9740 10 client.26279 fetch_fsmap learned FSMap version 24 2024-04-27T00:13:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.315+0000 7fdec07e9740 1 -- 172.21.15.40:0/2512409011 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e9782b1cb0 con 0x55e978392de0 2024-04-27T00:13:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.315+0000 7fdea0c00640 1 -- 172.21.15.40:0/2512409011 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdea4013460 con 0x55e978392de0 2024-04-27T00:13:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.315+0000 7fdea0c00640 1 -- 172.21.15.40:0/2512409011 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdea8049040 con 0x55e978392de0 2024-04-27T00:13:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.315+0000 7fdec07e9740 10 client.26279 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:13:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.315+0000 7fdec07e9740 10 client.26279 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:13:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.315+0000 7fdec07e9740 20 client.26279 populate_metadata read hostname 'smithi040' 2024-04-27T00:13:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.315+0000 7fdec07e9740 1 --2- 172.21.15.40:0/2512409011 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e97840f3b0 0x55e9784117d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:02.318 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.315+0000 7fdec07e9740 4 client.26279 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:13:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.315+0000 7fdec07e9740 1 -- 172.21.15.40:0/2512409011 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e9782fee00 con 0x55e97840f3b0 2024-04-27T00:13:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.316+0000 7fdeb6000640 1 --2- 172.21.15.40:0/2512409011 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e97840f3b0 0x55e9784117d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.316+0000 7fdeb6000640 1 --2- 172.21.15.40:0/2512409011 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e97840f3b0 0x55e9784117d0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89c8af5b05c09a2e server_cookie=615e434ae1b03f27 in_seq=0 out_seq=0 2024-04-27T00:13:02.319 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.316+0000 7fdea0c00640 10 client.26279 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:13:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdea0c00640 1 -- 172.21.15.40:0/2512409011 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e9782fee00 con 0x55e97840f3b0 2024-04-27T00:13:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdea0c00640 10 client.26279 handle_command_reply: tid=0 2024-04-27T00:13:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdec07e9740 1 client.26279 shutdown 2024-04-27T00:13:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdec07e9740 1 -- 172.21.15.40:0/2512409011 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978392de0 msgr2=0x55e9783a1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdec07e9740 1 --2- 172.21.15.40:0/2512409011 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978392de0 0x55e9783a1b20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdea80057c0 tx=0x7fdea802a340 comp rx=0 tx=0).stop 2024-04-27T00:13:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdec07e9740 1 -- 172.21.15.40:0/2512409011 shutdown_connections 2024-04-27T00:13:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdec07e9740 1 --2- 172.21.15.40:0/2512409011 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x55e97840f3b0 0x55e9784117d0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdec07e9740 1 --2- 172.21.15.40:0/2512409011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e978398940 0x55e9783a3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdec07e9740 1 --2- 172.21.15.40:0/2512409011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9783937d0 0x55e9783a2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdec07e9740 1 --2- 172.21.15.40:0/2512409011 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x55e978392de0 0x55e9783a1b20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdec07e9740 1 -- 172.21.15.40:0/2512409011 >> 172.21.15.40:0/2512409011 conn(0x55e97838e450 msgr2=0x55e9783913d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdea0c00640 0 client.26279 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:13:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdec07e9740 1 -- 172.21.15.40:0/2512409011 shutdown_connections 2024-04-27T00:13:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdec07e9740 1 -- 172.21.15.40:0/2512409011 wait complete. 2024-04-27T00:13:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.333+0000 7fdec07e9740 20 client.26279 trim_cache size 0 max 16384 2024-04-27T00:13:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:13:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:13:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:13:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:13:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.334+0000 7fdeb7e00640 1 -- 172.21.15.40:0/3639584597 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fde9c0a3c30 msgr2=0x7fde9c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.334+0000 7fdeb7e00640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fde9c0a3c30 0x7fde9c0a60f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdeb819c140 tx=0x7fdea802c040 comp rx=0 tx=0).stop 2024-04-27T00:13:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.334+0000 7fdeb7e00640 1 -- 172.21.15.40:0/3639584597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeb8104380 msgr2=0x7fdeb81a21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.334+0000 7fdeb7e00640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeb8104380 0x7fdeb81a21e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000eb10 tx=0x7fdeb000ef10 comp rx=0 tx=0).stop 2024-04-27T00:13:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.334+0000 7fdeb7e00640 1 -- 172.21.15.40:0/3639584597 shutdown_connections 2024-04-27T00:13:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.334+0000 7fdeb7e00640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fde9c0a3c30 0x7fde9c0a60f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.334+0000 7fdeb7e00640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdeb8104380 0x7fdeb81a21e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.334+0000 7fdeb7e00640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeb81039c0 0x7fdeb819b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.334+0000 7fdeb7e00640 1 --2- 172.21.15.40:0/3639584597 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdeb81027d0 0x7fdeb819ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.334+0000 7fdeb7e00640 1 -- 172.21.15.40:0/3639584597 >> 172.21.15.40:0/3639584597 conn(0x7fdeb80fdd40 msgr2=0x7fdeb8074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.335+0000 7fdeb7e00640 1 -- 172.21.15.40:0/3639584597 shutdown_connections 2024-04-27T00:13:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:02.335+0000 7fdeb7e00640 1 -- 172.21.15.40:0/3639584597 wait complete. 2024-04-27T00:13:02.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:02 smithi040 ceph-mon[28095]: pgmap v752: 129 pgs: 129 active+clean; 3.8 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:13:03.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:02 smithi080 ceph-mon[32362]: pgmap v752: 129 pgs: 129 active+clean; 3.8 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:13:03.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:02 smithi103 ceph-mon[32952]: pgmap v752: 129 pgs: 129 active+clean; 3.8 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:13:04.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:04 smithi040 ceph-mon[28095]: pgmap v753: 129 pgs: 129 active+clean; 3.8 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:13:05.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:04 smithi080 ceph-mon[32362]: pgmap v753: 129 pgs: 129 active+clean; 3.8 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:13:05.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:04 smithi103 ceph-mon[32952]: pgmap v753: 129 pgs: 129 active+clean; 3.8 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 13 op/s 2024-04-27T00:13:06.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:06 smithi040 ceph-mon[28095]: pgmap v754: 129 pgs: 129 active+clean; 3.8 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:13:07.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:06 smithi080 ceph-mon[32362]: pgmap v754: 129 pgs: 129 active+clean; 3.8 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:13:07.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:06 smithi103 ceph-mon[32952]: pgmap v754: 129 pgs: 129 active+clean; 3.8 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-27T00:13:08.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:08 smithi040 ceph-mon[28095]: pgmap v755: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:13:09.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:08 smithi080 ceph-mon[32362]: pgmap v755: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:13:09.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:08 smithi103 ceph-mon[32952]: pgmap v755: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 17 op/s 2024-04-27T00:13:11.049 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:10 smithi080 ceph-mon[32362]: pgmap v756: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:13:11.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:10 smithi103 ceph-mon[32952]: pgmap v756: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:13:11.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:10 smithi040 ceph-mon[28095]: pgmap v756: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:13:13.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:12 smithi080 ceph-mon[32362]: pgmap v757: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:13:13.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:12 smithi103 ceph-mon[32952]: pgmap v757: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:13:13.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:12 smithi040 ceph-mon[28095]: pgmap v757: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:13:15.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:14 smithi080 ceph-mon[32362]: pgmap v758: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:13:15.097 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:14 smithi103 ceph-mon[32952]: pgmap v758: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:13:15.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:14 smithi040 ceph-mon[28095]: pgmap v758: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:13:15.483 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.483+0000 7f7946800640 1 -- 172.21.15.40:0/4287775993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7940109fd0 msgr2=0x7f794010a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.483+0000 7f7946800640 1 --2- 172.21.15.40:0/4287775993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7940109fd0 0x7f794010a450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f792c009900 tx=0x7f792c02a400 comp rx=0 tx=0).stop 2024-04-27T00:13:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.483+0000 7f7946800640 1 -- 172.21.15.40:0/4287775993 shutdown_connections 2024-04-27T00:13:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.483+0000 7f7946800640 1 --2- 172.21.15.40:0/4287775993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f794010a990 0x7f7940111220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.483+0000 7f7946800640 1 --2- 172.21.15.40:0/4287775993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7940109fd0 0x7f794010a450 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.483+0000 7f7946800640 1 --2- 172.21.15.40:0/4287775993 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7940108de0 0x7f7940109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.483+0000 7f7946800640 1 -- 172.21.15.40:0/4287775993 >> 172.21.15.40:0/4287775993 conn(0x7f7940077dc0 msgr2=0x7f794007a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.483+0000 7f7946800640 1 -- 172.21.15.40:0/4287775993 shutdown_connections 2024-04-27T00:13:15.484 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.483+0000 7f7946800640 1 -- 172.21.15.40:0/4287775993 wait complete. 2024-04-27T00:13:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.484+0000 7f7946800640 1 Processor -- start 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.484+0000 7f7946800640 1 -- start start 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.484+0000 7f7946800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940108de0 0x7f794019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.484+0000 7f7946800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7940109fd0 0x7f794019f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.484+0000 7f7946800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794010a990 0x7f79401a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.484+0000 7f7946800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7940114940 con 0x7f7940108de0 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.484+0000 7f7946800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f79401147c0 con 0x7f7940109fd0 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.484+0000 7f7946800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7940114ac0 con 0x7f794010a990 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.485+0000 7f7945400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940108de0 0x7f794019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.485+0000 7f7945400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940108de0 0x7f794019f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41624/0 (socket says 172.21.15.40:41624) 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.485+0000 7f7945400640 1 -- 172.21.15.40:0/3010033943 learned_addr learned my addr 172.21.15.40:0/3010033943 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.485+0000 7f7944a00640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7940109fd0 0x7f794019f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.485+0000 7f7945e00640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794010a990 0x7f79401a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.485+0000 7f7945400640 1 -- 172.21.15.40:0/3010033943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794010a990 msgr2=0x7f79401a6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.485+0000 7f7945400640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794010a990 0x7f79401a6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.485+0000 7f7945400640 1 -- 172.21.15.40:0/3010033943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7940109fd0 msgr2=0x7f794019f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.485+0000 7f7945400640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7940109fd0 0x7f794019f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.485+0000 7f7945400640 1 -- 172.21.15.40:0/3010033943 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79401a6c00 con 0x7f7940108de0 2024-04-27T00:13:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.485+0000 7f7944a00640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7940109fd0 0x7f794019f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:13:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.486+0000 7f7945400640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940108de0 0x7f794019f030 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f793400e9c0 tx=0x7f793400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.486+0000 7f793e000640 1 -- 172.21.15.40:0/3010033943 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793400ccc0 con 0x7f7940108de0 2024-04-27T00:13:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.486+0000 7f793e000640 1 -- 172.21.15.40:0/3010033943 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79340035f0 con 0x7f7940108de0 2024-04-27T00:13:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.486+0000 7f793e000640 1 -- 172.21.15.40:0/3010033943 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793400ce60 con 0x7f7940108de0 2024-04-27T00:13:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.486+0000 7f7946800640 1 -- 172.21.15.40:0/3010033943 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79401a6ef0 con 0x7f7940108de0 2024-04-27T00:13:15.488 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.486+0000 7f7946800640 1 -- 172.21.15.40:0/3010033943 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79401a7300 con 0x7f7940108de0 2024-04-27T00:13:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.489+0000 7f794ac72740 1 Processor -- start 2024-04-27T00:13:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f794ac72740 1 -- start start 2024-04-27T00:13:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f793e000640 1 -- 172.21.15.40:0/3010033943 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f79340037a0 con 0x7f7940108de0 2024-04-27T00:13:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f794ac72740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61b9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f794ac72740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b97d0 0x5600c61be2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f794ac72740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f794ac72740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5600c611d380 con 0x5600c61be940 2024-04-27T00:13:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f794ac72740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5600c60b9c30 con 0x5600c61b8de0 2024-04-27T00:13:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f794ac72740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5600c6119340 con 0x5600c61b97d0 2024-04-27T00:13:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f7945e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61b9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f7945e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61b9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:59998/0 (socket says 172.21.15.40:59998) 2024-04-27T00:13:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f7945e00640 1 -- 172.21.15.40:0/1038622787 learned_addr learned my addr 172.21.15.40:0/1038622787 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f7945400640 1 --2- 172.21.15.40:0/1038622787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b97d0 0x5600c61be2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.490+0000 7f7944a00640 1 --2- 172.21.15.40:0/1038622787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.491+0000 7f7945e00640 1 -- 172.21.15.40:0/1038622787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b97d0 msgr2=0x5600c61be2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.491+0000 7f7945e00640 1 --2- 172.21.15.40:0/1038622787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b97d0 0x5600c61be2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.491+0000 7f7945e00640 1 -- 172.21.15.40:0/1038622787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 msgr2=0x5600c61c0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.491+0000 7f7945e00640 1 --2- 172.21.15.40:0/1038622787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61c0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.491+0000 7f7945e00640 1 -- 172.21.15.40:0/1038622787 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600c60db940 con 0x5600c61b8de0 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.491+0000 7f7944a00640 1 --2- 172.21.15.40:0/1038622787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.491+0000 7f7945e00640 1 --2- 172.21.15.40:0/1038622787 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61b9200 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7938011440 tx=0x7f7938011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd3db47fc7367c4e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f7932600640 1 -- 172.21.15.40:0/1038622787 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938003c30 con 0x5600c61b8de0 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f7932600640 1 -- 172.21.15.40:0/1038622787 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7938003dd0 con 0x5600c61b8de0 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f7932600640 1 -- 172.21.15.40:0/1038622787 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793800fa40 con 0x5600c61b8de0 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f793e000640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f79280a3c30 0x7f79280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f793e000640 1 -- 172.21.15.40:0/3010033943 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f79340d0e10 con 0x7f7940108de0 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f794ac72740 1 -- 172.21.15.40:0/1038622787 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 msgr2=0x5600c61b9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f794ac72740 1 --2- 172.21.15.40:0/1038622787 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61b9200 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7938011440 tx=0x7f7938011900 comp rx=0 tx=0).stop 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f794ac72740 1 -- 172.21.15.40:0/1038622787 shutdown_connections 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f794ac72740 1 --2- 172.21.15.40:0/1038622787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f794ac72740 1 --2- 172.21.15.40:0/1038622787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b97d0 0x5600c61be2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f794ac72740 1 --2- 172.21.15.40:0/1038622787 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61b9200 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f7945400640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f79280a3c30 0x7f79280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.492+0000 7f794ac72740 1 -- 172.21.15.40:0/1038622787 >> 172.21.15.40:0/1038622787 conn(0x5600c61b4450 msgr2=0x5600c61b4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.493+0000 7f794ac72740 1 -- 172.21.15.40:0/1038622787 shutdown_connections 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.493+0000 7f794ac72740 1 -- 172.21.15.40:0/1038622787 wait complete. 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.493+0000 7f794ac72740 1 -- start start 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.493+0000 7f794ac72740 1 Processor -- start 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.493+0000 7f7945400640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f79280a3c30 0x7f79280a60f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x5600c61b9bd0 tx=0x7f79340d48c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.494+0000 7f794ac72740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61c3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.494+0000 7f794ac72740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b97d0 0x5600c61c3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.494+0000 7f794ac72740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.494+0000 7f794ac72740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5600c61c18f0 con 0x5600c61be940 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.494+0000 7f794ac72740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x5600c60a4c60 con 0x5600c61b8de0 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.494+0000 7f794ac72740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5600c61c1a70 con 0x5600c61b97d0 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.494+0000 7f794ac72740 10 client.0 _ll_register_callbacks cb 0x5600c60d0e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.494+0000 7f7945e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61c3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.494+0000 7f7945e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61c3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:60008/0 (socket says 172.21.15.40:60008) 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.494+0000 7f7945e00640 1 -- 172.21.15.40:0/1787161152 learned_addr learned my addr 172.21.15.40:0/1787161152 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.494+0000 7f7944a00640 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b97d0 0x5600c61c3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.494+0000 7f7945e00640 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.495+0000 7f7944a00640 1 -- 172.21.15.40:0/1787161152 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 msgr2=0x5600c61c3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.495+0000 7f7944a00640 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61c3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.495+0000 7f7944a00640 1 -- 172.21.15.40:0/1787161152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 msgr2=0x5600c61ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.495+0000 7f7944a00640 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.495+0000 7f7944a00640 1 -- 172.21.15.40:0/1787161152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600c60b3510 con 0x5600c61b97d0 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.495+0000 7f7945e00640 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.495+0000 7f7944a00640 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b97d0 0x5600c61c3590 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f792c02a9f0 tx=0x7f792c005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de47d685038d85f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.495+0000 7f7926a00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.495+0000 7f7926a00640 1 -- 172.21.15.40:0/1787161152 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c035030 con 0x5600c61b97d0 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.495+0000 7f7926a00640 1 -- 172.21.15.40:0/1787161152 <== mon.1 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f792c0048c0 con 0x5600c61b97d0 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.495+0000 7f794ac72740 1 -- 172.21.15.40:0/1787161152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7938003dd0 con 0x5600c61b97d0 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.496+0000 7f7926a00640 1 -- 172.21.15.40:0/1787161152 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f792c04d050 con 0x5600c61b97d0 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.496+0000 7f794ac72740 10 client.26323 fetch_fsmap learned FSMap version 24 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.496+0000 7f794ac72740 1 -- 172.21.15.40:0/1787161152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5600c5e2f210 con 0x5600c61b97d0 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.496+0000 7f7926a00640 1 -- 172.21.15.40:0/1787161152 <== mon.1 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7938003dd0 con 0x5600c61b97d0 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.496+0000 7f7926a00640 1 -- 172.21.15.40:0/1787161152 <== mon.1 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f792c003b50 con 0x5600c61b97d0 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.496+0000 7f794ac72740 10 client.26323 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:13:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.496+0000 7f794ac72740 10 client.26323 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:13:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.496+0000 7f794ac72740 20 client.26323 populate_metadata read hostname 'smithi040' 2024-04-27T00:13:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.496+0000 7f794ac72740 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5600c61ceb70 0x5600c61d0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.496+0000 7f794ac72740 4 client.26323 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-27T00:13:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.496+0000 7f794ac72740 1 -- 172.21.15.40:0/1787161152 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600c612af90 con 0x5600c61ceb70 2024-04-27T00:13:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.497+0000 7f7945e00640 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5600c61ceb70 0x5600c61d0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.497+0000 7f7945e00640 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5600c61ceb70 0x5600c61d0f60 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=411dd36a15cfa408 server_cookie=a3bc32fa227f5f68 in_seq=0 out_seq=0 2024-04-27T00:13:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.497+0000 7f7926a00640 10 client.26323 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:13:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.498+0000 7f7926a00640 1 -- 172.21.15.40:0/1787161152 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5600c612af90 con 0x5600c61ceb70 2024-04-27T00:13:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.498+0000 7f7926a00640 10 client.26323 handle_command_reply: tid=0 2024-04-27T00:13:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.498+0000 7f794ac72740 1 client.26323 shutdown 2024-04-27T00:13:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.498+0000 7f794ac72740 1 -- 172.21.15.40:0/1787161152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b97d0 msgr2=0x5600c61c3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.498+0000 7f794ac72740 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b97d0 0x5600c61c3590 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f792c02a9f0 tx=0x7f792c005270 comp rx=0 tx=0).stop 2024-04-27T00:13:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.498+0000 7f794ac72740 1 -- 172.21.15.40:0/1787161152 shutdown_connections 2024-04-27T00:13:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.498+0000 7f794ac72740 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5600c61ceb70 0x5600c61d0f60 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.499+0000 7f794ac72740 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.499+0000 7f794ac72740 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b97d0 0x5600c61c3590 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.499+0000 7f7926a00640 0 client.26323 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:13:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.499+0000 7f794ac72740 1 --2- 172.21.15.40:0/1787161152 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61c3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.499+0000 7f794ac72740 1 -- 172.21.15.40:0/1787161152 >> 172.21.15.40:0/1787161152 conn(0x5600c61b4450 msgr2=0x5600c61b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.499+0000 7f794ac72740 1 -- 172.21.15.40:0/1787161152 shutdown_connections 2024-04-27T00:13:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.499+0000 7f794ac72740 1 -- 172.21.15.40:0/1787161152 wait complete. 2024-04-27T00:13:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.499+0000 7f794ac72740 20 client.26323 trim_cache size 0 max 16384 2024-04-27T00:13:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.516+0000 7f794ac72740 1 Processor -- start 2024-04-27T00:13:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.516+0000 7f794ac72740 1 -- start start 2024-04-27T00:13:15.517 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.516+0000 7f794ac72740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.518 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.516+0000 7f794ac72740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61b97d0 0x5600c61ca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.518 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.516+0000 7f794ac72740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61be940 0x5600c61d5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.518 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.517+0000 7f794ac72740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f79401b4190 con 0x5600c61b97d0 2024-04-27T00:13:15.518 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.517+0000 7f794ac72740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f79401b1a60 con 0x5600c61b8de0 2024-04-27T00:13:15.518 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.517+0000 7f794ac72740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79401af8b0 con 0x5600c61be940 2024-04-27T00:13:15.518 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.517+0000 7f7945e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.518 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.517+0000 7f7945e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61b7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:60014/0 (socket says 172.21.15.40:60014) 2024-04-27T00:13:15.518 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.517+0000 7f7945e00640 1 -- 172.21.15.40:0/3941525331 learned_addr learned my addr 172.21.15.40:0/3941525331 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:15.518 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.517+0000 7f7945e00640 1 --2- 172.21.15.40:0/3941525331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61be940 0x5600c61d5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.518 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.517+0000 7f7945e00640 1 -- 172.21.15.40:0/3941525331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61be940 msgr2=0x5600c61d5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:15.518 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.517+0000 7f7945e00640 1 --2- 172.21.15.40:0/3941525331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61be940 0x5600c61d5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.519 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.517+0000 7f7945e00640 1 -- 172.21.15.40:0/3941525331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61b97d0 msgr2=0x5600c61ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:15.519 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.517+0000 7f7945e00640 1 --2- 172.21.15.40:0/3941525331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61b97d0 0x5600c61ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.519 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.517+0000 7f7945e00640 1 -- 172.21.15.40:0/3941525331 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600c5efb620 con 0x5600c61b8de0 2024-04-27T00:13:15.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.518+0000 7f7945e00640 1 --2- 172.21.15.40:0/3941525331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61b7bf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7938009df0 tx=0x7f7938009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d821274b70b46c82 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:15.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.518+0000 7f7931c00640 1 -- 172.21.15.40:0/3941525331 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938004250 con 0x5600c61b8de0 2024-04-27T00:13:15.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.518+0000 7f7931c00640 1 -- 172.21.15.40:0/3941525331 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7938003db0 con 0x5600c61b8de0 2024-04-27T00:13:15.520 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.518+0000 7f7931c00640 1 -- 172.21.15.40:0/3941525331 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793800fc90 con 0x5600c61b8de0 2024-04-27T00:13:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.519+0000 7f794ac72740 1 -- 172.21.15.40:0/3941525331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 msgr2=0x5600c61b7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.519+0000 7f794ac72740 1 --2- 172.21.15.40:0/3941525331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61b7bf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7938009df0 tx=0x7f7938009500 comp rx=0 tx=0).stop 2024-04-27T00:13:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.519+0000 7f794ac72740 1 -- 172.21.15.40:0/3941525331 shutdown_connections 2024-04-27T00:13:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.519+0000 7f794ac72740 1 --2- 172.21.15.40:0/3941525331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61be940 0x5600c61d5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.519+0000 7f794ac72740 1 --2- 172.21.15.40:0/3941525331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61b97d0 0x5600c61ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.519+0000 7f794ac72740 1 --2- 172.21.15.40:0/3941525331 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b8de0 0x5600c61b7bf0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.519+0000 7f794ac72740 1 -- 172.21.15.40:0/3941525331 >> 172.21.15.40:0/3941525331 conn(0x5600c61b4450 msgr2=0x5600c61b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.520+0000 7f794ac72740 1 -- 172.21.15.40:0/3941525331 shutdown_connections 2024-04-27T00:13:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.520+0000 7f794ac72740 1 -- 172.21.15.40:0/3941525331 wait complete. 2024-04-27T00:13:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.520+0000 7f794ac72740 1 -- start start 2024-04-27T00:13:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.520+0000 7f794ac72740 1 Processor -- start 2024-04-27T00:13:15.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f794ac72740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b8de0 0x5600c61c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f794ac72740 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b97d0 0x5600c61c8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f794ac72740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61c9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f794ac72740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f79401aff20 con 0x5600c61be940 2024-04-27T00:13:15.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f794ac72740 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f79401b1390 con 0x5600c61b97d0 2024-04-27T00:13:15.524 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f794ac72740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79401b05f0 con 0x5600c61b8de0 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f794ac72740 10 client.0 _ll_register_callbacks cb 0x7f79380093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f7944a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b97d0 0x5600c61c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f7944a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b97d0 0x5600c61c8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:60016/0 (socket says 172.21.15.40:60016) 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f7944a00640 1 -- 172.21.15.40:0/3274521661 learned_addr learned my addr 172.21.15.40:0/3274521661 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f7945e00640 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b8de0 0x5600c61c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f7945e00640 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f7944a00640 1 -- 172.21.15.40:0/3274521661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b8de0 msgr2=0x5600c61c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f7944a00640 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b8de0 0x5600c61c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f7944a00640 1 -- 172.21.15.40:0/3274521661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 msgr2=0x5600c61c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f7944a00640 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.521+0000 7f7944a00640 1 -- 172.21.15.40:0/3274521661 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600c60d7cb0 con 0x5600c61b97d0 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.522+0000 7f7944a00640 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b97d0 0x5600c61c8830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f792c000c00 tx=0x7f792c02aa80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4b8be0326a6329d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.522+0000 7f7926a00640 10 client.0 ms_handle_connect on v2:172.21.15.80:3300/0 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.522+0000 7f7926a00640 1 -- 172.21.15.40:0/3274521661 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c035070 con 0x5600c61b97d0 2024-04-27T00:13:15.525 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.522+0000 7f794ac72740 1 -- 172.21.15.40:0/3274521661 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7938003db0 con 0x5600c61b97d0 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.522+0000 7f7926a00640 1 -- 172.21.15.40:0/3274521661 <== mon.2 v2:172.21.15.80:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f792c0035f0 con 0x5600c61b97d0 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.522+0000 7f7926a00640 1 -- 172.21.15.40:0/3274521661 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f792c005d50 con 0x5600c61b97d0 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.522+0000 7f794ac72740 10 client.26297 fetch_fsmap learned FSMap version 24 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.522+0000 7f794ac72740 1 -- 172.21.15.40:0/3274521661 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5600c62355c0 con 0x5600c61b97d0 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.522+0000 7f7926a00640 1 -- 172.21.15.40:0/3274521661 <== mon.2 v2:172.21.15.80:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7938003db0 con 0x5600c61b97d0 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.523+0000 7f7926a00640 1 -- 172.21.15.40:0/3274521661 <== mon.2 v2:172.21.15.80:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f792c0046c0 con 0x5600c61b97d0 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.523+0000 7f794ac72740 10 client.26297 fetch_fsmap finished waiting for FSMap version 24 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.523+0000 7f794ac72740 10 client.26297 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.523+0000 7f794ac72740 20 client.26297 populate_metadata read hostname 'smithi040' 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.523+0000 7f794ac72740 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5600c6235eb0 0x5600c62382d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.523+0000 7f794ac72740 4 client.26297 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.523+0000 7f794ac72740 1 -- 172.21.15.40:0/3274521661 --> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5600c6124e00 con 0x5600c6235eb0 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.523+0000 7f7945e00640 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5600c6235eb0 0x5600c62382d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.524+0000 7f7945e00640 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5600c6235eb0 0x5600c62382d0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf7e91ca3f33e7b7 server_cookie=ac13fbb377965a16 in_seq=0 out_seq=0 2024-04-27T00:13:15.527 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.524+0000 7f7926a00640 10 client.26297 ms_handle_connect on v2:172.21.15.80:6838/3536638444 2024-04-27T00:13:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f7926a00640 1 -- 172.21.15.40:0/3274521661 <== mds.0 v2:172.21.15.80:6838/3536638444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5600c6124e00 con 0x5600c6235eb0 2024-04-27T00:13:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f7926a00640 10 client.26297 handle_command_reply: tid=0 2024-04-27T00:13:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f794ac72740 1 client.26297 shutdown 2024-04-27T00:13:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f794ac72740 1 -- 172.21.15.40:0/3274521661 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b97d0 msgr2=0x5600c61c8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f794ac72740 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b97d0 0x5600c61c8830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f792c000c00 tx=0x7f792c02aa80 comp rx=0 tx=0).stop 2024-04-27T00:13:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f794ac72740 1 -- 172.21.15.40:0/3274521661 shutdown_connections 2024-04-27T00:13:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f794ac72740 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.80:6838/3536638444,v1:172.21.15.80:6839/3536638444] conn(0x5600c6235eb0 0x5600c62382d0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f794ac72740 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5600c61be940 0x5600c61c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f794ac72740 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x5600c61b97d0 0x5600c61c8830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f794ac72740 1 --2- 172.21.15.40:0/3274521661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600c61b8de0 0x5600c61c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f7926a00640 0 client.26297 ms_handle_reset on v2:172.21.15.80:6838/3536638444 2024-04-27T00:13:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f794ac72740 1 -- 172.21.15.40:0/3274521661 >> 172.21.15.40:0/3274521661 conn(0x5600c61b4450 msgr2=0x5600c61b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f794ac72740 1 -- 172.21.15.40:0/3274521661 shutdown_connections 2024-04-27T00:13:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f794ac72740 1 -- 172.21.15.40:0/3274521661 wait complete. 2024-04-27T00:13:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.539+0000 7f794ac72740 20 client.26297 trim_cache size 0 max 16384 2024-04-27T00:13:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:{ 2024-04-27T00:13:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "message": "", 2024-04-27T00:13:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout: "return_code": 0 2024-04-27T00:13:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stdout:} 2024-04-27T00:13:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.540+0000 7f7946800640 1 -- 172.21.15.40:0/3010033943 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f79280a3c30 msgr2=0x7f79280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.540+0000 7f7946800640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f79280a3c30 0x7f79280a60f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x5600c61b9bd0 tx=0x7f79340d48c0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.540+0000 7f7946800640 1 -- 172.21.15.40:0/3010033943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940108de0 msgr2=0x7f794019f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.540+0000 7f7946800640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940108de0 0x7f794019f030 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f793400e9c0 tx=0x7f793400edc0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.540+0000 7f7946800640 1 -- 172.21.15.40:0/3010033943 shutdown_connections 2024-04-27T00:13:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.540+0000 7f7946800640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f79280a3c30 0x7f79280a60f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.540+0000 7f7946800640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f794010a990 0x7f79401a6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.540+0000 7f7946800640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7940109fd0 0x7f794019f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.540+0000 7f7946800640 1 --2- 172.21.15.40:0/3010033943 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940108de0 0x7f794019f030 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.540+0000 7f7946800640 1 -- 172.21.15.40:0/3010033943 >> 172.21.15.40:0/3010033943 conn(0x7f7940077dc0 msgr2=0x7f794010d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.541+0000 7f7946800640 1 -- 172.21.15.40:0/3010033943 shutdown_connections 2024-04-27T00:13:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi040.front.sepia.ceph.com.smithi040.stderr:2024-04-27T00:13:15.541+0000 7f7946800640 1 -- 172.21.15.40:0/3010033943 wait complete. 2024-04-27T00:13:17.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:16 smithi080 ceph-mon[32362]: pgmap v759: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:13:17.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:16 smithi103 ceph-mon[32952]: pgmap v759: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:13:17.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:16 smithi040 ceph-mon[28095]: pgmap v759: 129 pgs: 129 active+clean; 3.9 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:13:18.048 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:17 smithi080 ceph-mon[32362]: pgmap v760: 129 pgs: 129 active+clean; 4.0 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:13:18.096 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:17 smithi103 ceph-mon[32952]: pgmap v760: 129 pgs: 129 active+clean; 4.0 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:13:18.117 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:17 smithi040 ceph-mon[28095]: pgmap v760: 129 pgs: 129 active+clean; 4.0 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-27T00:13:20.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:20 smithi040 ceph-mon[28095]: pgmap v761: 129 pgs: 129 active+clean; 4.0 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:13:20.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:20 smithi080 ceph-mon[32362]: pgmap v761: 129 pgs: 129 active+clean; 4.0 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:13:20.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:20 smithi103 ceph-mon[32952]: pgmap v761: 129 pgs: 129 active+clean; 4.0 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-27T00:13:22.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:22 smithi040 ceph-mon[28095]: pgmap v762: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:13:22.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:22 smithi080 ceph-mon[32362]: pgmap v762: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:13:22.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:22 smithi103 ceph-mon[32952]: pgmap v762: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 15 op/s 2024-04-27T00:13:24.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:24 smithi040 ceph-mon[28095]: pgmap v763: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:13:24.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:24 smithi080 ceph-mon[32362]: pgmap v763: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:13:24.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:24 smithi103 ceph-mon[32952]: pgmap v763: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-04-27T00:13:25.594 INFO:tasks.workunit.client.0.smithi040.stdout:561/600: open fd = 3 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:562/600: open fd = 3 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:563/600: open fd = 3 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:564/600: open fd = 3 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.595 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:565/600: open fd = 3 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:566/600: open fd = 3 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:567/600: open fd = 3 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.596 INFO:tasks.workunit.client.0.smithi040.stdout:568/600: open fd = 3 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:569/600: open fd = 3 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:570/600: open fd = 3 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:571/600: open fd = 3 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.597 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:572/600: open fd = 3 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:573/600: open fd = 3 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:574/600: open fd = 3 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:575/600: open fd = 3 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.598 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:576/600: open fd = 3 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:577/600: open fd = 3 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:578/600: open fd = 3 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:579/600: open fd = 3 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.599 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:580/600: open fd = 3 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:581/600: open fd = 3 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:582/600: open fd = 3 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:583/600: open fd = 3 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.600 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:584/600: open fd = 3 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:585/600: open fd = 3 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:586/600: open fd = 3 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:587/600: open fd = 3 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.601 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:588/600: open fd = 3 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:589/600: open fd = 3 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:590/600: open fd = 3 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:591/600: open fd = 3 2024-04-27T00:13:25.602 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:592/600: open fd = 3 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:593/600: open fd = 3 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:594/600: open fd = 3 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:595/600: open fd = 3 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.603 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:596/600: open fd = 3 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:597/600: open fd = 3 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:598/600: open fd = 3 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:599/600: open fd = 3 2024-04-27T00:13:25.604 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.605 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.605 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.605 INFO:tasks.workunit.client.0.smithi040.stdout:600/600: open fd = 3 2024-04-27T00:13:25.605 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.605 INFO:tasks.workunit.client.0.smithi040.stdout:write ret = 32 2024-04-27T00:13:25.605 INFO:tasks.workunit.client.0.smithi040.stdout:pread ret = 64 2024-04-27T00:13:25.605 INFO:tasks.workunit.client.0.smithi040.stderr:+ echo OK 2024-04-27T00:13:25.606 INFO:tasks.workunit.client.0.smithi040.stdout:OK 2024-04-27T00:13:25.607 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-27T00:13:25.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-27T00:13:25.763 INFO:tasks.workunit:Stopping ['fs/test_o_trunc.sh'] on client.0... 2024-04-27T00:13:25.763 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-27T00:13:26.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:26 smithi080 ceph-mon[32362]: pgmap v764: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:13:26.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:26 smithi103 ceph-mon[32952]: pgmap v764: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:13:26.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:26 smithi040 ceph-mon[28095]: pgmap v764: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-04-27T00:13:26.867 DEBUG:teuthology.parallel:result is None 2024-04-27T00:13:26.868 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-27T00:13:26.961 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-27T00:13:26.961 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-27T00:13:26.975 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi040.front.sepia.ceph.com 2024-04-27T00:13:26.993 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-27T00:13:27.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T00:13:27.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.254+0000 7fc66f200640 1 -- 172.21.15.40:0/436566184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6681049b0 msgr2=0x7fc668104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:27.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.254+0000 7fc66f200640 1 --2- 172.21.15.40:0/436566184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6681049b0 0x7fc668104dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc664009ca0 tx=0x7fc66402a8c0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.254+0000 7fc66f200640 1 -- 172.21.15.40:0/436566184 shutdown_connections 2024-04-27T00:13:27.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.254+0000 7fc66f200640 1 --2- 172.21.15.40:0/436566184 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc668106560 0x7fc66810cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.254+0000 7fc66f200640 1 --2- 172.21.15.40:0/436566184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc668105ba0 0x7fc668106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.254+0000 7fc66f200640 1 --2- 172.21.15.40:0/436566184 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6681049b0 0x7fc668104dd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.254+0000 7fc66f200640 1 -- 172.21.15.40:0/436566184 >> 172.21.15.40:0/436566184 conn(0x7fc6680fff40 msgr2=0x7fc668102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:27.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.254+0000 7fc66f200640 1 -- 172.21.15.40:0/436566184 shutdown_connections 2024-04-27T00:13:27.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.254+0000 7fc66f200640 1 -- 172.21.15.40:0/436566184 wait complete. 2024-04-27T00:13:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.255+0000 7fc66f200640 1 Processor -- start 2024-04-27T00:13:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.255+0000 7fc66f200640 1 -- start start 2024-04-27T00:13:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6681049b0 0x7fc66819ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc668105ba0 0x7fc66819b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66f200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc668106560 0x7fc6681a2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc6681104e0 con 0x7fc6681049b0 2024-04-27T00:13:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66f200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fc668110360 con 0x7fc668106560 2024-04-27T00:13:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc668110660 con 0x7fc668105ba0 2024-04-27T00:13:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6681049b0 0x7fc66819ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6681049b0 0x7fc66819ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41052/0 (socket says 172.21.15.40:41052) 2024-04-27T00:13:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66de00640 1 -- 172.21.15.40:0/3836901160 learned_addr learned my addr 172.21.15.40:0/3836901160 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66e800640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc668106560 0x7fc6681a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66d400640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc668105ba0 0x7fc66819b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66de00640 1 -- 172.21.15.40:0/3836901160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc668105ba0 msgr2=0x7fc66819b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66de00640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc668105ba0 0x7fc66819b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66de00640 1 -- 172.21.15.40:0/3836901160 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc668106560 msgr2=0x7fc6681a2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66de00640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc668106560 0x7fc6681a2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.256+0000 7fc66de00640 1 -- 172.21.15.40:0/3836901160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6681a2770 con 0x7fc6681049b0 2024-04-27T00:13:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.257+0000 7fc66de00640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6681049b0 0x7fc66819ab50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc664009430 tx=0x7fc6640316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.257+0000 7fc65ea00640 1 -- 172.21.15.40:0/3836901160 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664003b70 con 0x7fc6681049b0 2024-04-27T00:13:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.257+0000 7fc66f200640 1 -- 172.21.15.40:0/3836901160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6681a2a00 con 0x7fc6681049b0 2024-04-27T00:13:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.257+0000 7fc65ea00640 1 -- 172.21.15.40:0/3836901160 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc664003d10 con 0x7fc6681049b0 2024-04-27T00:13:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.257+0000 7fc65ea00640 1 -- 172.21.15.40:0/3836901160 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66402fb60 con 0x7fc6681049b0 2024-04-27T00:13:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.257+0000 7fc66f200640 1 -- 172.21.15.40:0/3836901160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6681a2fa0 con 0x7fc6681049b0 2024-04-27T00:13:27.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.258+0000 7fc66f200640 1 -- 172.21.15.40:0/3836901160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc66810bcd0 con 0x7fc6681049b0 2024-04-27T00:13:27.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.260+0000 7fc65ea00640 1 -- 172.21.15.40:0/3836901160 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc66402fd00 con 0x7fc6681049b0 2024-04-27T00:13:27.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.262+0000 7fc65ea00640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6540a3c40 0x7fc6540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:27.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.262+0000 7fc65ea00640 1 -- 172.21.15.40:0/3836901160 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fc6640f1c70 con 0x7fc6681049b0 2024-04-27T00:13:27.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.262+0000 7fc66d400640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6540a3c40 0x7fc6540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:27.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.263+0000 7fc66d400640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6540a3c40 0x7fc6540a6100 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc66819c060 tx=0x7fc658005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:27.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.268+0000 7fc65ea00640 1 -- 172.21.15.40:0/3836901160 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc6640be200 con 0x7fc6681049b0 2024-04-27T00:13:27.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.476+0000 7fc66f200640 1 -- 172.21.15.40:0/3836901160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc66802dd10 con 0x7fc6681049b0 2024-04-27T00:13:27.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.480+0000 7fc65ea00640 1 -- 172.21.15.40:0/3836901160 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24727 (secure 0 0 0) 0x7fc6640bdb00 con 0x7fc6681049b0 2024-04-27T00:13:27.481 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:13:27.481 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":94,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-27T00:01:46.569457+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9},{"from":5,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T00:13:27.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.483+0000 7fc66f200640 1 -- 172.21.15.40:0/3836901160 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6540a3c40 msgr2=0x7fc6540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:27.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.483+0000 7fc66f200640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6540a3c40 0x7fc6540a6100 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc66819c060 tx=0x7fc658005f90 comp rx=0 tx=0).stop 2024-04-27T00:13:27.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.483+0000 7fc66f200640 1 -- 172.21.15.40:0/3836901160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6681049b0 msgr2=0x7fc66819ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:27.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.483+0000 7fc66f200640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6681049b0 0x7fc66819ab50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc664009430 tx=0x7fc6640316e0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.483+0000 7fc66f200640 1 -- 172.21.15.40:0/3836901160 shutdown_connections 2024-04-27T00:13:27.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.483+0000 7fc66f200640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc668106560 0x7fc6681a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.483+0000 7fc66f200640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc6540a3c40 0x7fc6540a6100 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.483+0000 7fc66f200640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc668105ba0 0x7fc66819b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.483+0000 7fc66f200640 1 --2- 172.21.15.40:0/3836901160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6681049b0 0x7fc66819ab50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.483+0000 7fc66f200640 1 -- 172.21.15.40:0/3836901160 >> 172.21.15.40:0/3836901160 conn(0x7fc6680fff40 msgr2=0x7fc668108ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:27.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.483+0000 7fc66f200640 1 -- 172.21.15.40:0/3836901160 shutdown_connections 2024-04-27T00:13:27.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.483+0000 7fc66f200640 1 -- 172.21.15.40:0/3836901160 wait complete. 2024-04-27T00:13:27.505 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:53:32.708818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:58:48.506601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:58:49.540141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:58:53.179820+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-27T00:13:27.505 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-27T00:13:27.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.723+0000 7fdb8f200640 1 -- 172.21.15.40:0/280699771 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb9010a700 msgr2=0x7fdb9010cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:27.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.723+0000 7fdb8f200640 1 --2- 172.21.15.40:0/280699771 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb9010a700 0x7fdb9010cb90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009ca0 tx=0x7fdb8802a8c0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.723+0000 7fdb8f200640 1 -- 172.21.15.40:0/280699771 shutdown_connections 2024-04-27T00:13:27.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.723+0000 7fdb8f200640 1 --2- 172.21.15.40:0/280699771 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb9010a700 0x7fdb9010cb90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.723+0000 7fdb8f200640 1 --2- 172.21.15.40:0/280699771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb90075390 0x7fdb90075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.723+0000 7fdb8f200640 1 --2- 172.21.15.40:0/280699771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb90076970 0x7fdb90074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.723+0000 7fdb8f200640 1 -- 172.21.15.40:0/280699771 >> 172.21.15.40:0/280699771 conn(0x7fdb900fddd0 msgr2=0x7fdb901001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:27.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.723+0000 7fdb8f200640 1 -- 172.21.15.40:0/280699771 shutdown_connections 2024-04-27T00:13:27.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.723+0000 7fdb8f200640 1 -- 172.21.15.40:0/280699771 wait complete. 2024-04-27T00:13:27.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.725+0000 7fdb8f200640 1 Processor -- start 2024-04-27T00:13:27.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.725+0000 7fdb8f200640 1 -- start start 2024-04-27T00:13:27.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.725+0000 7fdb8f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb90075390 0x7fdb9019ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:27.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.725+0000 7fdb8f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb90076970 0x7fdb9019b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:27.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.725+0000 7fdb8f200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb9010a700 0x7fdb901a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:27.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.725+0000 7fdb8f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90110550 con 0x7fdb90076970 2024-04-27T00:13:27.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.725+0000 7fdb8f200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901103d0 con 0x7fdb9010a700 2024-04-27T00:13:27.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.725+0000 7fdb8f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901106d0 con 0x7fdb90075390 2024-04-27T00:13:27.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb90076970 0x7fdb9019b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:27.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb90075390 0x7fdb9019ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:27.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb90076970 0x7fdb9019b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41076/0 (socket says 172.21.15.40:41076) 2024-04-27T00:13:27.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb90075390 0x7fdb9019ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:35516/0 (socket says 172.21.15.40:35516) 2024-04-27T00:13:27.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8d400640 1 -- 172.21.15.40:0/800786167 learned_addr learned my addr 172.21.15.40:0/800786167 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:27.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8e800640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb9010a700 0x7fdb901a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:27.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8d400640 1 -- 172.21.15.40:0/800786167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb90075390 msgr2=0x7fdb9019ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:27.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8d400640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb90075390 0x7fdb9019ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8d400640 1 -- 172.21.15.40:0/800786167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb9010a700 msgr2=0x7fdb901a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:27.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8d400640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb9010a700 0x7fdb901a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8d400640 1 -- 172.21.15.40:0/800786167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb901a2840 con 0x7fdb90076970 2024-04-27T00:13:27.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8de00640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb90075390 0x7fdb9019ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:13:27.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.726+0000 7fdb8e800640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb9010a700 0x7fdb901a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:13:27.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.727+0000 7fdb8d400640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb90076970 0x7fdb9019b170 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00e9f0 tx=0x7fdb7c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:27.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.727+0000 7fdb86a00640 1 -- 172.21.15.40:0/800786167 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c00cd80 con 0x7fdb90076970 2024-04-27T00:13:27.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.727+0000 7fdb86a00640 1 -- 172.21.15.40:0/800786167 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb7c0054d0 con 0x7fdb90076970 2024-04-27T00:13:27.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.727+0000 7fdb86a00640 1 -- 172.21.15.40:0/800786167 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c010690 con 0x7fdb90076970 2024-04-27T00:13:27.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.727+0000 7fdb8f200640 1 -- 172.21.15.40:0/800786167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb901a2b30 con 0x7fdb90076970 2024-04-27T00:13:27.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.727+0000 7fdb8f200640 1 -- 172.21.15.40:0/800786167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb901a3080 con 0x7fdb90076970 2024-04-27T00:13:27.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.728+0000 7fdb8f200640 1 -- 172.21.15.40:0/800786167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb90109430 con 0x7fdb90076970 2024-04-27T00:13:27.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.731+0000 7fdb86a00640 1 -- 172.21.15.40:0/800786167 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdb7c004e90 con 0x7fdb90076970 2024-04-27T00:13:27.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.732+0000 7fdb86a00640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdb740a3cc0 0x7fdb740a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:27.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.732+0000 7fdb86a00640 1 -- 172.21.15.40:0/800786167 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fdb7c0d0db0 con 0x7fdb90076970 2024-04-27T00:13:27.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.732+0000 7fdb8de00640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdb740a3cc0 0x7fdb740a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:27.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.733+0000 7fdb8de00640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdb740a3cc0 0x7fdb740a6180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdb9019c100 tx=0x7fdb80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:27.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.736+0000 7fdb86a00640 1 -- 172.21.15.40:0/800786167 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdb7c0d5070 con 0x7fdb90076970 2024-04-27T00:13:27.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.946+0000 7fdb8f200640 1 -- 172.21.15.40:0/800786167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdb9002dd20 con 0x7fdb90076970 2024-04-27T00:13:27.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.947+0000 7fdb86a00640 1 -- 172.21.15.40:0/800786167 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fdb7c09d230 con 0x7fdb90076970 2024-04-27T00:13:27.947 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-27T00:13:27.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.949+0000 7fdb8f200640 1 -- 172.21.15.40:0/800786167 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdb740a3cc0 msgr2=0x7fdb740a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:27.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.949+0000 7fdb8f200640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdb740a3cc0 0x7fdb740a6180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdb9019c100 tx=0x7fdb80005f90 comp rx=0 tx=0).stop 2024-04-27T00:13:27.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.949+0000 7fdb8f200640 1 -- 172.21.15.40:0/800786167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb90076970 msgr2=0x7fdb9019b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:27.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.949+0000 7fdb8f200640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb90076970 0x7fdb9019b170 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00e9f0 tx=0x7fdb7c00edf0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.950+0000 7fdb8f200640 1 -- 172.21.15.40:0/800786167 shutdown_connections 2024-04-27T00:13:27.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.950+0000 7fdb8f200640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fdb9010a700 0x7fdb901a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.950+0000 7fdb8f200640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb90076970 0x7fdb9019b170 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.950+0000 7fdb8f200640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fdb740a3cc0 0x7fdb740a6180 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.950+0000 7fdb8f200640 1 --2- 172.21.15.40:0/800786167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb90075390 0x7fdb9019ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:27.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.950+0000 7fdb8f200640 1 -- 172.21.15.40:0/800786167 >> 172.21.15.40:0/800786167 conn(0x7fdb900fddd0 msgr2=0x7fdb90106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:27.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.950+0000 7fdb8f200640 1 -- 172.21.15.40:0/800786167 shutdown_connections 2024-04-27T00:13:27.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:27.950+0000 7fdb8f200640 1 -- 172.21.15.40:0/800786167 wait complete. 2024-04-27T00:13:27.971 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-27T00:13:28.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.181+0000 7fc4c3e00640 1 -- 172.21.15.40:0/4137270802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4c4078ed0 msgr2=0x7fc4c4079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:28.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.181+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/4137270802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4c4078ed0 0x7fc4c4079770 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc009be0 tx=0x7fc4bc02a6d0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.181+0000 7fc4c3e00640 1 -- 172.21.15.40:0/4137270802 shutdown_connections 2024-04-27T00:13:28.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.181+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/4137270802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4c4078ed0 0x7fc4c4079770 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.181+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/4137270802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4c4078530 0x7fc4c4078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.181+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/4137270802 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4c40772f0 0x7fc4c4077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.181+0000 7fc4c3e00640 1 -- 172.21.15.40:0/4137270802 >> 172.21.15.40:0/4137270802 conn(0x7fc4c40fffe0 msgr2=0x7fc4c4102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:28.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.181+0000 7fc4c3e00640 1 -- 172.21.15.40:0/4137270802 shutdown_connections 2024-04-27T00:13:28.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.181+0000 7fc4c3e00640 1 -- 172.21.15.40:0/4137270802 wait complete. 2024-04-27T00:13:28.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.182+0000 7fc4c3e00640 1 Processor -- start 2024-04-27T00:13:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.182+0000 7fc4c3e00640 1 -- start start 2024-04-27T00:13:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.182+0000 7fc4c3e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4c40772f0 0x7fc4c419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.182+0000 7fc4c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4c4078530 0x7fc4c419f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.182+0000 7fc4c3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4c4078ed0 0x7fc4c41a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.182+0000 7fc4c3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c41149b0 con 0x7fc4c4078530 2024-04-27T00:13:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.182+0000 7fc4c3e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c4114830 con 0x7fc4c40772f0 2024-04-27T00:13:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.182+0000 7fc4c3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c4114b30 con 0x7fc4c4078ed0 2024-04-27T00:13:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.183+0000 7fc4c2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4c4078530 0x7fc4c419f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.183+0000 7fc4c2a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4c40772f0 0x7fc4c419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.183+0000 7fc4c2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4c4078530 0x7fc4c419f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41094/0 (socket says 172.21.15.40:41094) 2024-04-27T00:13:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.183+0000 7fc4c2000640 1 -- 172.21.15.40:0/662436319 learned_addr learned my addr 172.21.15.40:0/662436319 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.183+0000 7fc4c2a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4c40772f0 0x7fc4c419efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:42730/0 (socket says 172.21.15.40:42730) 2024-04-27T00:13:28.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.183+0000 7fc4c3400640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4c4078ed0 0x7fc4c41a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:28.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.183+0000 7fc4c2000640 1 -- 172.21.15.40:0/662436319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4c4078ed0 msgr2=0x7fc4c41a6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:28.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.183+0000 7fc4c2000640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4c4078ed0 0x7fc4c41a6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.183+0000 7fc4c2000640 1 -- 172.21.15.40:0/662436319 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4c40772f0 msgr2=0x7fc4c419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:28.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.183+0000 7fc4c2000640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4c40772f0 0x7fc4c419efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.183+0000 7fc4c2000640 1 -- 172.21.15.40:0/662436319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c41a6c00 con 0x7fc4c4078530 2024-04-27T00:13:28.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.184+0000 7fc4c2000640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4c4078530 0x7fc4c419f520 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000d950 tx=0x7fc4b000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:28.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.184+0000 7fc4bb400640 1 -- 172.21.15.40:0/662436319 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0017030 con 0x7fc4c4078530 2024-04-27T00:13:28.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.184+0000 7fc4bb400640 1 -- 172.21.15.40:0/662436319 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4b0004020 con 0x7fc4c4078530 2024-04-27T00:13:28.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.184+0000 7fc4bb400640 1 -- 172.21.15.40:0/662436319 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0005560 con 0x7fc4c4078530 2024-04-27T00:13:28.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.184+0000 7fc4c3e00640 1 -- 172.21.15.40:0/662436319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4c41a6ef0 con 0x7fc4c4078530 2024-04-27T00:13:28.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.184+0000 7fc4c3e00640 1 -- 172.21.15.40:0/662436319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4c41a7400 con 0x7fc4c4078530 2024-04-27T00:13:28.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.185+0000 7fc4c3e00640 1 -- 172.21.15.40:0/662436319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4c41113e0 con 0x7fc4c4078530 2024-04-27T00:13:28.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.188+0000 7fc4bb400640 1 -- 172.21.15.40:0/662436319 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc4b0009a30 con 0x7fc4c4078530 2024-04-27T00:13:28.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.189+0000 7fc4bb400640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4a80a3cc0 0x7fc4a80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:28.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.189+0000 7fc4bb400640 1 -- 172.21.15.40:0/662436319 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fc4b00d01e0 con 0x7fc4c4078530 2024-04-27T00:13:28.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.189+0000 7fc4c2a00640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4a80a3cc0 0x7fc4a80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:28.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.190+0000 7fc4c2a00640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4a80a3cc0 0x7fc4a80a6180 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005fd0 tx=0x7fc4b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:28.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.193+0000 7fc4bb400640 1 -- 172.21.15.40:0/662436319 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc4b009c660 con 0x7fc4c4078530 2024-04-27T00:13:28.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:28 smithi080 ceph-mon[32362]: pgmap v765: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:13:28.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:28 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3836901160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:13:28.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:28 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/800786167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:13:28.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:28 smithi103 ceph-mon[32952]: pgmap v765: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:13:28.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:28 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3836901160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:13:28.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:28 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/800786167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:13:28.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.399+0000 7fc4c3e00640 1 -- 172.21.15.40:0/662436319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc4c402dd20 con 0x7fc4c4078530 2024-04-27T00:13:28.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.400+0000 7fc4bb400640 1 -- 172.21.15.40:0/662436319 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fc4b009bf60 con 0x7fc4c4078530 2024-04-27T00:13:28.400 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-27T00:13:28.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.402+0000 7fc4c3e00640 1 -- 172.21.15.40:0/662436319 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4a80a3cc0 msgr2=0x7fc4a80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:28.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.402+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4a80a3cc0 0x7fc4a80a6180 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005fd0 tx=0x7fc4b4009260 comp rx=0 tx=0).stop 2024-04-27T00:13:28.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.402+0000 7fc4c3e00640 1 -- 172.21.15.40:0/662436319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4c4078530 msgr2=0x7fc4c419f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:28.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.402+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4c4078530 0x7fc4c419f520 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc4b000d950 tx=0x7fc4b000dd50 comp rx=0 tx=0).stop 2024-04-27T00:13:28.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.403+0000 7fc4c3e00640 1 -- 172.21.15.40:0/662436319 shutdown_connections 2024-04-27T00:13:28.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.403+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fc4a80a3cc0 0x7fc4a80a6180 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.403+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4c4078ed0 0x7fc4c41a6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.403+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4c4078530 0x7fc4c419f520 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.403+0000 7fc4c3e00640 1 --2- 172.21.15.40:0/662436319 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fc4c40772f0 0x7fc4c419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.403+0000 7fc4c3e00640 1 -- 172.21.15.40:0/662436319 >> 172.21.15.40:0/662436319 conn(0x7fc4c40fffe0 msgr2=0x7fc4c4101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:28.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.403+0000 7fc4c3e00640 1 -- 172.21.15.40:0/662436319 shutdown_connections 2024-04-27T00:13:28.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.403+0000 7fc4c3e00640 1 -- 172.21.15.40:0/662436319 wait complete. 2024-04-27T00:13:28.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:28 smithi040 ceph-mon[28095]: pgmap v765: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 16 op/s 2024-04-27T00:13:28.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:28 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3836901160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:13:28.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:28 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/800786167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:13:28.423 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-27T00:13:28.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.644+0000 7f2fdbe00640 1 -- 172.21.15.40:0/4010053074 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc102340 msgr2=0x7f2fdc1027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:28.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.644+0000 7f2fdbe00640 1 --2- 172.21.15.40:0/4010053074 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc102340 0x7f2fdc1027c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4009900 tx=0x7f2fd402a480 comp rx=0 tx=0).stop 2024-04-27T00:13:28.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.644+0000 7f2fdbe00640 1 -- 172.21.15.40:0/4010053074 shutdown_connections 2024-04-27T00:13:28.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.644+0000 7f2fdbe00640 1 --2- 172.21.15.40:0/4010053074 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc102340 0x7f2fdc1027c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.644+0000 7f2fdbe00640 1 --2- 172.21.15.40:0/4010053074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fdc1019a0 0x7f2fdc101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.644+0000 7f2fdbe00640 1 --2- 172.21.15.40:0/4010053074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc101040 0x7f2fdc101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.644+0000 7f2fdbe00640 1 -- 172.21.15.40:0/4010053074 >> 172.21.15.40:0/4010053074 conn(0x7f2fdc0fcc00 msgr2=0x7f2fdc0ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:28.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.644+0000 7f2fdbe00640 1 -- 172.21.15.40:0/4010053074 shutdown_connections 2024-04-27T00:13:28.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.644+0000 7f2fdbe00640 1 -- 172.21.15.40:0/4010053074 wait complete. 2024-04-27T00:13:28.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.645+0000 7f2fdbe00640 1 Processor -- start 2024-04-27T00:13:28.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.645+0000 7f2fdbe00640 1 -- start start 2024-04-27T00:13:28.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.646+0000 7f2fdbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc101040 0x7f2fdc19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:28.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.646+0000 7f2fdbe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fdc1019a0 0x7f2fdc19b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:28.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.646+0000 7f2fdbe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc102340 0x7f2fdc1a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:28.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.646+0000 7f2fdbe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2fdc1104d0 con 0x7f2fdc101040 2024-04-27T00:13:28.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.646+0000 7f2fdbe00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f2fdc110350 con 0x7f2fdc102340 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.646+0000 7f2fdbe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2fdc110650 con 0x7f2fdc1019a0 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.646+0000 7f2fdaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc101040 0x7f2fdc19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.646+0000 7f2fdb400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc102340 0x7f2fdc1a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.646+0000 7f2fdaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc101040 0x7f2fdc19ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41116/0 (socket says 172.21.15.40:41116) 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.646+0000 7f2fdaa00640 1 -- 172.21.15.40:0/1427805167 learned_addr learned my addr 172.21.15.40:0/1427805167 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.646+0000 7f2fda000640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fdc1019a0 0x7f2fdc19b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.647+0000 7f2fdb400640 1 -- 172.21.15.40:0/1427805167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fdc1019a0 msgr2=0x7f2fdc19b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.647+0000 7f2fdb400640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fdc1019a0 0x7f2fdc19b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.647+0000 7f2fdb400640 1 -- 172.21.15.40:0/1427805167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc101040 msgr2=0x7f2fdc19ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.647+0000 7f2fdb400640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc101040 0x7f2fdc19ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.647+0000 7f2fdb400640 1 -- 172.21.15.40:0/1427805167 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fdc1a2720 con 0x7f2fdc102340 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.647+0000 7f2fdaa00640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc101040 0x7f2fdc19ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:13:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.647+0000 7f2fdb400640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc102340 0x7f2fdc1a20b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2fd402a9a0 tx=0x7f2fd4003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:28.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.647+0000 7f2fd3400640 1 -- 172.21.15.40:0/1427805167 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd403f070 con 0x7f2fdc102340 2024-04-27T00:13:28.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.647+0000 7f2fd3400640 1 -- 172.21.15.40:0/1427805167 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fd4031c50 con 0x7f2fdc102340 2024-04-27T00:13:28.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.647+0000 7f2fdbe00640 1 -- 172.21.15.40:0/1427805167 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fdc1a29b0 con 0x7f2fdc102340 2024-04-27T00:13:28.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.647+0000 7f2fd3400640 1 -- 172.21.15.40:0/1427805167 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4035030 con 0x7f2fdc102340 2024-04-27T00:13:28.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.647+0000 7f2fdbe00640 1 -- 172.21.15.40:0/1427805167 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fdc1a2ec0 con 0x7f2fdc102340 2024-04-27T00:13:28.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.649+0000 7f2fdbe00640 1 -- 172.21.15.40:0/1427805167 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fdc069670 con 0x7f2fdc102340 2024-04-27T00:13:28.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.651+0000 7f2fd3400640 1 -- 172.21.15.40:0/1427805167 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2fd4042070 con 0x7f2fdc102340 2024-04-27T00:13:28.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.652+0000 7f2fd3400640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2fc00a3cc0 0x7f2fc00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:28.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.652+0000 7f2fd3400640 1 -- 172.21.15.40:0/1427805167 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f2fd40f1230 con 0x7f2fdc102340 2024-04-27T00:13:28.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.653+0000 7f2fdaa00640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2fc00a3cc0 0x7f2fc00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:28.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.653+0000 7f2fdaa00640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2fc00a3cc0 0x7f2fc00a6180 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00f7d0 tx=0x7f2fcc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:28.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.657+0000 7f2fd3400640 1 -- 172.21.15.40:0/1427805167 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2fd40bd060 con 0x7f2fdc102340 2024-04-27T00:13:28.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.864+0000 7f2fdbe00640 1 -- 172.21.15.40:0/1427805167 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2fdc02db00 con 0x7f2fdc102340 2024-04-27T00:13:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.864+0000 7f2fd3400640 1 -- 172.21.15.40:0/1427805167 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f2fd40c1360 con 0x7f2fdc102340 2024-04-27T00:13:28.865 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-27T00:13:28.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.867+0000 7f2fdbe00640 1 -- 172.21.15.40:0/1427805167 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2fc00a3cc0 msgr2=0x7f2fc00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:28.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.867+0000 7f2fdbe00640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2fc00a3cc0 0x7f2fc00a6180 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00f7d0 tx=0x7f2fcc009500 comp rx=0 tx=0).stop 2024-04-27T00:13:28.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.867+0000 7f2fdbe00640 1 -- 172.21.15.40:0/1427805167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc102340 msgr2=0x7f2fdc1a20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:28.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.867+0000 7f2fdbe00640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc102340 0x7f2fdc1a20b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2fd402a9a0 tx=0x7f2fd4003b90 comp rx=0 tx=0).stop 2024-04-27T00:13:28.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.867+0000 7f2fdbe00640 1 -- 172.21.15.40:0/1427805167 shutdown_connections 2024-04-27T00:13:28.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.867+0000 7f2fdbe00640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f2fdc102340 0x7f2fdc1a20b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.867+0000 7f2fdbe00640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f2fc00a3cc0 0x7f2fc00a6180 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.867+0000 7f2fdbe00640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fdc1019a0 0x7f2fdc19b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.867+0000 7f2fdbe00640 1 --2- 172.21.15.40:0/1427805167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fdc101040 0x7f2fdc19ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:28.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.867+0000 7f2fdbe00640 1 -- 172.21.15.40:0/1427805167 >> 172.21.15.40:0/1427805167 conn(0x7f2fdc0fcc00 msgr2=0x7f2fdc0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:28.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.868+0000 7f2fdbe00640 1 -- 172.21.15.40:0/1427805167 shutdown_connections 2024-04-27T00:13:28.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:28.868+0000 7f2fdbe00640 1 -- 172.21.15.40:0/1427805167 wait complete. 2024-04-27T00:13:28.891 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-27T00:13:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.113+0000 7f185de00640 1 -- 172.21.15.40:0/3988649193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1858104a70 msgr2=0x7f1858104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.113+0000 7f185de00640 1 --2- 172.21.15.40:0/3988649193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1858104a70 0x7f1858104e90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1848009860 tx=0x7f184802a340 comp rx=0 tx=0).stop 2024-04-27T00:13:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.113+0000 7f185de00640 1 -- 172.21.15.40:0/3988649193 shutdown_connections 2024-04-27T00:13:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.113+0000 7f185de00640 1 --2- 172.21.15.40:0/3988649193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1858106620 0x7f185810ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.113+0000 7f185de00640 1 --2- 172.21.15.40:0/3988649193 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1858105c60 0x7f18581060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.113+0000 7f185de00640 1 --2- 172.21.15.40:0/3988649193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1858104a70 0x7f1858104e90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.113+0000 7f185de00640 1 -- 172.21.15.40:0/3988649193 >> 172.21.15.40:0/3988649193 conn(0x7f18580fffe0 msgr2=0x7f1858102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.113+0000 7f185de00640 1 -- 172.21.15.40:0/3988649193 shutdown_connections 2024-04-27T00:13:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.113+0000 7f185de00640 1 -- 172.21.15.40:0/3988649193 wait complete. 2024-04-27T00:13:29.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.114+0000 7f185de00640 1 Processor -- start 2024-04-27T00:13:29.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.114+0000 7f185de00640 1 -- start start 2024-04-27T00:13:29.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.115+0000 7f185de00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1858104a70 0x7f185819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:29.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.115+0000 7f185de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1858105c60 0x7f185819b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:29.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.115+0000 7f185de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1858106620 0x7f18581a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:29.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.115+0000 7f185de00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1858110570 con 0x7f1858106620 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.115+0000 7f185de00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f18581103f0 con 0x7f1858104a70 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.115+0000 7f185de00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18581106f0 con 0x7f1858105c60 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.115+0000 7f185d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1858106620 0x7f18581a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.115+0000 7f1857e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1858105c60 0x7f185819b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.115+0000 7f185ca00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1858104a70 0x7f185819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.115+0000 7f185ca00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1858104a70 0x7f185819ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:42766/0 (socket says 172.21.15.40:42766) 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.115+0000 7f185ca00640 1 -- 172.21.15.40:0/2814249440 learned_addr learned my addr 172.21.15.40:0/2814249440 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.115+0000 7f1857e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1858105c60 0x7f185819b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:35570/0 (socket says 172.21.15.40:35570) 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.116+0000 7f185ca00640 1 -- 172.21.15.40:0/2814249440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1858105c60 msgr2=0x7f185819b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.116+0000 7f185ca00640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1858105c60 0x7f185819b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.116+0000 7f185ca00640 1 -- 172.21.15.40:0/2814249440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1858106620 msgr2=0x7f18581a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.116+0000 7f185ca00640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1858106620 0x7f18581a2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.116+0000 7f185ca00640 1 -- 172.21.15.40:0/2814249440 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18581a28d0 con 0x7f1858104a70 2024-04-27T00:13:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.116+0000 7f1857e00640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1858105c60 0x7f185819b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:13:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.116+0000 7f185ca00640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1858104a70 0x7f185819ac90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f18480057c0 tx=0x7f1848032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.116+0000 7f1855600640 1 -- 172.21.15.40:0/2814249440 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848037040 con 0x7f1858104a70 2024-04-27T00:13:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.116+0000 7f1855600640 1 -- 172.21.15.40:0/2814249440 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1848030bd0 con 0x7f1858104a70 2024-04-27T00:13:29.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.116+0000 7f185de00640 1 -- 172.21.15.40:0/2814249440 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18581a2b60 con 0x7f1858104a70 2024-04-27T00:13:29.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.116+0000 7f1855600640 1 -- 172.21.15.40:0/2814249440 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848005280 con 0x7f1858104a70 2024-04-27T00:13:29.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.117+0000 7f185de00640 1 -- 172.21.15.40:0/2814249440 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18581a3040 con 0x7f1858104a70 2024-04-27T00:13:29.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.118+0000 7f185de00640 1 -- 172.21.15.40:0/2814249440 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f185810be60 con 0x7f1858104a70 2024-04-27T00:13:29.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.120+0000 7f1855600640 1 -- 172.21.15.40:0/2814249440 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1848043020 con 0x7f1858104a70 2024-04-27T00:13:29.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.121+0000 7f1855600640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f183c0a3cc0 0x7f183c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:29.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.121+0000 7f1855600640 1 -- 172.21.15.40:0/2814249440 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f18480efd80 con 0x7f1858104a70 2024-04-27T00:13:29.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.122+0000 7f1857e00640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f183c0a3cc0 0x7f183c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:29.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.122+0000 7f1857e00640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f183c0a3cc0 0x7f183c0a6180 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f18400096c0 tx=0x7f1840009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:29.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.126+0000 7f1855600640 1 -- 172.21.15.40:0/2814249440 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f18480bc310 con 0x7f1858104a70 2024-04-27T00:13:29.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:29 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/662436319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:13:29.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:29 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1427805167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:13:29.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:29 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/662436319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:13:29.299 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:29 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1427805167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:13:29.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.338+0000 7f185de00640 1 -- 172.21.15.40:0/2814249440 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f185802dd20 con 0x7f1858104a70 2024-04-27T00:13:29.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.339+0000 7f1855600640 1 -- 172.21.15.40:0/2814249440 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f18480bbc10 con 0x7f1858104a70 2024-04-27T00:13:29.340 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-27T00:13:29.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.342+0000 7f185de00640 1 -- 172.21.15.40:0/2814249440 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f183c0a3cc0 msgr2=0x7f183c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:29.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.342+0000 7f185de00640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f183c0a3cc0 0x7f183c0a6180 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f18400096c0 tx=0x7f1840009260 comp rx=0 tx=0).stop 2024-04-27T00:13:29.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.342+0000 7f185de00640 1 -- 172.21.15.40:0/2814249440 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1858104a70 msgr2=0x7f185819ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:29.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.342+0000 7f185de00640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1858104a70 0x7f185819ac90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f18480057c0 tx=0x7f1848032da0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.342+0000 7f185de00640 1 -- 172.21.15.40:0/2814249440 shutdown_connections 2024-04-27T00:13:29.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.343+0000 7f185de00640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1858106620 0x7f18581a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.343+0000 7f185de00640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f183c0a3cc0 0x7f183c0a6180 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.343+0000 7f185de00640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1858105c60 0x7f185819b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.343+0000 7f185de00640 1 --2- 172.21.15.40:0/2814249440 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1858104a70 0x7f185819ac90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.343+0000 7f185de00640 1 -- 172.21.15.40:0/2814249440 >> 172.21.15.40:0/2814249440 conn(0x7f18580fffe0 msgr2=0x7f1858109030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:29.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.343+0000 7f185de00640 1 -- 172.21.15.40:0/2814249440 shutdown_connections 2024-04-27T00:13:29.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.343+0000 7f185de00640 1 -- 172.21.15.40:0/2814249440 wait complete. 2024-04-27T00:13:29.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:29 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/662436319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:13:29.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:29 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1427805167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:13:29.364 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-27T00:13:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.582+0000 7f69cea00640 1 -- 172.21.15.40:0/1996634019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69c8102980 msgr2=0x7f69c8102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.582+0000 7f69cea00640 1 --2- 172.21.15.40:0/1996634019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69c8102980 0x7f69c8102da0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009900 tx=0x7f69b002a410 comp rx=0 tx=0).stop 2024-04-27T00:13:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.582+0000 7f69cea00640 1 -- 172.21.15.40:0/1996634019 shutdown_connections 2024-04-27T00:13:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.582+0000 7f69cea00640 1 --2- 172.21.15.40:0/1996634019 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69c8104530 0x7f69c810adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.582+0000 7f69cea00640 1 --2- 172.21.15.40:0/1996634019 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69c8103b70 0x7f69c8103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.582+0000 7f69cea00640 1 --2- 172.21.15.40:0/1996634019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69c8102980 0x7f69c8102da0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.582+0000 7f69cea00640 1 -- 172.21.15.40:0/1996634019 >> 172.21.15.40:0/1996634019 conn(0x7f69c80fdf30 msgr2=0x7f69c8100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.582+0000 7f69cea00640 1 -- 172.21.15.40:0/1996634019 shutdown_connections 2024-04-27T00:13:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.583+0000 7f69cea00640 1 -- 172.21.15.40:0/1996634019 wait complete. 2024-04-27T00:13:29.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.584+0000 7f69cea00640 1 Processor -- start 2024-04-27T00:13:29.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.584+0000 7f69cea00640 1 -- start start 2024-04-27T00:13:29.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.584+0000 7f69cea00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69c8102980 0x7f69c819ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:29.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.584+0000 7f69cea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69c8103b70 0x7f69c819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:29.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.584+0000 7f69cea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69c8104530 0x7f69c81a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:29.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.584+0000 7f69cea00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f69c810e4f0 con 0x7f69c8104530 2024-04-27T00:13:29.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.584+0000 7f69cea00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f69c810e370 con 0x7f69c8102980 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.584+0000 7f69cea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f69c810e670 con 0x7f69c8103b70 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.584+0000 7f69cca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69c8104530 0x7f69c81a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.585+0000 7f69c7e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69c8102980 0x7f69c819ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.585+0000 7f69cca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69c8104530 0x7f69c81a23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41136/0 (socket says 172.21.15.40:41136) 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.585+0000 7f69cca00640 1 -- 172.21.15.40:0/4220327487 learned_addr learned my addr 172.21.15.40:0/4220327487 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.585+0000 7f69c7400640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69c8103b70 0x7f69c819b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.585+0000 7f69cca00640 1 -- 172.21.15.40:0/4220327487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69c8103b70 msgr2=0x7f69c819b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.585+0000 7f69cca00640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69c8103b70 0x7f69c819b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.585+0000 7f69cca00640 1 -- 172.21.15.40:0/4220327487 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69c8102980 msgr2=0x7f69c819ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.585+0000 7f69cca00640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69c8102980 0x7f69c819ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.585+0000 7f69cca00640 1 -- 172.21.15.40:0/4220327487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c81a2a40 con 0x7f69c8104530 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.585+0000 7f69cca00640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69c8104530 0x7f69c81a23d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f69c000eb10 tx=0x7f69c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:29.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.586+0000 7f69c4c00640 1 -- 172.21.15.40:0/4220327487 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0004b20 con 0x7f69c8104530 2024-04-27T00:13:29.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.586+0000 7f69cea00640 1 -- 172.21.15.40:0/4220327487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c81a2d30 con 0x7f69c8104530 2024-04-27T00:13:29.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.586+0000 7f69c4c00640 1 -- 172.21.15.40:0/4220327487 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69c0004cc0 con 0x7f69c8104530 2024-04-27T00:13:29.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.586+0000 7f69c4c00640 1 -- 172.21.15.40:0/4220327487 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0010670 con 0x7f69c8104530 2024-04-27T00:13:29.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.586+0000 7f69cea00640 1 -- 172.21.15.40:0/4220327487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c81a31d0 con 0x7f69c8104530 2024-04-27T00:13:29.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.587+0000 7f69cea00640 1 -- 172.21.15.40:0/4220327487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69c81a3460 con 0x7f69c8104530 2024-04-27T00:13:29.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.590+0000 7f69c4c00640 1 -- 172.21.15.40:0/4220327487 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f69c0005020 con 0x7f69c8104530 2024-04-27T00:13:29.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.591+0000 7f69c4c00640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f69ac0a3c40 0x7f69ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:13:29.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.591+0000 7f69c4c00640 1 -- 172.21.15.40:0/4220327487 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f69c00d0e90 con 0x7f69c8104530 2024-04-27T00:13:29.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.591+0000 7f69c7e00640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f69ac0a3c40 0x7f69ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:13:29.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.592+0000 7f69c7e00640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f69ac0a3c40 0x7f69ac0a6100 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f69b0005ab0 tx=0x7f69b0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:13:29.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.595+0000 7f69c4c00640 1 -- 172.21.15.40:0/4220327487 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f69c009d370 con 0x7f69c8104530 2024-04-27T00:13:29.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.860+0000 7f69cea00640 1 -- 172.21.15.40:0/4220327487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f69c81a37c0 con 0x7f69c8104530 2024-04-27T00:13:29.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.891+0000 7f69c4c00640 1 -- 172.21.15.40:0/4220327487 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v17) ==== 81+0+745778 (secure 0 0 0) 0x7f69c009cc70 con 0x7f69c8104530 2024-04-27T00:13:29.897 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:13:29.897 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-27T00:13:29.897 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 17, 2024-04-27T00:13:29.897 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.897 INFO:teuthology.orchestra.run.smithi040.stdout: "active_gid": 14152, 2024-04-27T00:13:29.897 INFO:teuthology.orchestra.run.smithi040.stdout: "active_name": "x", 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: "active_addrs": { 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6800", 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3777720410 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3777720410 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:29.898 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: "active_addr": "172.21.15.40:6801/3777720410", 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: "active_change": "2024-04-26T23:50:44.546680+0000", 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: "active_mgr_features": 4540701547738038271, 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: "available": true, 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: "standbys": [ 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: "gid": 24109, 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "z", 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_features": 4540701547738038271, 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: "available_modules": [ 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:29.899 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "alerts", 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.900 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How frequently to reexamine health status", 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:29.901 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:29.902 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:29.903 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:29.904 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.905 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_destination": { 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_destination", 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email address to send alerts to", 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.906 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_from_name": { 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_from_name", 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "Ceph", 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email From: name", 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.907 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_host": { 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_host", 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP server", 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.908 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_password": { 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_password", 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Password to authenticate with", 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_port": { 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_port", 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "465", 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP port", 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.910 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_sender": { 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_sender", 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP envelope sender", 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.911 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_ssl": { 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_ssl", 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_user": { 2024-04-27T00:13:29.912 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_user", 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "User to authenticate as", 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:29.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "balancer", 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "active": { 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "active", 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.914 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically balance PGs across cluster", 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_time": { 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_time", 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0000", 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.915 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "beginning time of day to automatically balance", 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_weekday": { 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_weekday", 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.916 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_max_iterations": { 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_max_iterations", 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "25", 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "250", 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.917 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-27T00:13:29.918 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.918 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.918 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.918 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_metrics": { 2024-04-27T00:13:29.918 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_metrics", 2024-04-27T00:13:29.918 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.918 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.918 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.918 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "pgs,objects,bytes", 2024-04-27T00:13:29.918 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_step": { 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_step", 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.5", 2024-04-27T00:13:29.919 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0.001", 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "0.999", 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "aggressiveness of optimization", 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "end_time": { 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_time", 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2359", 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ending time of day to automatically balance", 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "end_weekday": { 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_weekday", 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-04-27T00:13:29.921 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.922 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:29.923 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:29.924 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.924 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.924 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.925 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.925 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.925 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.925 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.925 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:29.926 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:29.927 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "min_score": { 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_score", 2024-04-27T00:13:29.928 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.929 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "upmap", 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-compat", 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: "none", 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: "read", 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap", 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap-read" 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Balancer mode", 2024-04-27T00:13:29.930 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_ids": { 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_ids", 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-27T00:13:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_deviation": { 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_deviation", 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.933 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_optimizations": { 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_optimizations", 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.934 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm", 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_down_multiplier": { 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_down_multiplier", 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.935 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_refresh_rate": { 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_refresh_rate", 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.936 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "20", 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_starting_port": { 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_starting_port", 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4721", 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.937 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_ptrace": { 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_ptrace", 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.938 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_interval": { 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_interval", 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.939 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_memory_target_ratio": { 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_memory_target_ratio", 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.7", 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.940 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm_log_destination": { 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm_log_destination", 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "file", 2024-04-27T00:13:29.941 INFO:teuthology.orchestra.run.smithi040.stdout: "file,syslog", 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: "syslog" 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: "cgroups_split": { 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cgroups_split", 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:29.942 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "config_checks_enabled": { 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_checks_enabled", 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.943 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "config_dashboard": { 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_dashboard", 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-27T00:13:29.944 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_alertmanager": { 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_alertmanager", 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.945 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_base": { 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_base", 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Container image name, without the tag", 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_elasticsearch": { 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_elasticsearch", 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "elasticsearch container image", 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_grafana": { 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_grafana", 2024-04-27T00:13:29.947 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_haproxy": { 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_haproxy", 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.948 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "HAproxy container image", 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_agent": { 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_agent", 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.949 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger agent container image", 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_collector": { 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_collector", 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.950 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger collector container image", 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_query": { 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_query", 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.951 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger query container image", 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_keepalived": { 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_keepalived", 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-27T00:13:29.952 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Keepalived container image", 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_loki": { 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_loki", 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.953 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Loki container image", 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_node_exporter": { 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_node_exporter", 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.954 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_nvmeof": { 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_nvmeof", 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Nvme-of container image", 2024-04-27T00:13:29.955 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_prometheus": { 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_prometheus", 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.956 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_promtail": { 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_promtail", 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Promtail container image", 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_samba": { 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_samba", 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Samba/SMB container image", 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.958 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_snmp_gateway": { 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_snmp_gateway", 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SNMP Gateway container image", 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "container_init": { 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_init", 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.959 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Run podman/docker with `--init`", 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "daemon_cache_timeout": { 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "daemon_cache_timeout", 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.960 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "default_cephadm_command_timeout": { 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_cephadm_command_timeout", 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "900", 2024-04-27T00:13:29.961 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "default_registry": { 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_registry", 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io", 2024-04-27T00:13:29.962 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "device_cache_timeout": { 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_cache_timeout", 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1800", 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.963 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache device inventory", 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "device_enhanced_scan": { 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_enhanced_scan", 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.964 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "facts_cache_timeout": { 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "facts_cache_timeout", 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache host facts data", 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.965 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "grafana_dashboards_path": { 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "grafana_dashboards_path", 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.966 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "host_check_interval": { 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "host_check_interval", 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to perform a host check", 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.967 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "hw_monitoring": { 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hw_monitoring", 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "inventory_list_all": { 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "inventory_list_all", 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.969 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:29.970 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "log_refresh_metadata": { 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_refresh_metadata", 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.971 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.973 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:29.974 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-27T00:13:29.975 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-27T00:13:29.976 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "*", 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "max_count_per_host": { 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_count_per_host", 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.977 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of daemons per service per host", 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd_draining_count": { 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_osd_draining_count", 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.978 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "migration_current": { 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "migration_current", 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.979 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "internal - do not modify", 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "root", 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.980 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm-package", 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "root" 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "mode for remote execution of cephadm", 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "oob_default_addr": { 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "oob_default_addr", 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "169.254.1.1", 2024-04-27T00:13:29.981 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "prometheus_alerts_path": { 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus_alerts_path", 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-27T00:13:29.982 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_insecure": { 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_insecure", 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.983 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_password": { 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_password", 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.984 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_url": { 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_url", 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.985 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_username": { 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_username", 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.986 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "secure_monitoring_stack": { 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "secure_monitoring_stack", 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-27T00:13:29.987 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "service_discovery_port": { 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "service_discovery_port", 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8765", 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "cephadm service discovery port", 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.988 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "ssh_config_file": { 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssh_config_file", 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.989 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "use_agent": { 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_agent", 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.990 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "use_repo_digest": { 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_repo_digest", 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_failed_host_check": { 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_failed_host_check", 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if the host check fails", 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_daemons": { 2024-04-27T00:13:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_daemons", 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_hosts": { 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_hosts", 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.993 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:29.994 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crash", 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:29.995 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.996 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:29.997 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:29.999 INFO:teuthology.orchestra.run.smithi040.stdout: "retain_interval": { 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retain_interval", 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "31536000", 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.000 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_recent_interval": { 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_recent_interval", 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1209600", 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dashboard", 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.002 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_HOST": { 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.003 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.004 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_ENABLED": { 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_ENABLED", 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.005 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.006 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "ENABLE_BROWSABLE_API": { 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.007 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.008 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.009 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.010 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.013 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_PASSWORD": { 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.016 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_URL": { 2024-04-27T00:13:30.017 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_URL", 2024-04-27T00:13:30.018 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.018 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.018 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.018 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.018 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.018 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.018 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.018 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.018 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.018 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.018 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_USERNAME": { 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.019 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.020 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.021 INFO:teuthology.orchestra.run.smithi040.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "[]", 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "MULTICLUSTER_CONFIG": { 2024-04-27T00:13:30.024 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "{}", 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_HOST": { 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-27T00:13:30.025 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-27T00:13:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-27T00:13:30.027 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.028 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.029 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.030 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.031 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.032 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.033 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_ENABLED": { 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.034 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.035 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.036 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8", 2024-04-27T00:13:30.037 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "45", 2024-04-27T00:13:30.038 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ACCESS_KEY": { 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.039 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.040 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SECRET_KEY": { 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.041 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SSL_VERIFY": { 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.042 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "UNSAFE_TLS_v1_2": { 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.043 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.044 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.045 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-27T00:13:30.046 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-27T00:13:30.046 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.046 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.046 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.046 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-27T00:13:30.046 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.046 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.046 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.046 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.046 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.046 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.046 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "cross_origin_url": { 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cross_origin_url", 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.047 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "crt_file": { 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crt_file", 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "debug": { 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "debug", 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable/disable debug options", 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.049 INFO:teuthology.orchestra.run.smithi040.stdout: "jwt_token_ttl": { 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "jwt_token_ttl", 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "28800", 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.050 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.051 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.052 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.053 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.054 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.055 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "motd": { 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "motd", 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.056 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "The message of the day", 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect_resolve_ip_addr": { 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "redirect_resolve_ip_addr", 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-04-27T00:13:30.058 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8080", 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.059 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.060 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl_server_port": { 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl_server_port", 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8443", 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.061 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "redirect", 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect" 2024-04-27T00:13:30.062 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.063 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "url_prefix": { 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url_prefix", 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.064 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "devicehealth", 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_monitoring": { 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_monitoring", 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.065 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "monitor device health metrics", 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.066 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.067 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.068 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.070 INFO:teuthology.orchestra.run.smithi040.stdout: "mark_out_threshold": { 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mark_out_threshold", 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2419200", 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.071 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": { 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_name", 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "device_health_metrics", 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.072 INFO:teuthology.orchestra.run.smithi040.stdout: "retention_period": { 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retention_period", 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15552000", 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain device health metrics", 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_frequency": { 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_frequency", 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.074 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "self_heal": { 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "self_heal", 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and check device health", 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_threshold": { 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_threshold", 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7257600", 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.077 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "diskprediction_local", 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.078 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.080 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.081 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "predict_interval": { 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predict_interval", 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.083 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "predictor_model": { 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predictor_model", 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "prophetstor", 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.084 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.085 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "influx", 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": false, 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "influxdb python module not found", 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: "batch_size": { 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "batch_size", 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.086 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5000", 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "database": { 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "database", 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.087 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "ceph", 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": { 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hostname", 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.088 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server hostname", 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "30", 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "5", 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.089 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.090 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.091 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.092 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.094 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "password": { 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "password", 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "password of InfluxDB server user", 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.095 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "port": { 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "port", 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8086", 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server port", 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.096 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "false", 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "threads": { 2024-04-27T00:13:30.097 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threads", 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "32", 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "username": { 2024-04-27T00:13:30.098 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "username", 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "username of InfluxDB server user", 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "verify_ssl": { 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "verify_ssl", 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.099 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "true", 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "insights", 2024-04-27T00:13:30.100 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.101 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.102 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.103 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.104 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "iostat", 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.106 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.107 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.108 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.109 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "k8sevents", 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_event_retention_days": { 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ceph_event_retention_days", 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7", 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-27T00:13:30.111 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "config_check_secs": { 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_check_secs", 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "10", 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.112 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.113 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.114 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.115 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.116 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "localpool", 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "failure_domain": { 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "failure_domain", 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "host", 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "failure domain for any created local pool", 2024-04-27T00:13:30.118 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.119 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.120 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.121 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": { 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_size", 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.123 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default min_size for any created local pool", 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "num_rep": { 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "num_rep", 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3", 2024-04-27T00:13:30.124 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default replica count for any created local pool", 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": { 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_num", 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "128", 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.125 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default pg_num for any created local pool", 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "prefix": { 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prefix", 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.126 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name prefix for any created local pool", 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "subtree": { 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "subtree", 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "rack", 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.127 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mds_autoscaler", 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.128 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.129 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.130 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.131 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.132 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirroring", 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.133 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.134 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.135 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.136 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.137 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "nfs", 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.138 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.139 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.140 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.141 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.142 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.143 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "fail_fs": { 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "fail_fs", 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.144 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.145 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.146 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.147 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.148 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator": { 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm", 2024-04-27T00:13:30.149 INFO:teuthology.orchestra.run.smithi040.stdout: "rook", 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: "test_orchestrator" 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Orchestrator backend", 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_perf_query", 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.150 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.151 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.152 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.153 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.154 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_support", 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.155 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.156 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.157 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.158 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.159 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_autoscaler", 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.160 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.161 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.162 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.163 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.164 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:30.165 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "threshold": { 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threshold", 2024-04-27T00:13:30.166 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1.0", 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "scaling threshold", 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "progress", 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_pg_recovery_event": { 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_pg_recovery_event", 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-27T00:13:30.168 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.169 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.170 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.171 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.172 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.173 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "max_completed_events": { 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_completed_events", 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.174 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "50", 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "number of past completed events to remember", 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.175 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long the module is going to sleep", 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus", 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.176 INFO:teuthology.orchestra.run.smithi040.stdout: "cache": { 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cache", 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "exclude_perf_counters": { 2024-04-27T00:13:30.177 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "exclude_perf_counters", 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.178 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.179 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.181 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.182 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools": { 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools", 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.183 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "300", 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.184 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_interval": { 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_interval", 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15.0", 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.185 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-27T00:13:30.186 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "9283", 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.187 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "stale_cache_strategy": { 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stale_cache_strategy", 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "log", 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.188 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "default", 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "default", 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "error" 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.189 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.190 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_support", 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.191 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.192 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.193 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.194 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.195 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_snap_create": { 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_snap_create", 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.196 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "mirror_snapshot_schedule": { 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirror_snapshot_schedule", 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.197 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.198 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.198 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.198 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.198 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.198 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.198 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.198 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.198 INFO:teuthology.orchestra.run.smithi040.stdout: "trash_purge_schedule": { 2024-04-27T00:13:30.198 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "trash_purge_schedule", 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "restful", 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_auth": { 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_auth", 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.200 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.201 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.202 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.203 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.204 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.205 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.206 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.207 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.208 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rgw", 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.209 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.210 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.211 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.212 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rook", 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.215 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.216 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.217 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "storage_class": { 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "storage_class", 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "local", 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.219 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "selftest", 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.220 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.221 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.222 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.223 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.224 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "roption1": { 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption1", 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.225 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "roption2": { 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption2", 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "xyz", 2024-04-27T00:13:30.226 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption1": { 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption1", 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.227 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption2": { 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption2", 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.228 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption3": { 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption3", 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.229 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption4": { 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption4", 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.230 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption5": { 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption5", 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.231 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption6": { 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption6", 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.232 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption7": { 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption7", 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "42", 2024-04-27T00:13:30.233 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: "testkey": { 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testkey", 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.234 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "testlkey": { 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testlkey", 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.235 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "testnewline": { 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testnewline", 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.236 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snap_schedule", 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_m_granularity": { 2024-04-27T00:13:30.237 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_m_granularity", 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow minute scheduled snapshots", 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "dump_on_update": { 2024-04-27T00:13:30.238 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dump_on_update", 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "dump database to debug log on update", 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.239 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.240 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.241 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.242 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.243 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stats", 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.245 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.246 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.247 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.249 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "status", 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.250 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.251 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.252 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.253 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.254 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.254 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.254 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.254 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.254 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.254 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.254 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.254 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.254 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.254 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.254 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telegraf", 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.255 INFO:teuthology.orchestra.run.smithi040.stdout: "address": { 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "address", 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:30.256 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15", 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.257 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.258 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.259 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.260 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.261 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.262 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telemetry", 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_basic": { 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_basic", 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-27T00:13:30.263 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_crash": { 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_crash", 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-27T00:13:30.264 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_device": { 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_device", 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.265 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_ident": { 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_ident", 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.266 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_perf": { 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_perf", 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "contact": { 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "contact", 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.268 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "description": { 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "description", 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.269 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: "device_url": { 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_url", 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.270 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.271 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "24", 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "8", 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.272 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "last_opt_revision": { 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "last_opt_revision", 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1", 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard": { 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard", 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.274 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard_description": { 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard_description", 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.275 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.276 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.277 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.278 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.280 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "organization": { 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "organization", 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.281 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "proxy": { 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "proxy", 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "url": { 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url", 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.283 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "test_orchestrator", 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.284 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.285 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.286 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.287 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.288 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.289 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.290 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.291 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.292 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.293 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_clones": { 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_clones", 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4", 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.294 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "periodic_async_work": { 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "periodic_async_work", 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.295 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Periodically check for async work", 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_delay": { 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_delay", 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.296 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_no_wait": { 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_no_wait", 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.297 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix", 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: "discovery_interval": { 2024-04-27T00:13:30.298 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "discovery_interval", 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "100", 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "identifier": { 2024-04-27T00:13:30.299 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "identifier", 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.300 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.301 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.302 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.303 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.304 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.305 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.306 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.306 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.306 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.306 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.306 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.306 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.306 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.306 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.306 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.306 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.307 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:30 smithi080 ceph-mon[32362]: pgmap v766: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-27T00:13:30.307 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:30 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2814249440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:13:30.307 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:30 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/4220327487' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-27T00:13:30.310 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.310 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_host": { 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_host", 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_port": { 2024-04-27T00:13:30.311 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_port", 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10051", 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_sender": { 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_sender", 2024-04-27T00:13:30.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.313 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: "gid": 24110, 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "y", 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_features": 4540701547738038271, 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: "available_modules": [ 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "alerts", 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:30.314 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How frequently to reexamine health status", 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.315 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.316 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.318 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.319 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_destination": { 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_destination", 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.320 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email address to send alerts to", 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_from_name": { 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_from_name", 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "Ceph", 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email From: name", 2024-04-27T00:13:30.321 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_host": { 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_host", 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP server", 2024-04-27T00:13:30.322 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_password": { 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_password", 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Password to authenticate with", 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.323 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_port": { 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_port", 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "465", 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP port", 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_sender": { 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_sender", 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP envelope sender", 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.325 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_ssl": { 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_ssl", 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.326 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_user": { 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_user", 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "User to authenticate as", 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "balancer", 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "active": { 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "active", 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.328 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically balance PGs across cluster", 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_time": { 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_time", 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0000", 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.329 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "beginning time of day to automatically balance", 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_weekday": { 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_weekday", 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-04-27T00:13:30.330 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-27T00:13:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-27T00:13:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_max_iterations": { 2024-04-27T00:13:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_max_iterations", 2024-04-27T00:13:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "25", 2024-04-27T00:13:30.331 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-27T00:13:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "250", 2024-04-27T00:13:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-27T00:13:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.333 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_metrics": { 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_metrics", 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "pgs,objects,bytes", 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.334 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_step": { 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_step", 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.5", 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0.001", 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "0.999", 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "aggressiveness of optimization", 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi040.stdout: "end_time": { 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_time", 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2359", 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ending time of day to automatically balance", 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "end_weekday": { 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_weekday", 2024-04-27T00:13:30.336 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.337 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.338 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.339 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.340 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.341 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "min_score": { 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_score", 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.342 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "upmap", 2024-04-27T00:13:30.343 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-compat", 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "none", 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "read", 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap", 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap-read" 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Balancer mode", 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_ids": { 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_ids", 2024-04-27T00:13:30.344 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.345 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_deviation": { 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_deviation", 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.346 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_optimizations": { 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_optimizations", 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.347 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm", 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.348 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_down_multiplier": { 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_down_multiplier", 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_refresh_rate": { 2024-04-27T00:13:30.349 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_refresh_rate", 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "20", 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.350 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_starting_port": { 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_starting_port", 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4721", 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_ptrace": { 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_ptrace", 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_interval": { 2024-04-27T00:13:30.352 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_interval", 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_memory_target_ratio": { 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_memory_target_ratio", 2024-04-27T00:13:30.353 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.7", 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm_log_destination": { 2024-04-27T00:13:30.354 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm_log_destination", 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "file", 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "file,syslog", 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "syslog" 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.355 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "cgroups_split": { 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cgroups_split", 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.356 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "config_checks_enabled": { 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_checks_enabled", 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.357 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "config_dashboard": { 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_dashboard", 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_alertmanager": { 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_alertmanager", 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.359 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_base": { 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_base", 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Container image name, without the tag", 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.360 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_elasticsearch": { 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_elasticsearch", 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "elasticsearch container image", 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.361 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_grafana": { 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_grafana", 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_haproxy": { 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_haproxy", 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "HAproxy container image", 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_agent": { 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_agent", 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger agent container image", 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.364 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_collector": { 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_collector", 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger collector container image", 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_query": { 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_query", 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger query container image", 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_keepalived": { 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_keepalived", 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Keepalived container image", 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.367 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_loki": { 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_loki", 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Loki container image", 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.368 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_node_exporter": { 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_node_exporter", 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_nvmeof": { 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_nvmeof", 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Nvme-of container image", 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_prometheus": { 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_prometheus", 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.371 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_promtail": { 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_promtail", 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Promtail container image", 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_samba": { 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_samba", 2024-04-27T00:13:30.372 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Samba/SMB container image", 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_snmp_gateway": { 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_snmp_gateway", 2024-04-27T00:13:30.373 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SNMP Gateway container image", 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "container_init": { 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_init", 2024-04-27T00:13:30.374 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Run podman/docker with `--init`", 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "daemon_cache_timeout": { 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "daemon_cache_timeout", 2024-04-27T00:13:30.375 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "default_cephadm_command_timeout": { 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_cephadm_command_timeout", 2024-04-27T00:13:30.376 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "900", 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "default_registry": { 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_registry", 2024-04-27T00:13:30.377 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io", 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "device_cache_timeout": { 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_cache_timeout", 2024-04-27T00:13:30.378 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1800", 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache device inventory", 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "device_enhanced_scan": { 2024-04-27T00:13:30.379 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_enhanced_scan", 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "facts_cache_timeout": { 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "facts_cache_timeout", 2024-04-27T00:13:30.380 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache host facts data", 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "grafana_dashboards_path": { 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "grafana_dashboards_path", 2024-04-27T00:13:30.381 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "host_check_interval": { 2024-04-27T00:13:30.382 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "host_check_interval", 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to perform a host check", 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "hw_monitoring": { 2024-04-27T00:13:30.383 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hw_monitoring", 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "inventory_list_all": { 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "inventory_list_all", 2024-04-27T00:13:30.384 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.386 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "log_refresh_metadata": { 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_refresh_metadata", 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.389 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.390 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.391 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "*", 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.392 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "max_count_per_host": { 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_count_per_host", 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of daemons per service per host", 2024-04-27T00:13:30.393 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd_draining_count": { 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_osd_draining_count", 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.394 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "migration_current": { 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "migration_current", 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "internal - do not modify", 2024-04-27T00:13:30.395 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-04-27T00:13:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-04-27T00:13:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.396 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "root", 2024-04-27T00:13:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm-package", 2024-04-27T00:13:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: "root" 2024-04-27T00:13:30.397 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "mode for remote execution of cephadm", 2024-04-27T00:13:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "oob_default_addr": { 2024-04-27T00:13:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "oob_default_addr", 2024-04-27T00:13:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "169.254.1.1", 2024-04-27T00:13:30.398 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "prometheus_alerts_path": { 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus_alerts_path", 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-27T00:13:30.399 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_insecure": { 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_insecure", 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_password": { 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_password", 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.401 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_url": { 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_url", 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.402 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_username": { 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_username", 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.403 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "secure_monitoring_stack": { 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "secure_monitoring_stack", 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.404 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "service_discovery_port": { 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "service_discovery_port", 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.405 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8765", 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "cephadm service discovery port", 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "ssh_config_file": { 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssh_config_file", 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.406 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "use_agent": { 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_agent", 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.407 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "use_repo_digest": { 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_repo_digest", 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.408 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_failed_host_check": { 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_failed_host_check", 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.409 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if the host check fails", 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_daemons": { 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_daemons", 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.410 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_hosts": { 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_hosts", 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.411 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crash", 2024-04-27T00:13:30.412 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.413 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.414 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.415 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.416 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.417 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "retain_interval": { 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retain_interval", 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "31536000", 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.418 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_recent_interval": { 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_recent_interval", 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1209600", 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.419 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dashboard", 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.420 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_HOST": { 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.421 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.422 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_ENABLED": { 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_ENABLED", 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.423 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.424 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "ENABLE_BROWSABLE_API": { 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.425 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.426 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.427 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.428 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.429 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.430 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.430 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.430 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.430 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-27T00:13:30.430 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-27T00:13:30.430 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.430 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.430 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.430 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.430 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.431 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.432 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.433 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_PASSWORD": { 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.434 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.435 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.436 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.436 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.436 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.436 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.436 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.436 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_URL": { 2024-04-27T00:13:30.436 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_URL", 2024-04-27T00:13:30.436 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.436 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.437 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.438 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.438 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.438 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.438 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.438 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.438 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.438 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.438 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_USERNAME": { 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.441 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.442 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.443 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "[]", 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.444 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "MULTICLUSTER_CONFIG": { 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "{}", 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.445 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_HOST": { 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.446 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.447 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.448 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.449 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.450 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.451 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.452 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-27T00:13:30.452 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-27T00:13:30.452 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.452 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.452 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.452 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.452 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.452 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.452 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.452 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.452 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.453 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.454 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_ENABLED": { 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.455 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.456 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.457 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8", 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.458 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "45", 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.459 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ACCESS_KEY": { 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.460 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.461 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SECRET_KEY": { 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.462 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SSL_VERIFY": { 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.463 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "UNSAFE_TLS_v1_2": { 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.464 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.465 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.466 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.467 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "cross_origin_url": { 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cross_origin_url", 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.468 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "crt_file": { 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crt_file", 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.469 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "debug": { 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "debug", 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable/disable debug options", 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.470 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.471 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.471 INFO:teuthology.orchestra.run.smithi040.stdout: "jwt_token_ttl": { 2024-04-27T00:13:30.471 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "jwt_token_ttl", 2024-04-27T00:13:30.471 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.471 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.471 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.471 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "28800", 2024-04-27T00:13:30.471 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.471 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.471 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.471 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.472 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.473 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.474 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.475 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.476 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.477 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: "motd": { 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "motd", 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.478 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "The message of the day", 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect_resolve_ip_addr": { 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "redirect_resolve_ip_addr", 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.479 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.480 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8080", 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.481 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.482 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.482 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.482 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-04-27T00:13:30.482 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-04-27T00:13:30.482 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.482 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.482 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.482 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.482 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.482 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.482 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl_server_port": { 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl_server_port", 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8443", 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.483 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "redirect", 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.484 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect" 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.485 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: "url_prefix": { 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url_prefix", 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.486 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "devicehealth", 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.487 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.488 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.488 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_monitoring": { 2024-04-27T00:13:30.488 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_monitoring", 2024-04-27T00:13:30.488 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.488 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.488 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.488 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.488 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.488 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.488 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.488 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "monitor device health metrics", 2024-04-27T00:13:30.488 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.489 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.490 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.491 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.492 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.493 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "mark_out_threshold": { 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mark_out_threshold", 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2419200", 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-27T00:13:30.494 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": { 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_name", 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "device_health_metrics", 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.495 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "retention_period": { 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retention_period", 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15552000", 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain device health metrics", 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.496 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_frequency": { 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_frequency", 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "self_heal": { 2024-04-27T00:13:30.497 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "self_heal", 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:30.498 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and check device health", 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_threshold": { 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_threshold", 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.499 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7257600", 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "diskprediction_local", 2024-04-27T00:13:30.500 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.501 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.502 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.503 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.505 INFO:teuthology.orchestra.run.smithi040.stdout: "predict_interval": { 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predict_interval", 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.506 INFO:teuthology.orchestra.run.smithi040.stdout: "predictor_model": { 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predictor_model", 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "prophetstor", 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:30.507 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "influx", 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": false, 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "influxdb python module not found", 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "batch_size": { 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "batch_size", 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5000", 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.509 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: "database": { 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "database", 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "ceph", 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-27T00:13:30.510 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": { 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hostname", 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server hostname", 2024-04-27T00:13:30.511 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "30", 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "5", 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.512 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.513 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.514 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.515 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.516 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.517 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: "password": { 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "password", 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.518 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "password of InfluxDB server user", 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: "port": { 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "port", 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8086", 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.519 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server port", 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "false", 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.520 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "threads": { 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threads", 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "32", 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-27T00:13:30.521 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "username": { 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "username", 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "username of InfluxDB server user", 2024-04-27T00:13:30.522 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "verify_ssl": { 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "verify_ssl", 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "true", 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.523 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "insights", 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.524 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.525 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.526 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.527 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.528 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "iostat", 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.529 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.531 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.532 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.533 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "k8sevents", 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_event_retention_days": { 2024-04-27T00:13:30.534 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ceph_event_retention_days", 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7", 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "config_check_secs": { 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_check_secs", 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.535 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "10", 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.539 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.540 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "localpool", 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: "failure_domain": { 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "failure_domain", 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.541 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "host", 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "failure domain for any created local pool", 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.542 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.543 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.544 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.544 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.544 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.544 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.544 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.544 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.544 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.544 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.544 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.544 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.544 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.545 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.546 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": { 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_size", 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.547 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default min_size for any created local pool", 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "num_rep": { 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "num_rep", 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3", 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.548 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default replica count for any created local pool", 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": { 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_num", 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "128", 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.549 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default pg_num for any created local pool", 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "prefix": { 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prefix", 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name prefix for any created local pool", 2024-04-27T00:13:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "subtree": { 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "subtree", 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "rack", 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-27T00:13:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mds_autoscaler", 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.552 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.553 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.556 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirroring", 2024-04-27T00:13:30.557 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.558 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.559 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.560 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.561 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.562 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "nfs", 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.563 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.564 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.565 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.566 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.567 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: "fail_fs": { 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "fail_fs", 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.568 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.569 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.570 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.571 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.572 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.573 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator": { 2024-04-27T00:13:30.574 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm", 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rook", 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "test_orchestrator" 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Orchestrator backend", 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.575 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.576 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.576 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.576 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_perf_query", 2024-04-27T00:13:30.576 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.576 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.576 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.577 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.577 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.577 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.577 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.577 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.577 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.577 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.577 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.578 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.578 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.578 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.578 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.578 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.578 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.578 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.579 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.580 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.581 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_support", 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.583 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.584 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.585 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.586 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.587 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_autoscaler", 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.588 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.589 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.590 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.591 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.592 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.593 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: "threshold": { 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threshold", 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.594 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1.0", 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "scaling threshold", 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "progress", 2024-04-27T00:13:30.595 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_pg_recovery_event": { 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_pg_recovery_event", 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.596 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.598 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.599 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.600 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.601 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.602 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "max_completed_events": { 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_completed_events", 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "50", 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "number of past completed events to remember", 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.603 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long the module is going to sleep", 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.604 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus", 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: "cache": { 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cache", 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.605 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: "exclude_perf_counters": { 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "exclude_perf_counters", 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.606 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.607 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.608 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.609 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.610 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.611 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools": { 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools", 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.612 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "300", 2024-04-27T00:13:30.613 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_interval": { 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_interval", 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.614 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15.0", 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.615 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "9283", 2024-04-27T00:13:30.616 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "stale_cache_strategy": { 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stale_cache_strategy", 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "log", 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.617 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "default", 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.618 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: "default", 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: "error" 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.619 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_support", 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.620 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.621 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.622 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.623 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.624 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.625 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_snap_create": { 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_snap_create", 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.626 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "mirror_snapshot_schedule": { 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirror_snapshot_schedule", 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.627 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "trash_purge_schedule": { 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "trash_purge_schedule", 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.628 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "restful", 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_auth": { 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_auth", 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.629 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.630 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.633 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.634 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.635 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.636 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.637 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rgw", 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.638 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.639 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.640 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.641 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.642 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rook", 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.644 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.645 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.646 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.647 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.648 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "storage_class": { 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "storage_class", 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "local", 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.649 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "selftest", 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.650 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.652 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.653 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.654 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "roption1": { 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption1", 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.655 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "roption2": { 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption2", 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "xyz", 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.656 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption1": { 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption1", 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.657 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption2": { 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption2", 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption3": { 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption3", 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.659 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption4": { 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption4", 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption5": { 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption5", 2024-04-27T00:13:30.660 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption6": { 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption6", 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.661 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption7": { 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption7", 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.662 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "42", 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "testkey": { 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testkey", 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "testlkey": { 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testlkey", 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "testnewline": { 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testnewline", 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.665 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snap_schedule", 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_m_granularity": { 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_m_granularity", 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.666 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow minute scheduled snapshots", 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "dump_on_update": { 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dump_on_update", 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.667 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "dump database to debug log on update", 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.668 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.669 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.670 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.671 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.672 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stats", 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.673 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.674 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.675 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.676 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.677 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "status", 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.678 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.679 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.680 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.681 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.682 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telegraf", 2024-04-27T00:13:30.683 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "address": { 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "address", 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.684 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15", 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.685 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.686 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.687 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.688 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.690 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telemetry", 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_basic": { 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_basic", 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.691 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_crash": { 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_crash", 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.692 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_device": { 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_device", 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.693 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_ident": { 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_ident", 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.694 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_perf": { 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_perf", 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.695 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "contact": { 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "contact", 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.696 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "description": { 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "description", 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "device_url": { 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_url", 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.698 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.699 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:30.700 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "24", 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "8", 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "last_opt_revision": { 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "last_opt_revision", 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.701 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1", 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard": { 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard", 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.702 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard_description": { 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard_description", 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.703 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.704 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.705 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.706 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.706 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.908+0000 7f69cea00640 1 -- 172.21.15.40:0/4220327487 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f69ac0a3c40 msgr2=0x7f69ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:30.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.908+0000 7f69cea00640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f69ac0a3c40 0x7f69ac0a6100 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f69b0005ab0 tx=0x7f69b0005a20 comp rx=0 tx=0).stop 2024-04-27T00:13:30.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.908+0000 7f69cea00640 1 -- 172.21.15.40:0/4220327487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69c8104530 msgr2=0x7f69c81a23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:13:30.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.908+0000 7f69cea00640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69c8104530 0x7f69c81a23d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f69c000eb10 tx=0x7f69c000c2d0 comp rx=0 tx=0).stop 2024-04-27T00:13:30.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.908+0000 7f69cea00640 1 -- 172.21.15.40:0/4220327487 shutdown_connections 2024-04-27T00:13:30.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.908+0000 7f69cea00640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69c8104530 0x7f69c81a23d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:30.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.908+0000 7f69cea00640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f69ac0a3c40 0x7f69ac0a6100 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:30.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.908+0000 7f69cea00640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69c8103b70 0x7f69c819b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:30.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.908+0000 7f69cea00640 1 --2- 172.21.15.40:0/4220327487 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f69c8102980 0x7f69c819ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:13:30.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.908+0000 7f69cea00640 1 -- 172.21.15.40:0/4220327487 >> 172.21.15.40:0/4220327487 conn(0x7f69c80fdf30 msgr2=0x7f69c8074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:13:30.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.909+0000 7f69cea00640 1 -- 172.21.15.40:0/4220327487 shutdown_connections 2024-04-27T00:13:30.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:13:29.909+0000 7f69cea00640 1 -- 172.21.15.40:0/4220327487 wait complete. 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.710 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.711 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "organization": { 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "organization", 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.713 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "proxy": { 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "proxy", 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.714 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "url": { 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url", 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.715 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "test_orchestrator", 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.717 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.718 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.719 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.720 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.721 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.722 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.723 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.724 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.725 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_clones": { 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_clones", 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4", 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.726 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: "periodic_async_work": { 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "periodic_async_work", 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Periodically check for async work", 2024-04-27T00:13:30.727 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_delay": { 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_delay", 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-27T00:13:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_no_wait": { 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_no_wait", 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-27T00:13:30.729 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix", 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: "discovery_interval": { 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "discovery_interval", 2024-04-27T00:13:30.730 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "100", 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "identifier": { 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "identifier", 2024-04-27T00:13:30.731 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:30.732 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.733 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.734 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.735 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.736 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.737 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_host": { 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_host", 2024-04-27T00:13:30.738 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_port": { 2024-04-27T00:13:30.739 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_port", 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10051", 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.740 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_sender": { 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_sender", 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.741 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: "modules": [ 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm", 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: "dashboard", 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: "iostat", 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: "nfs", 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: "restful" 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: "available_modules": [ 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.742 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "alerts", 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How frequently to reexamine health status", 2024-04-27T00:13:30.743 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.744 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.745 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.746 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.747 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.747 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.747 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.747 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.747 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.747 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.749 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.749 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.749 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.749 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.750 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_destination": { 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_destination", 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.751 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email address to send alerts to", 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_from_name": { 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_from_name", 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.752 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "Ceph", 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email From: name", 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_host": { 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_host", 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.753 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP server", 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_password": { 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_password", 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.754 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Password to authenticate with", 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_port": { 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_port", 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "465", 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP port", 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_sender": { 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_sender", 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.756 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP envelope sender", 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_ssl": { 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_ssl", 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.757 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_user": { 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_user", 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.758 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "User to authenticate as", 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "balancer", 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.759 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "active": { 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "active", 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically balance PGs across cluster", 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.760 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_time": { 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_time", 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0000", 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "beginning time of day to automatically balance", 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_weekday": { 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_weekday", 2024-04-27T00:13:30.761 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_max_iterations": { 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_max_iterations", 2024-04-27T00:13:30.762 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "25", 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "250", 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_metrics": { 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_metrics", 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.763 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "pgs,objects,bytes", 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_step": { 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_step", 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.764 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.5", 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0.001", 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "0.999", 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "aggressiveness of optimization", 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "end_time": { 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_time", 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.765 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2359", 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ending time of day to automatically balance", 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "end_weekday": { 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_weekday", 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:30.766 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.767 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.768 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.769 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.770 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.771 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "min_score": { 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_score", 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.772 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "upmap", 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-compat", 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "none", 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "read", 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap", 2024-04-27T00:13:30.773 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap-read" 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Balancer mode", 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_ids": { 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_ids", 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.774 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-27T00:13:30.775 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_deviation": { 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_deviation", 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.776 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_optimizations": { 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_optimizations", 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.777 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm", 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_down_multiplier": { 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_down_multiplier", 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.778 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_refresh_rate": { 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_refresh_rate", 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "20", 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.779 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_starting_port": { 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_starting_port", 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4721", 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.780 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_ptrace": { 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_ptrace", 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_interval": { 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_interval", 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.782 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_memory_target_ratio": { 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_memory_target_ratio", 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.7", 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm_log_destination": { 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm_log_destination", 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "file", 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "file,syslog", 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "syslog" 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.784 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: "cgroups_split": { 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cgroups_split", 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.785 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: "config_checks_enabled": { 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_checks_enabled", 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.786 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "config_dashboard": { 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_dashboard", 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.787 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_alertmanager": { 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_alertmanager", 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.788 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_base": { 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_base", 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.789 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Container image name, without the tag", 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_elasticsearch": { 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_elasticsearch", 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.790 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "elasticsearch container image", 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_grafana": { 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_grafana", 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.791 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_haproxy": { 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_haproxy", 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.792 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "HAproxy container image", 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_agent": { 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_agent", 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.793 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger agent container image", 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_collector": { 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_collector", 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.794 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger collector container image", 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_query": { 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_query", 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.795 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger query container image", 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_keepalived": { 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_keepalived", 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.796 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Keepalived container image", 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_loki": { 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_loki", 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.797 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Loki container image", 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_node_exporter": { 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_node_exporter", 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.798 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_nvmeof": { 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_nvmeof", 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-27T00:13:30.799 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Nvme-of container image", 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_prometheus": { 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_prometheus", 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-27T00:13:30.800 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_promtail": { 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_promtail", 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.801 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Promtail container image", 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_samba": { 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_samba", 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-27T00:13:30.802 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Samba/SMB container image", 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_snmp_gateway": { 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_snmp_gateway", 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-27T00:13:30.803 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SNMP Gateway container image", 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: "container_init": { 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_init", 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.804 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Run podman/docker with `--init`", 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "daemon_cache_timeout": { 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "daemon_cache_timeout", 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:30.805 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: "default_cephadm_command_timeout": { 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_cephadm_command_timeout", 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.806 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "900", 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "default_registry": { 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_registry", 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.807 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io", 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "device_cache_timeout": { 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_cache_timeout", 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1800", 2024-04-27T00:13:30.808 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache device inventory", 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "device_enhanced_scan": { 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_enhanced_scan", 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.809 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: "facts_cache_timeout": { 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "facts_cache_timeout", 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.810 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache host facts data", 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "grafana_dashboards_path": { 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "grafana_dashboards_path", 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-27T00:13:30.811 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "host_check_interval": { 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "host_check_interval", 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:30.812 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to perform a host check", 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: "hw_monitoring": { 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hw_monitoring", 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.813 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: "inventory_list_all": { 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "inventory_list_all", 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.814 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.815 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.816 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "log_refresh_metadata": { 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_refresh_metadata", 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.818 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.819 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.820 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.821 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "*", 2024-04-27T00:13:30.822 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "max_count_per_host": { 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_count_per_host", 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.823 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of daemons per service per host", 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd_draining_count": { 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_osd_draining_count", 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.824 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "migration_current": { 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "migration_current", 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.825 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "internal - do not modify", 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "root", 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm-package", 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: "root" 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "mode for remote execution of cephadm", 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: "oob_default_addr": { 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "oob_default_addr", 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "169.254.1.1", 2024-04-27T00:13:30.827 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "prometheus_alerts_path": { 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus_alerts_path", 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-27T00:13:30.828 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_insecure": { 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_insecure", 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.829 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_password": { 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_password", 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.830 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_url": { 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_url", 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.831 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_username": { 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_username", 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.832 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "secure_monitoring_stack": { 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "secure_monitoring_stack", 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.833 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "service_discovery_port": { 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "service_discovery_port", 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8765", 2024-04-27T00:13:30.834 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "cephadm service discovery port", 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "ssh_config_file": { 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssh_config_file", 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.835 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "use_agent": { 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_agent", 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.836 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "use_repo_digest": { 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_repo_digest", 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.837 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_failed_host_check": { 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_failed_host_check", 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.838 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if the host check fails", 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_daemons": { 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_daemons", 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.839 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_hosts": { 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_hosts", 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.840 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crash", 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.841 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.842 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.842 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.842 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.842 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.842 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.842 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.842 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.842 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.842 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.842 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.842 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.842 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.843 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.844 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.845 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "retain_interval": { 2024-04-27T00:13:30.846 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retain_interval", 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "31536000", 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_recent_interval": { 2024-04-27T00:13:30.847 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_recent_interval", 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1209600", 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dashboard", 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.849 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_HOST": { 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.851 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_ENABLED": { 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_ENABLED", 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.852 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.853 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: "ENABLE_BROWSABLE_API": { 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.854 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.855 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.856 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.860 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-27T00:13:30.861 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_PASSWORD": { 2024-04-27T00:13:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-27T00:13:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_URL": { 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_URL", 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.864 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_USERNAME": { 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.865 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.866 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "[]", 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "MULTICLUSTER_CONFIG": { 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "{}", 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_HOST": { 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.872 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.873 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.874 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.875 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.876 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-27T00:13:30.877 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.878 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-27T00:13:30.879 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_ENABLED": { 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-27T00:13:30.880 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.881 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.882 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.883 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8", 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.884 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "45", 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ACCESS_KEY": { 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.885 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.886 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SECRET_KEY": { 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.887 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SSL_VERIFY": { 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.888 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "UNSAFE_TLS_v1_2": { 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.889 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:30.890 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.891 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.892 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "cross_origin_url": { 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cross_origin_url", 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.893 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "crt_file": { 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crt_file", 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.894 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "debug": { 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "debug", 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable/disable debug options", 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.895 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "jwt_token_ttl": { 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "jwt_token_ttl", 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "28800", 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.897 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.898 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.899 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.900 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.901 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "motd": { 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "motd", 2024-04-27T00:13:30.902 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "The message of the day", 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect_resolve_ip_addr": { 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "redirect_resolve_ip_addr", 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.903 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.904 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.905 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8080", 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-04-27T00:13:30.906 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl_server_port": { 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl_server_port", 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.907 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8443", 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.908 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "redirect", 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect" 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-04-27T00:13:30.909 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "url_prefix": { 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url_prefix", 2024-04-27T00:13:30.910 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "devicehealth", 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_monitoring": { 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_monitoring", 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "monitor device health metrics", 2024-04-27T00:13:30.912 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.913 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.914 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.915 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.916 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "mark_out_threshold": { 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mark_out_threshold", 2024-04-27T00:13:30.917 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.918 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.918 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.918 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2419200", 2024-04-27T00:13:30.918 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.918 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.918 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.918 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-27T00:13:30.918 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.918 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.918 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.918 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": { 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_name", 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "device_health_metrics", 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "retention_period": { 2024-04-27T00:13:30.919 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retention_period", 2024-04-27T00:13:30.920 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.920 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.920 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.920 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15552000", 2024-04-27T00:13:30.920 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.920 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.920 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.920 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain device health metrics", 2024-04-27T00:13:30.920 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.920 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.920 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_frequency": { 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_frequency", 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.921 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: "self_heal": { 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "self_heal", 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.922 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and check device health", 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.923 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_threshold": { 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_threshold", 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7257600", 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.924 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "diskprediction_local", 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.925 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.926 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.928 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.929 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "predict_interval": { 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predict_interval", 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.930 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "predictor_model": { 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predictor_model", 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "prophetstor", 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "influx", 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": false, 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "influxdb python module not found", 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: "batch_size": { 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "batch_size", 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5000", 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "database": { 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "database", 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "ceph", 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": { 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hostname", 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server hostname", 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "30", 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "5", 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.936 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.937 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.938 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.939 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.940 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "password": { 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "password", 2024-04-27T00:13:30.941 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "password of InfluxDB server user", 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "port": { 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "port", 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.942 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8086", 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server port", 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.943 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "false", 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "threads": { 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threads", 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-27T00:13:30.944 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "32", 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "username": { 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "username", 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "username of InfluxDB server user", 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "verify_ssl": { 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "verify_ssl", 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "true", 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.946 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "insights", 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.947 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.948 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.951 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.952 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "iostat", 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.953 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.954 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.955 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.955 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.955 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.955 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.957 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.958 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.959 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "k8sevents", 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_event_retention_days": { 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ceph_event_retention_days", 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7", 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.961 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: "config_check_secs": { 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_check_secs", 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "10", 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-27T00:13:30.962 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.963 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.963 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.963 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.963 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.963 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.963 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.963 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.963 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.963 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.963 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.963 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.964 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.964 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.964 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.964 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.964 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.964 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.964 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.964 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.964 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.964 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.965 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.965 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.965 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.965 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.965 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.965 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.965 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.965 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.965 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.965 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.965 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.966 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.966 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.966 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.966 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.966 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.966 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.966 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.966 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.966 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.967 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.967 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.967 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.967 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.967 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.967 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.967 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.967 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.967 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.967 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.968 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "localpool", 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: "failure_domain": { 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "failure_domain", 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.969 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "host", 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "failure domain for any created local pool", 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.970 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.972 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.973 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.974 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": { 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_size", 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default min_size for any created local pool", 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.975 INFO:teuthology.orchestra.run.smithi040.stdout: "num_rep": { 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "num_rep", 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3", 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default replica count for any created local pool", 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": { 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_num", 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:30.976 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "128", 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default pg_num for any created local pool", 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "prefix": { 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prefix", 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.977 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name prefix for any created local pool", 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "subtree": { 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "subtree", 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.978 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "rack", 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mds_autoscaler", 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.979 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.980 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.981 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.982 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.983 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirroring", 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.984 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.985 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.986 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.987 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.988 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "nfs", 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.989 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.990 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.991 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.992 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.993 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: "fail_fs": { 2024-04-27T00:13:30.994 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "fail_fs", 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.995 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:30.997 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:30.998 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:30.999 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator": { 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm", 2024-04-27T00:13:31.000 INFO:teuthology.orchestra.run.smithi040.stdout: "rook", 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: "test_orchestrator" 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Orchestrator backend", 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_perf_query", 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.001 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.002 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.003 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.004 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.006 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_support", 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.007 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.008 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.009 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.010 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.011 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_autoscaler", 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.012 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.013 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.014 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.015 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.017 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "threshold": { 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threshold", 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.018 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1.0", 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "scaling threshold", 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "progress", 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.019 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_pg_recovery_event": { 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_pg_recovery_event", 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-04-27T00:13:31.020 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.021 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.022 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.023 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.024 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.025 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "max_completed_events": { 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_completed_events", 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "50", 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.026 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "number of past completed events to remember", 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.027 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long the module is going to sleep", 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus", 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: "cache": { 2024-04-27T00:13:31.028 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cache", 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "exclude_perf_counters": { 2024-04-27T00:13:31.029 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "exclude_perf_counters", 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.030 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.031 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.033 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.034 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools": { 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools", 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.035 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.036 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "300", 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_interval": { 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_interval", 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15.0", 2024-04-27T00:13:31.037 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-04-27T00:13:31.038 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "9283", 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.039 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "stale_cache_strategy": { 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stale_cache_strategy", 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "log", 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.040 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "default", 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "default", 2024-04-27T00:13:31.041 INFO:teuthology.orchestra.run.smithi040.stdout: "error" 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-04-27T00:13:31.042 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_support", 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.043 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.044 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.045 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.046 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.047 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_snap_create": { 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_snap_create", 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.048 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "mirror_snapshot_schedule": { 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirror_snapshot_schedule", 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.049 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "trash_purge_schedule": { 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "trash_purge_schedule", 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.050 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "restful", 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_auth": { 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_auth", 2024-04-27T00:13:31.051 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.052 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.053 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.054 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.055 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.056 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.057 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.058 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.059 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rgw", 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.060 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.061 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.062 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.063 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.064 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rook", 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.065 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.067 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.068 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.069 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "storage_class": { 2024-04-27T00:13:31.070 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "storage_class", 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "local", 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "selftest", 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.072 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.073 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.075 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "roption1": { 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption1", 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.076 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "roption2": { 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption2", 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "xyz", 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.077 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption1": { 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption1", 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.078 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption2": { 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption2", 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.079 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption3": { 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption3", 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.080 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption4": { 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption4", 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.081 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption5": { 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption5", 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.082 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption6": { 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption6", 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.083 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption7": { 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption7", 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "42", 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "testkey": { 2024-04-27T00:13:31.084 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testkey", 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "testlkey": { 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testlkey", 2024-04-27T00:13:31.085 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "testnewline": { 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testnewline", 2024-04-27T00:13:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snap_schedule", 2024-04-27T00:13:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_m_granularity": { 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_m_granularity", 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow minute scheduled snapshots", 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.088 INFO:teuthology.orchestra.run.smithi040.stdout: "dump_on_update": { 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dump_on_update", 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "dump database to debug log on update", 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.089 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.090 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.091 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.092 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.093 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stats", 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.094 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.095 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.096 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.097 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.098 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "status", 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.099 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.100 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.101 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.102 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.103 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telegraf", 2024-04-27T00:13:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "address": { 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "address", 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15", 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.107 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.108 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.109 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.110 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.111 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telemetry", 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_basic": { 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_basic", 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-27T00:13:31.112 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_crash": { 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_crash", 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.113 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_device": { 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_device", 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-27T00:13:31.114 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_ident": { 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_ident", 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.115 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_perf": { 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_perf", 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.116 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "contact": { 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "contact", 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "description": { 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "description", 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.118 INFO:teuthology.orchestra.run.smithi040.stdout: "device_url": { 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_url", 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.119 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.120 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "24", 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "8", 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "last_opt_revision": { 2024-04-27T00:13:31.121 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "last_opt_revision", 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1", 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard": { 2024-04-27T00:13:31.122 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard", 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard_description": { 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard_description", 2024-04-27T00:13:31.123 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.124 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.125 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.126 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.127 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.128 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "organization": { 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "organization", 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.129 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: "proxy": { 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "proxy", 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.130 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "url": { 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url", 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.131 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "test_orchestrator", 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.132 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.133 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.134 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.135 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.136 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.137 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.138 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.140 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.141 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.142 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_clones": { 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_clones", 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4", 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.143 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "periodic_async_work": { 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "periodic_async_work", 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.144 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Periodically check for async work", 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_delay": { 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_delay", 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.145 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_no_wait": { 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_no_wait", 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.146 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix", 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: "discovery_interval": { 2024-04-27T00:13:31.147 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "discovery_interval", 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "100", 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "identifier": { 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "identifier", 2024-04-27T00:13:31.148 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-04-27T00:13:31.149 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.150 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.151 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.152 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-04-27T00:13:31.153 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-04-27T00:13:31.154 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-04-27T00:13:31.155 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.155 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.155 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.155 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.155 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.155 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.155 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.156 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_host": { 2024-04-27T00:13:31.156 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_host", 2024-04-27T00:13:31.156 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.156 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.156 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.156 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-04-27T00:13:31.156 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.157 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.157 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.157 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.157 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.157 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.157 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.157 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_port": { 2024-04-27T00:13:31.158 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_port", 2024-04-27T00:13:31.158 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-04-27T00:13:31.158 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.158 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.158 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10051", 2024-04-27T00:13:31.158 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.158 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.158 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.159 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.159 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.159 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.159 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.159 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_sender": { 2024-04-27T00:13:31.159 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_sender", 2024-04-27T00:13:31.160 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-04-27T00:13:31.160 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-04-27T00:13:31.160 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-04-27T00:13:31.160 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-27T00:13:31.160 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-04-27T00:13:31.160 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-04-27T00:13:31.160 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-04-27T00:13:31.160 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-04-27T00:13:31.161 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-04-27T00:13:31.161 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-04-27T00:13:31.161 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-04-27T00:13:31.161 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.161 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.161 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.161 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.161 INFO:teuthology.orchestra.run.smithi040.stdout: "services": { 2024-04-27T00:13:31.162 INFO:teuthology.orchestra.run.smithi040.stdout: "dashboard": "https://172.21.15.40:8443/" 2024-04-27T00:13:31.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.162 INFO:teuthology.orchestra.run.smithi040.stdout: "always_on_modules": { 2024-04-27T00:13:31.162 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus": [ 2024-04-27T00:13:31.162 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-04-27T00:13:31.162 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-04-27T00:13:31.162 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-04-27T00:13:31.162 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-04-27T00:13:31.163 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-04-27T00:13:31.163 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-04-27T00:13:31.163 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-04-27T00:13:31.163 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-04-27T00:13:31.163 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-04-27T00:13:31.163 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-04-27T00:13:31.163 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.163 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific": [ 2024-04-27T00:13:31.164 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-04-27T00:13:31.164 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-04-27T00:13:31.164 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-04-27T00:13:31.164 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-04-27T00:13:31.164 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-04-27T00:13:31.164 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-04-27T00:13:31.164 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-04-27T00:13:31.164 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-04-27T00:13:31.165 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-04-27T00:13:31.165 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-04-27T00:13:31.165 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.165 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy": [ 2024-04-27T00:13:31.165 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-04-27T00:13:31.165 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-04-27T00:13:31.165 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-04-27T00:13:31.166 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-04-27T00:13:31.166 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-04-27T00:13:31.166 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-04-27T00:13:31.166 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-04-27T00:13:31.166 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-04-27T00:13:31.166 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-04-27T00:13:31.166 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-04-27T00:13:31.166 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.166 INFO:teuthology.orchestra.run.smithi040.stdout: "reef": [ 2024-04-27T00:13:31.166 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-04-27T00:13:31.166 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-04-27T00:13:31.166 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "squid": [ 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-04-27T00:13:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: "last_failure_osd_epoch": 3, 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: "active_clients": [ 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "devicehealth", 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2940103588 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.168 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "libcephsqlite", 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3266558063 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_support", 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-27T00:13:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2451356116 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2409110486 2024-04-27T00:13:31.170 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3115468040 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-04-27T00:13:31.171 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-27T00:13:31.172 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:30 smithi103 ceph-mon[32952]: pgmap v766: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-27T00:13:31.172 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:30 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2814249440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:13:31.172 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:30 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/4220327487' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-27T00:13:31.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:30 smithi040 ceph-mon[28095]: pgmap v766: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-27T00:13:31.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:30 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2814249440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:13:31.172 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:30 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/4220327487' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-27T00:13:31.178 DEBUG:tasks.check_counter:Getting stats from a 2024-04-27T00:13:31.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-27T00:13:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:13:32.466 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:32 smithi040 ceph-mon[28095]: pgmap v767: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:13:32.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:32 smithi080 ceph-mon[32362]: pgmap v767: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:13:32.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:32 smithi103 ceph-mon[32952]: pgmap v767: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-27T00:13:33.957 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-27T00:13:33.957 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-27T00:13:33.957 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 244, 2024-04-27T00:13:33.958 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 240, 2024-04-27T00:13:33.958 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 350816, 2024-04-27T00:13:33.958 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 147552, 2024-04-27T00:13:33.958 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-04-27T00:13:33.958 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2024-04-27T00:13:33.958 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.090991944, 2024-04-27T00:13:33.958 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.037301762, 2024-04-27T00:13:33.958 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.039024297, 2024-04-27T00:13:33.958 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.011290193, 2024-04-27T00:13:33.958 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:33.958 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 240, 2024-04-27T00:13:33.958 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.026440226, 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000110167 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 350816, 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 147552 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1, 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 182, 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 160, 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 110496, 2024-04-27T00:13:33.959 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 3, 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 2, 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.035334055, 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.028333052, 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.002669913, 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 180, 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.018954572, 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000105303 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:33.960 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 110496 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 0, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 0, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 0, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.001009034, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000000000, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.000843274, 2024-04-27T00:13:33.961 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-04-27T00:13:33.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-27T00:13:33.963 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-27T00:13:33.964 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 149001, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 31, 2024-04-27T00:13:33.965 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-27T00:13:33.966 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-27T00:13:33.966 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-27T00:13:33.966 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-27T00:13:33.966 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-27T00:13:33.966 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-27T00:13:33.966 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-27T00:13:33.966 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-27T00:13:33.967 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2024-04-27T00:13:33.967 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2024-04-27T00:13:33.967 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-27T00:13:33.967 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-27T00:13:33.967 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-27T00:13:33.967 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-27T00:13:33.967 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2024-04-27T00:13:33.967 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2024-04-27T00:13:33.967 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-27T00:13:33.967 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-27T00:13:33.967 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-27T00:13:33.967 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 241, 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 313881, 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 241, 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 241, 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 313881, 2024-04-27T00:13:33.968 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-27T00:13:33.969 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:33.969 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:33.969 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:33.969 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:33.969 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:33.969 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-27T00:13:34.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:34 smithi040 ceph-mon[28095]: pgmap v768: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 KiB/s wr, 10 op/s 2024-04-27T00:13:34.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:34 smithi080 ceph-mon[32362]: pgmap v768: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 KiB/s wr, 10 op/s 2024-04-27T00:13:34.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:34 smithi103 ceph-mon[32952]: pgmap v768: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 KiB/s wr, 10 op/s 2024-04-27T00:13:34.780 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-27T00:13:34.780 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-27T00:13:34.780 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-04-27T00:13:34.781 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-04-27T00:13:34.781 DEBUG:tasks.check_counter:Getting stats from d 2024-04-27T00:13:34.781 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-27T00:13:35.274 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:13:36.407 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:36 smithi040 ceph-mon[28095]: pgmap v769: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-04-27T00:13:36.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:36 smithi080 ceph-mon[32362]: pgmap v769: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-04-27T00:13:36.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:36 smithi103 ceph-mon[32952]: pgmap v769: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-04-27T00:13:37.316 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-27T00:13:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-27T00:13:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 0, 2024-04-27T00:13:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2, 2024-04-27T00:13:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 0, 2024-04-27T00:13:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 0, 2024-04-27T00:13:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-04-27T00:13:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2024-04-27T00:13:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.003849376, 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000000000, 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.000651279, 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:37.317 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 251, 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 247, 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 353344, 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 151728, 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.076063255, 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.026132743, 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.037393533, 2024-04-27T00:13:37.318 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.011871709, 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 247, 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.025596407, 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000103629 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 353344, 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 151728 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:37.319 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1, 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 187, 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 160, 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 111376, 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 3, 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 2, 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.038350478, 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.029857231, 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.003937658, 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 185, 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.019605230, 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000105974 2024-04-27T00:13:37.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 111376 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-27T00:13:37.321 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-27T00:13:37.322 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-27T00:13:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-27T00:13:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-27T00:13:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-27T00:13:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-27T00:13:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-27T00:13:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-27T00:13:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-27T00:13:37.323 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 148967, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 29, 2024-04-27T00:13:37.324 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-27T00:13:37.325 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 248, 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 315428, 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 248, 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-27T00:13:37.326 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 248, 2024-04-27T00:13:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 315428, 2024-04-27T00:13:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-27T00:13:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:37.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:37.327 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-27T00:13:38.192 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-27T00:13:38.192 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-27T00:13:38.192 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-27T00:13:38.192 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-27T00:13:38.192 DEBUG:tasks.check_counter:Getting stats from g 2024-04-27T00:13:38.193 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-27T00:13:38.465 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:38 smithi040 ceph-mon[28095]: pgmap v770: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.6 KiB/s wr, 11 op/s 2024-04-27T00:13:38.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:38 smithi080 ceph-mon[32362]: pgmap v770: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.6 KiB/s wr, 11 op/s 2024-04-27T00:13:38.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:38 smithi103 ceph-mon[32952]: pgmap v770: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.6 KiB/s wr, 11 op/s 2024-04-27T00:13:38.655 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:13:40.424 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:40 smithi040 ceph-mon[28095]: pgmap v771: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.3 KiB/s wr, 7 op/s 2024-04-27T00:13:40.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:40 smithi080 ceph-mon[32362]: pgmap v771: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.3 KiB/s wr, 7 op/s 2024-04-27T00:13:40.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:40 smithi103 ceph-mon[32952]: pgmap v771: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.3 KiB/s wr, 7 op/s 2024-04-27T00:13:40.975 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-27T00:13:40.975 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-27T00:13:40.975 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1224, 2024-04-27T00:13:40.975 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 854, 2024-04-27T00:13:40.975 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 294724, 2024-04-27T00:13:40.975 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 308231, 2024-04-27T00:13:40.975 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 9, 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.204844191, 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.090955886, 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.130252112, 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.008058987, 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 850, 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.149799715, 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000176234 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1995, 2024-04-27T00:13:40.976 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.001049185, 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000525 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 304, 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 297, 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 387820, 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 161770, 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 7, 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.090436784, 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.030853225, 2024-04-27T00:13:40.977 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.050313696, 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.012775020, 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 296, 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.048300066, 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000163175 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 22, 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000008040, 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000365 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 385024, 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 156752 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.978 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 74, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 253, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 26258, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 561938, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 12, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 7, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.059237367, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.037011820, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.017988224, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000231859, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 254, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.041834390, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000164702 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.979 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 116, 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000083160, 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000716 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 485184 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 33, 2024-04-27T00:13:40.980 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.003894734, 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000118022 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000010498, 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000005249 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 0, 2024-04-27T00:13:40.981 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 0, 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 11, 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 16, 2024-04-27T00:13:40.982 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 0, 2024-04-27T00:13:40.983 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 16, 2024-04-27T00:13:40.983 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-04-27T00:13:40.983 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 13, 2024-04-27T00:13:40.983 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 4, 2024-04-27T00:13:40.983 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 0, 2024-04-27T00:13:40.983 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 0, 2024-04-27T00:13:40.983 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 3, 2024-04-27T00:13:40.983 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 7, 2024-04-27T00:13:40.983 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 4, 2024-04-27T00:13:40.983 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-04-27T00:13:40.983 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 3, 2024-04-27T00:13:40.984 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-04-27T00:13:40.984 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-04-27T00:13:40.984 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-04-27T00:13:40.984 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-04-27T00:13:40.984 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-04-27T00:13:40.984 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 0, 2024-04-27T00:13:40.984 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 0, 2024-04-27T00:13:40.984 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 0, 2024-04-27T00:13:40.984 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2024-04-27T00:13:40.984 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 5, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 298, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-04-27T00:13:40.985 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 0, 2024-04-27T00:13:40.986 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 0, 2024-04-27T00:13:40.986 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-04-27T00:13:40.986 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-27T00:13:40.986 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-27T00:13:40.986 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 0, 2024-04-27T00:13:40.986 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 0, 2024-04-27T00:13:40.986 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 0, 2024-04-27T00:13:40.986 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 0 2024-04-27T00:13:40.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.986 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-04-27T00:13:40.986 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 11, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 7, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 8, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 0, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-04-27T00:13:40.987 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-04-27T00:13:40.988 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-04-27T00:13:40.988 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_path": 0, 2024-04-27T00:13:40.988 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_inode": 0, 2024-04-27T00:13:40.988 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-04-27T00:13:40.988 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 0, 2024-04-27T00:13:40.988 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-04-27T00:13:40.988 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2024-04-27T00:13:40.988 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 3, 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 3, 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 2, 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-04-27T00:13:40.989 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 2, 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 2, 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 4194304, 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 4195283, 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.003662200, 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001831100 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.990 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-04-27T00:13:40.991 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 20, 2024-04-27T00:13:40.991 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 22, 2024-04-27T00:13:40.991 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 2, 2024-04-27T00:13:40.991 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 19, 2024-04-27T00:13:40.991 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 23, 2024-04-27T00:13:40.991 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 4, 2024-04-27T00:13:40.991 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 16, 2024-04-27T00:13:40.991 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 20, 2024-04-27T00:13:40.991 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 4, 2024-04-27T00:13:40.991 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 0, 2024-04-27T00:13:40.991 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 0, 2024-04-27T00:13:40.991 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 0, 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 50816, 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 223608 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 0, 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 0, 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 11, 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.993 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.994 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-04-27T00:13:40.995 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.996 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.996 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.996 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-04-27T00:13:40.996 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.996 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.996 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.996 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-04-27T00:13:40.996 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.997 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.997 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.997 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-04-27T00:13:40.997 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.997 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.997 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.997 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-04-27T00:13:40.997 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.997 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.998 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:40.999 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-04-27T00:13:41.000 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.001 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-04-27T00:13:41.002 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 1, 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 1, 2024-04-27T00:13:41.003 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 0, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 1, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 0, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 0, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 851, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-27T00:13:41.004 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-27T00:13:41.005 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 238601, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 82, 2024-04-27T00:13:41.006 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 8896, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 171, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-27T00:13:41.007 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 80901, 2024-04-27T00:13:41.008 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 319, 2024-04-27T00:13:41.008 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-27T00:13:41.008 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-27T00:13:41.008 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-27T00:13:41.008 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-27T00:13:41.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.008 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-04-27T00:13:41.008 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-04-27T00:13:41.008 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 36, 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 11101, 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 28, 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 36, 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 36, 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.271996529, 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.007555459 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 36, 2024-04-27T00:13:41.009 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 58 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 36, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 0, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 36, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 0, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 10, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 0, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 2, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 5, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-04-27T00:13:41.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 6, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 20, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-04-27T00:13:41.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-04-27T00:13:41.012 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 94, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 4, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 9, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 9, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 14, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 0, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 1 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-04-27T00:13:41.013 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 0, 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-04-27T00:13:41.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 1598, 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 547077, 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 1598, 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 1598, 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 547077, 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-27T00:13:41.015 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:41.016 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 36, 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 11101, 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 36, 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 11101, 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-27T00:13:41.017 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 36, 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 36, 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 36, 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 36, 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:41.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.019 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x55e7df67e500": { 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 3, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 979, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 3, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 979, 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-27T00:13:41.020 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:41.021 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:41.021 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:41.021 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:41.021 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:41.021 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-27T00:13:41.905 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-27T00:13:41.905 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-27T00:13:41.906 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-04-27T00:13:41.906 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=11 2024-04-27T00:13:41.906 DEBUG:tasks.check_counter:Getting stats from j 2024-04-27T00:13:41.906 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-27T00:13:42.362 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T00:13:42.519 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:42 smithi040 ceph-mon[28095]: pgmap v772: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.3 KiB/s wr, 8 op/s 2024-04-27T00:13:42.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:42 smithi080 ceph-mon[32362]: pgmap v772: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.3 KiB/s wr, 8 op/s 2024-04-27T00:13:42.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:42 smithi103 ceph-mon[32952]: pgmap v772: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.3 KiB/s wr, 8 op/s 2024-04-27T00:13:44.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:44 smithi040 ceph-mon[28095]: pgmap v773: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-04-27T00:13:44.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:44 smithi080 ceph-mon[32362]: pgmap v773: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-04-27T00:13:44.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:44 smithi103 ceph-mon[32952]: pgmap v773: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-04-27T00:13:44.635 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-04-27T00:13:44.635 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-04-27T00:13:44.635 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 250, 2024-04-27T00:13:44.635 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 248, 2024-04-27T00:13:44.635 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 351168, 2024-04-27T00:13:44.635 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 153200, 2024-04-27T00:13:44.635 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-04-27T00:13:44.635 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2024-04-27T00:13:44.635 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.095834562, 2024-04-27T00:13:44.635 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.040204378, 2024-04-27T00:13:44.635 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.039806471, 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.012260834, 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 247, 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.024603949, 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000099611 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 351168, 2024-04-27T00:13:44.636 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 153200 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2, 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 190, 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 320, 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 190720, 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 2, 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.039786292, 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.030892190, 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.004170499, 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:44.637 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 188, 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.022416358, 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000119235 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 190720 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-04-27T00:13:44.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 4, 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 4, 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2992, 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 448, 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.001643586, 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000071307, 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.001464105, 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3, 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.001597776, 2024-04-27T00:13:44.639 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000532592 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 448 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-04-27T00:13:44.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-27T00:13:44.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-04-27T00:13:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 149982, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 29, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-04-27T00:13:44.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-04-27T00:13:44.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 252, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 315845, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 252, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 252, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 315845, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-04-27T00:13:44.645 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-04-27T00:13:44.646 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-04-27T00:13:44.646 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:44.646 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-04-27T00:13:44.646 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-04-27T00:13:45.488 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-27T00:13:45.488 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-27T00:13:45.488 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-27T00:13:45.488 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-27T00:13:45.488 DEBUG:tasks.check_counter:Getting stats from b 2024-04-27T00:13:45.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-27T00:13:45.748 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T00:13:46.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:46 smithi080 ceph-mon[32362]: pgmap v774: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-27T00:13:46.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:46 smithi103 ceph-mon[32952]: pgmap v774: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-27T00:13:46.599 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:46 smithi040 ceph-mon[28095]: pgmap v774: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-27T00:13:47.537 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-04-27T00:13:47.537 INFO:teuthology.orchestra.run.smithi080.stdout: "AsyncMessenger::Worker-0": { 2024-04-27T00:13:47.538 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_messages": 3688, 2024-04-27T00:13:47.538 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages": 4060, 2024-04-27T00:13:47.538 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_bytes": 1003155, 2024-04-27T00:13:47.538 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_bytes": 10460282, 2024-04-27T00:13:47.538 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_created_connections": 112, 2024-04-27T00:13:47.538 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_active_connections": 2, 2024-04-27T00:13:47.538 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_total_time": 0.384673486, 2024-04-27T00:13:47.538 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_send_time": 0.133805146, 2024-04-27T00:13:47.538 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_recv_time": 0.123256103, 2024-04-27T00:13:47.539 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_fast_dispatch_time": 0.096158586, 2024-04-27T00:13:47.539 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:47.539 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 4062, 2024-04-27T00:13:47.539 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.172214811, 2024-04-27T00:13:47.539 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000042396 2024-04-27T00:13:47.539 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.539 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:47.539 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 1870, 2024-04-27T00:13:47.539 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000438528, 2024-04-27T00:13:47.539 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000234 2024-04-27T00:13:47.540 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.540 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-27T00:13:47.540 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_encrypted_bytes": 448 2024-04-27T00:13:47.540 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.540 INFO:teuthology.orchestra.run.smithi080.stdout: "AsyncMessenger::Worker-1": { 2024-04-27T00:13:47.540 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_messages": 1904, 2024-04-27T00:13:47.540 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages": 2270, 2024-04-27T00:13:47.540 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_bytes": 1020733, 2024-04-27T00:13:47.540 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_bytes": 808340, 2024-04-27T00:13:47.541 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_created_connections": 7, 2024-04-27T00:13:47.541 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_active_connections": 5, 2024-04-27T00:13:47.541 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_total_time": 0.208082493, 2024-04-27T00:13:47.541 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_send_time": 0.067805843, 2024-04-27T00:13:47.541 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_recv_time": 0.077613028, 2024-04-27T00:13:47.541 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_fast_dispatch_time": 0.054250876, 2024-04-27T00:13:47.541 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:47.542 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 2272, 2024-04-27T00:13:47.542 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.264448720, 2024-04-27T00:13:47.542 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000116394 2024-04-27T00:13:47.542 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.542 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:47.542 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 1933, 2024-04-27T00:13:47.542 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000461354, 2024-04-27T00:13:47.542 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000238 2024-04-27T00:13:47.543 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.543 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_encrypted_bytes": 395824, 2024-04-27T00:13:47.543 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_encrypted_bytes": 157216 2024-04-27T00:13:47.543 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.543 INFO:teuthology.orchestra.run.smithi080.stdout: "AsyncMessenger::Worker-2": { 2024-04-27T00:13:47.543 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_messages": 3223, 2024-04-27T00:13:47.543 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages": 4247, 2024-04-27T00:13:47.543 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_bytes": 1351168, 2024-04-27T00:13:47.543 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_bytes": 2281970, 2024-04-27T00:13:47.543 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_created_connections": 17, 2024-04-27T00:13:47.543 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_active_connections": 7, 2024-04-27T00:13:47.544 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_total_time": 0.463191887, 2024-04-27T00:13:47.544 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_send_time": 0.161295641, 2024-04-27T00:13:47.544 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_recv_time": 0.201353540, 2024-04-27T00:13:47.544 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_fast_dispatch_time": 0.111197246, 2024-04-27T00:13:47.544 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:47.544 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 4239, 2024-04-27T00:13:47.544 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.173481165, 2024-04-27T00:13:47.544 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000040925 2024-04-27T00:13:47.544 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.544 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:47.544 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 5427, 2024-04-27T00:13:47.544 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.001181581, 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000217 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_encrypted_bytes": 1273680, 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_encrypted_bytes": 1993344 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: "cct": { 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: "total_workers": 1, 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: "unhealthy_workers": 0 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: "finisher-MDSRank": { 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: "queue_len": 0, 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: "complete_latency": { 2024-04-27T00:13:47.545 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 6077, 2024-04-27T00:13:47.546 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 1.749821141, 2024-04-27T00:13:47.546 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000287941 2024-04-27T00:13:47.546 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:47.546 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.546 INFO:teuthology.orchestra.run.smithi080.stdout: "finisher-PurgeQueue": { 2024-04-27T00:13:47.546 INFO:teuthology.orchestra.run.smithi080.stdout: "queue_len": 0, 2024-04-27T00:13:47.546 INFO:teuthology.orchestra.run.smithi080.stdout: "complete_latency": { 2024-04-27T00:13:47.546 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 18, 2024-04-27T00:13:47.546 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000602648, 2024-04-27T00:13:47.546 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000033480 2024-04-27T00:13:47.546 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:47.547 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.547 INFO:teuthology.orchestra.run.smithi080.stdout: "mds": { 2024-04-27T00:13:47.547 INFO:teuthology.orchestra.run.smithi080.stdout: "request": 658, 2024-04-27T00:13:47.547 INFO:teuthology.orchestra.run.smithi080.stdout: "reply": 658, 2024-04-27T00:13:47.547 INFO:teuthology.orchestra.run.smithi080.stdout: "reply_latency": { 2024-04-27T00:13:47.547 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 658, 2024-04-27T00:13:47.547 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 3.386380189, 2024-04-27T00:13:47.547 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.005146474 2024-04-27T00:13:47.547 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.547 INFO:teuthology.orchestra.run.smithi080.stdout: "slow_reply": 0, 2024-04-27T00:13:47.547 INFO:teuthology.orchestra.run.smithi080.stdout: "forward": 0, 2024-04-27T00:13:47.547 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_fetch_complete": 0, 2024-04-27T00:13:47.548 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_fetch_keys": 0, 2024-04-27T00:13:47.548 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_commit": 340, 2024-04-27T00:13:47.548 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_split": 0, 2024-04-27T00:13:47.548 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_merge": 0, 2024-04-27T00:13:47.548 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes": 24, 2024-04-27T00:13:47.548 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes_top": 4, 2024-04-27T00:13:47.548 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes_bottom": 20, 2024-04-27T00:13:47.548 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes_pin_tail": 0, 2024-04-27T00:13:47.548 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes_pinned": 18, 2024-04-27T00:13:47.548 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes_expired": 0, 2024-04-27T00:13:47.548 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes_with_caps": 3, 2024-04-27T00:13:47.549 INFO:teuthology.orchestra.run.smithi080.stdout: "caps": 3, 2024-04-27T00:13:47.549 INFO:teuthology.orchestra.run.smithi080.stdout: "subtrees": 2, 2024-04-27T00:13:47.549 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse": 686, 2024-04-27T00:13:47.549 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse_hit": 678, 2024-04-27T00:13:47.549 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse_forward": 0, 2024-04-27T00:13:47.549 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse_discover": 0, 2024-04-27T00:13:47.549 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse_dir_fetch": 0, 2024-04-27T00:13:47.549 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse_remote_ino": 0, 2024-04-27T00:13:47.549 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse_lock": 0, 2024-04-27T00:13:47.549 INFO:teuthology.orchestra.run.smithi080.stdout: "load_cent": 316, 2024-04-27T00:13:47.549 INFO:teuthology.orchestra.run.smithi080.stdout: "q": 0, 2024-04-27T00:13:47.549 INFO:teuthology.orchestra.run.smithi080.stdout: "exported": 0, 2024-04-27T00:13:47.550 INFO:teuthology.orchestra.run.smithi080.stdout: "exported_inodes": 0, 2024-04-27T00:13:47.550 INFO:teuthology.orchestra.run.smithi080.stdout: "imported": 0, 2024-04-27T00:13:47.550 INFO:teuthology.orchestra.run.smithi080.stdout: "imported_inodes": 0, 2024-04-27T00:13:47.550 INFO:teuthology.orchestra.run.smithi080.stdout: "openino_dir_fetch": 0, 2024-04-27T00:13:47.550 INFO:teuthology.orchestra.run.smithi080.stdout: "openino_backtrace_fetch": 0, 2024-04-27T00:13:47.550 INFO:teuthology.orchestra.run.smithi080.stdout: "openino_peer_discover": 0, 2024-04-27T00:13:47.550 INFO:teuthology.orchestra.run.smithi080.stdout: "root_rfiles": 4, 2024-04-27T00:13:47.550 INFO:teuthology.orchestra.run.smithi080.stdout: "root_rbytes": 234, 2024-04-27T00:13:47.550 INFO:teuthology.orchestra.run.smithi080.stdout: "root_rsnaps": 0, 2024-04-27T00:13:47.550 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_backtrace_fetch": 0, 2024-04-27T00:13:47.550 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_set_tag": 0, 2024-04-27T00:13:47.550 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_backtrace_repaired": 0, 2024-04-27T00:13:47.551 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_inotable_repaired": 0, 2024-04-27T00:13:47.551 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_dir_inodes": 0, 2024-04-27T00:13:47.551 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_dir_base_inodes": 0, 2024-04-27T00:13:47.551 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_dirfrag_rstats": 0, 2024-04-27T00:13:47.551 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_file_inodes": 0, 2024-04-27T00:13:47.551 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_inode_file_caps": 0, 2024-04-27T00:13:47.551 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_cap_op_revoke": 611, 2024-04-27T00:13:47.551 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_cap_op_grant": 612, 2024-04-27T00:13:47.551 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_cap_op_trunc": 599, 2024-04-27T00:13:47.551 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-27T00:13:47.551 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-27T00:13:47.551 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_client_caps": 719, 2024-04-27T00:13:47.552 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_client_caps_dirty": 704, 2024-04-27T00:13:47.552 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_client_cap_release": 2, 2024-04-27T00:13:47.552 INFO:teuthology.orchestra.run.smithi080.stdout: "process_request_cap_release": 622 2024-04-27T00:13:47.552 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.552 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_cache": { 2024-04-27T00:13:47.552 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_update": 7, 2024-04-27T00:13:47.552 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_update_receipt": 0, 2024-04-27T00:13:47.552 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_try_discover": 0, 2024-04-27T00:13:47.552 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_send_discover": 0, 2024-04-27T00:13:47.552 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_handle_discover": 15, 2024-04-27T00:13:47.552 INFO:teuthology.orchestra.run.smithi080.stdout: "num_strays": 0, 2024-04-27T00:13:47.552 INFO:teuthology.orchestra.run.smithi080.stdout: "num_strays_delayed": 0, 2024-04-27T00:13:47.553 INFO:teuthology.orchestra.run.smithi080.stdout: "num_strays_enqueuing": 0, 2024-04-27T00:13:47.553 INFO:teuthology.orchestra.run.smithi080.stdout: "strays_created": 3, 2024-04-27T00:13:47.553 INFO:teuthology.orchestra.run.smithi080.stdout: "strays_enqueued": 3, 2024-04-27T00:13:47.553 INFO:teuthology.orchestra.run.smithi080.stdout: "strays_reintegrated": 0, 2024-04-27T00:13:47.553 INFO:teuthology.orchestra.run.smithi080.stdout: "strays_migrated": 0, 2024-04-27T00:13:47.553 INFO:teuthology.orchestra.run.smithi080.stdout: "num_recovering_processing": 0, 2024-04-27T00:13:47.553 INFO:teuthology.orchestra.run.smithi080.stdout: "num_recovering_enqueued": 0, 2024-04-27T00:13:47.553 INFO:teuthology.orchestra.run.smithi080.stdout: "num_recovering_prioritized": 0, 2024-04-27T00:13:47.553 INFO:teuthology.orchestra.run.smithi080.stdout: "recovery_started": 0, 2024-04-27T00:13:47.553 INFO:teuthology.orchestra.run.smithi080.stdout: "recovery_completed": 0, 2024-04-27T00:13:47.553 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_quiesce_path": 0, 2024-04-27T00:13:47.553 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_quiesce_inode": 0, 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_enqueue_scrub": 0, 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_exportdir": 0, 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_flush": 0, 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_fragmentdir": 0, 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_fragstats": 0, 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_inodestats": 0 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log": { 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: "evlrg": 0, 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: "evadd": 2483, 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: "evex": 2432, 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: "evtrm": 2432, 2024-04-27T00:13:47.554 INFO:teuthology.orchestra.run.smithi080.stdout: "ev": 51, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "evexg": 0, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "evexd": 55, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "segadd": 55, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "segex": 54, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "segtrm": 54, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "seg": 1, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "segmjr": 1, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "segexg": 0, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "segexd": 1, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "expos": 11974663, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "wrpos": 12134137, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "rdpos": 0, 2024-04-27T00:13:47.555 INFO:teuthology.orchestra.run.smithi080.stdout: "jlat": { 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 2465, 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 5.573692957, 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.002261133 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: "replayed": 0 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_mem": { 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: "ino": 23, 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: "ino+": 26, 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: "ino-": 3, 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: "dir": 18, 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: "dir+": 20, 2024-04-27T00:13:47.556 INFO:teuthology.orchestra.run.smithi080.stdout: "dir-": 2, 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: "dn": 24, 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: "dn+": 29, 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: "dn-": 5, 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: "cap": 3, 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: "cap+": 16, 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: "cap-": 13, 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: "rss": 58016, 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: "heap": 231800 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_server": { 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: "dispatch_client_request": 1878, 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: "dispatch_server_request": 0, 2024-04-27T00:13:47.557 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_client_request": 658, 2024-04-27T00:13:47.558 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_client_session": 27, 2024-04-27T00:13:47.558 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_peer_request": 0, 2024-04-27T00:13:47.558 INFO:teuthology.orchestra.run.smithi080.stdout: "req_create_latency": { 2024-04-27T00:13:47.558 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 5, 2024-04-27T00:13:47.558 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.003820678, 2024-04-27T00:13:47.558 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000764135 2024-04-27T00:13:47.558 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.558 INFO:teuthology.orchestra.run.smithi080.stdout: "req_getattr_latency": { 2024-04-27T00:13:47.558 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 28, 2024-04-27T00:13:47.558 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.006998977, 2024-04-27T00:13:47.559 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000249963 2024-04-27T00:13:47.559 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.559 INFO:teuthology.orchestra.run.smithi080.stdout: "req_getfilelock_latency": { 2024-04-27T00:13:47.559 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.559 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.559 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.559 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.559 INFO:teuthology.orchestra.run.smithi080.stdout: "req_link_latency": { 2024-04-27T00:13:47.560 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.560 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.560 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.560 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.560 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lookup_latency": { 2024-04-27T00:13:47.560 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.560 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.560 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.560 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.560 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lookuphash_latency": { 2024-04-27T00:13:47.560 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.561 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.561 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.561 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lookupino_latency": { 2024-04-27T00:13:47.561 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.561 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.561 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.561 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.561 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lookupname_latency": { 2024-04-27T00:13:47.561 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.562 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.562 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.562 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.562 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lookupparent_latency": { 2024-04-27T00:13:47.562 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.562 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.562 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.562 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.563 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lookupsnap_latency": { 2024-04-27T00:13:47.563 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.563 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.563 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.563 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.563 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lssnap_latency": { 2024-04-27T00:13:47.563 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.563 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.564 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.564 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.564 INFO:teuthology.orchestra.run.smithi080.stdout: "req_mkdir_latency": { 2024-04-27T00:13:47.564 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 8, 2024-04-27T00:13:47.564 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.013282305, 2024-04-27T00:13:47.565 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.001660288 2024-04-27T00:13:47.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.565 INFO:teuthology.orchestra.run.smithi080.stdout: "req_mknod_latency": { 2024-04-27T00:13:47.565 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.565 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.565 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.565 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.565 INFO:teuthology.orchestra.run.smithi080.stdout: "req_mksnap_latency": { 2024-04-27T00:13:47.565 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.565 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.566 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.566 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.566 INFO:teuthology.orchestra.run.smithi080.stdout: "req_open_latency": { 2024-04-27T00:13:47.566 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 1, 2024-04-27T00:13:47.566 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000251726, 2024-04-27T00:13:47.566 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000251726 2024-04-27T00:13:47.566 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.566 INFO:teuthology.orchestra.run.smithi080.stdout: "req_readdir_latency": { 2024-04-27T00:13:47.566 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 2, 2024-04-27T00:13:47.567 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.004229494, 2024-04-27T00:13:47.567 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.002114747 2024-04-27T00:13:47.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.567 INFO:teuthology.orchestra.run.smithi080.stdout: "req_rename_latency": { 2024-04-27T00:13:47.567 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 2, 2024-04-27T00:13:47.567 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.002958570, 2024-04-27T00:13:47.567 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.001479285 2024-04-27T00:13:47.567 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.567 INFO:teuthology.orchestra.run.smithi080.stdout: "req_renamesnap_latency": { 2024-04-27T00:13:47.567 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.567 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.568 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.568 INFO:teuthology.orchestra.run.smithi080.stdout: "req_snapdiff_latency": { 2024-04-27T00:13:47.568 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.568 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.568 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.568 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.568 INFO:teuthology.orchestra.run.smithi080.stdout: "req_rmdir_latency": { 2024-04-27T00:13:47.568 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 2, 2024-04-27T00:13:47.568 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.013103976, 2024-04-27T00:13:47.568 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.006551988 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: "req_rmsnap_latency": { 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: "req_rmxattr_latency": { 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: "req_setattr_latency": { 2024-04-27T00:13:47.569 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 601, 2024-04-27T00:13:47.570 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 3.319046590, 2024-04-27T00:13:47.570 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.005522540 2024-04-27T00:13:47.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.570 INFO:teuthology.orchestra.run.smithi080.stdout: "req_setdirlayout_latency": { 2024-04-27T00:13:47.570 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.570 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.570 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.570 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.570 INFO:teuthology.orchestra.run.smithi080.stdout: "req_setfilelock_latency": { 2024-04-27T00:13:47.570 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.571 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.571 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.571 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.571 INFO:teuthology.orchestra.run.smithi080.stdout: "req_setlayout_latency": { 2024-04-27T00:13:47.571 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.571 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.571 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.571 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.571 INFO:teuthology.orchestra.run.smithi080.stdout: "req_setxattr_latency": { 2024-04-27T00:13:47.571 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 8, 2024-04-27T00:13:47.571 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.020898658, 2024-04-27T00:13:47.571 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.002612332 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: "req_symlink_latency": { 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: "req_unlink_latency": { 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 1, 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.001789215, 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.001789215 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: "cap_revoke_eviction": 0, 2024-04-27T00:13:47.572 INFO:teuthology.orchestra.run.smithi080.stdout: "cap_acquisition_throttle": 0, 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: "req_getvxattr_latency": { 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_sessions": { 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: "session_count": 2, 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: "session_add": 3, 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: "session_remove": 1, 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: "sessions_open": 2, 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: "sessions_stale": 0, 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: "total_load": 88, 2024-04-27T00:13:47.573 INFO:teuthology.orchestra.run.smithi080.stdout: "average_load": 44, 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: "avg_session_uptime": 788, 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: "mdthresh_evicted": 0 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: "mempool": { 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: "bloom_filter_bytes": 0, 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: "bloom_filter_items": 0, 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_alloc_bytes": 0, 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_alloc_items": 0, 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_data_bytes": 0, 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_data_items": 0, 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_onode_items": 0, 2024-04-27T00:13:47.574 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_meta_items": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_other_bytes": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_other_items": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_buffer_items": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_bytes": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_items": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_blob_bytes": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_blob_items": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_shared_blob_items": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-27T00:13:47.575 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_inline_bl_items": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_bytes": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_items": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_txc_bytes": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_txc_items": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_deferred_items": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_bytes": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_items": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_bytes": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_items": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_reader_bytes": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_reader_items": 0, 2024-04-27T00:13:47.576 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_writer_bytes": 0, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_writer_items": 0, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_anon_bytes": 268087, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_anon_items": 104, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_meta_bytes": 0, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_meta_items": 0, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_bytes": 0, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_items": 0, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mapbl_bytes": 0, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mapbl_items": 0, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pglog_bytes": 0, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pglog_items": 0, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_bytes": 8896, 2024-04-27T00:13:47.577 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_items": 171, 2024-04-27T00:13:47.578 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_mapping_bytes": 0, 2024-04-27T00:13:47.578 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_mapping_items": 0, 2024-04-27T00:13:47.578 INFO:teuthology.orchestra.run.smithi080.stdout: "pgmap_bytes": 0, 2024-04-27T00:13:47.578 INFO:teuthology.orchestra.run.smithi080.stdout: "pgmap_items": 0, 2024-04-27T00:13:47.578 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_co_bytes": 96082, 2024-04-27T00:13:47.578 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_co_items": 646, 2024-04-27T00:13:47.578 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_1_bytes": 0, 2024-04-27T00:13:47.578 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_1_items": 0, 2024-04-27T00:13:47.578 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_2_bytes": 0, 2024-04-27T00:13:47.578 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_2_items": 0 2024-04-27T00:13:47.578 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.579 INFO:teuthology.orchestra.run.smithi080.stdout: "objecter": { 2024-04-27T00:13:47.579 INFO:teuthology.orchestra.run.smithi080.stdout: "op_active": 0, 2024-04-27T00:13:47.579 INFO:teuthology.orchestra.run.smithi080.stdout: "op_laggy": 0, 2024-04-27T00:13:47.579 INFO:teuthology.orchestra.run.smithi080.stdout: "op_send": 3740, 2024-04-27T00:13:47.579 INFO:teuthology.orchestra.run.smithi080.stdout: "op_send_bytes": 8292903, 2024-04-27T00:13:47.579 INFO:teuthology.orchestra.run.smithi080.stdout: "op_resend": 37, 2024-04-27T00:13:47.579 INFO:teuthology.orchestra.run.smithi080.stdout: "op_reply": 3740, 2024-04-27T00:13:47.579 INFO:teuthology.orchestra.run.smithi080.stdout: "op_latency": { 2024-04-27T00:13:47.579 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 3740, 2024-04-27T00:13:47.579 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 11.267739143, 2024-04-27T00:13:47.580 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.003012764 2024-04-27T00:13:47.580 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.580 INFO:teuthology.orchestra.run.smithi080.stdout: "op_inflight": 0, 2024-04-27T00:13:47.580 INFO:teuthology.orchestra.run.smithi080.stdout: "oplen_avg": { 2024-04-27T00:13:47.580 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 3740, 2024-04-27T00:13:47.580 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 4467 2024-04-27T00:13:47.580 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.580 INFO:teuthology.orchestra.run.smithi080.stdout: "op": 3740, 2024-04-27T00:13:47.580 INFO:teuthology.orchestra.run.smithi080.stdout: "op_r": 1, 2024-04-27T00:13:47.580 INFO:teuthology.orchestra.run.smithi080.stdout: "op_w": 3739, 2024-04-27T00:13:47.580 INFO:teuthology.orchestra.run.smithi080.stdout: "op_rmw": 0, 2024-04-27T00:13:47.581 INFO:teuthology.orchestra.run.smithi080.stdout: "op_pg": 0, 2024-04-27T00:13:47.581 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_stat": 320, 2024-04-27T00:13:47.581 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_create": 30, 2024-04-27T00:13:47.581 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_read": 1, 2024-04-27T00:13:47.581 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_write": 2466, 2024-04-27T00:13:47.581 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_writefull": 168, 2024-04-27T00:13:47.581 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_writesame": 0, 2024-04-27T00:13:47.581 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_append": 0, 2024-04-27T00:13:47.581 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_zero": 0, 2024-04-27T00:13:47.581 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_truncate": 0, 2024-04-27T00:13:47.581 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_delete": 18, 2024-04-27T00:13:47.582 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_mapext": 0, 2024-04-27T00:13:47.582 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_sparse_read": 0, 2024-04-27T00:13:47.582 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_clonerange": 0, 2024-04-27T00:13:47.582 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_getxattr": 0, 2024-04-27T00:13:47.582 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_setxattr": 55, 2024-04-27T00:13:47.582 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_cmpxattr": 0, 2024-04-27T00:13:47.582 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_rmxattr": 0, 2024-04-27T00:13:47.582 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_resetxattrs": 0, 2024-04-27T00:13:47.582 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_call": 0, 2024-04-27T00:13:47.582 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_watch": 0, 2024-04-27T00:13:47.583 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_notify": 0, 2024-04-27T00:13:47.583 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_src_cmpxattr": 0, 2024-04-27T00:13:47.583 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_pgls": 0, 2024-04-27T00:13:47.583 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_pgls_filter": 0, 2024-04-27T00:13:47.583 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_other": 599, 2024-04-27T00:13:47.583 INFO:teuthology.orchestra.run.smithi080.stdout: "linger_active": 0, 2024-04-27T00:13:47.583 INFO:teuthology.orchestra.run.smithi080.stdout: "linger_send": 0, 2024-04-27T00:13:47.583 INFO:teuthology.orchestra.run.smithi080.stdout: "linger_resend": 0, 2024-04-27T00:13:47.583 INFO:teuthology.orchestra.run.smithi080.stdout: "linger_ping": 0, 2024-04-27T00:13:47.583 INFO:teuthology.orchestra.run.smithi080.stdout: "poolop_active": 0, 2024-04-27T00:13:47.583 INFO:teuthology.orchestra.run.smithi080.stdout: "poolop_send": 0, 2024-04-27T00:13:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: "poolop_resend": 0, 2024-04-27T00:13:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: "poolstat_active": 0, 2024-04-27T00:13:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: "poolstat_send": 0, 2024-04-27T00:13:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: "poolstat_resend": 0, 2024-04-27T00:13:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: "statfs_active": 0, 2024-04-27T00:13:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: "statfs_send": 0, 2024-04-27T00:13:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: "statfs_resend": 0, 2024-04-27T00:13:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: "command_active": 0, 2024-04-27T00:13:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: "command_send": 0, 2024-04-27T00:13:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: "command_resend": 0, 2024-04-27T00:13:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: "map_epoch": 94, 2024-04-27T00:13:47.584 INFO:teuthology.orchestra.run.smithi080.stdout: "map_full": 0, 2024-04-27T00:13:47.585 INFO:teuthology.orchestra.run.smithi080.stdout: "map_inc": 9, 2024-04-27T00:13:47.585 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_sessions": 9, 2024-04-27T00:13:47.585 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_session_open": 9, 2024-04-27T00:13:47.585 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_session_close": 0, 2024-04-27T00:13:47.585 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_laggy": 0, 2024-04-27T00:13:47.585 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_wr": 800, 2024-04-27T00:13:47.585 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_rd": 0, 2024-04-27T00:13:47.585 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_del": 10 2024-04-27T00:13:47.585 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.585 INFO:teuthology.orchestra.run.smithi080.stdout: "oft": { 2024-04-27T00:13:47.585 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_total_objs": 1, 2024-04-27T00:13:47.586 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_total_kv_pairs": 0, 2024-04-27T00:13:47.586 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_total_updates": 0, 2024-04-27T00:13:47.586 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_total_removes": 0 2024-04-27T00:13:47.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.586 INFO:teuthology.orchestra.run.smithi080.stdout: "osd-slow-ops": { 2024-04-27T00:13:47.586 INFO:teuthology.orchestra.run.smithi080.stdout: "slow_ops_count": 0 2024-04-27T00:13:47.586 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.586 INFO:teuthology.orchestra.run.smithi080.stdout: "purge_queue": { 2024-04-27T00:13:47.586 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_executing_ops": 0, 2024-04-27T00:13:47.586 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_executing_ops_high_water": 4, 2024-04-27T00:13:47.586 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_executing": 0, 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_executing_high_water": 3, 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_executed_ops": 4, 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_executed": 3, 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_item_in_journal": 0 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 0, 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "max": 104857600, 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "get_started": 0, 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "get": 8811, 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "get_sum": 2427623, 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_success": 8811, 2024-04-27T00:13:47.587 INFO:teuthology.orchestra.run.smithi080.stdout: "take": 0, 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: "take_sum": 0, 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: "put": 8811, 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: "put_sum": 2427623, 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: "wait": { 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: "throttle-objecter_bytes": { 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 0, 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: "max": 104857600, 2024-04-27T00:13:47.588 INFO:teuthology.orchestra.run.smithi080.stdout: "get_started": 0, 2024-04-27T00:13:47.589 INFO:teuthology.orchestra.run.smithi080.stdout: "get": 0, 2024-04-27T00:13:47.589 INFO:teuthology.orchestra.run.smithi080.stdout: "get_sum": 0, 2024-04-27T00:13:47.589 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:47.589 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:47.589 INFO:teuthology.orchestra.run.smithi080.stdout: "take": 3740, 2024-04-27T00:13:47.589 INFO:teuthology.orchestra.run.smithi080.stdout: "take_sum": 8293214, 2024-04-27T00:13:47.589 INFO:teuthology.orchestra.run.smithi080.stdout: "put": 3740, 2024-04-27T00:13:47.589 INFO:teuthology.orchestra.run.smithi080.stdout: "put_sum": 8293214, 2024-04-27T00:13:47.589 INFO:teuthology.orchestra.run.smithi080.stdout: "wait": { 2024-04-27T00:13:47.589 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.589 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.589 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: "throttle-objecter_ops": { 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 0, 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: "max": 1024, 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: "get_started": 0, 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: "get": 0, 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: "get_sum": 0, 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: "take": 3740, 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: "take_sum": 3740, 2024-04-27T00:13:47.590 INFO:teuthology.orchestra.run.smithi080.stdout: "put": 3740, 2024-04-27T00:13:47.591 INFO:teuthology.orchestra.run.smithi080.stdout: "put_sum": 3740, 2024-04-27T00:13:47.591 INFO:teuthology.orchestra.run.smithi080.stdout: "wait": { 2024-04-27T00:13:47.591 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.591 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.591 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.591 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:47.591 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.591 INFO:teuthology.orchestra.run.smithi080.stdout: "throttle-write_buf_throttle": { 2024-04-27T00:13:47.591 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 0, 2024-04-27T00:13:47.591 INFO:teuthology.orchestra.run.smithi080.stdout: "max": 3758096384, 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "get_started": 0, 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "get": 3, 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "get_sum": 311, 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_success": 3, 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "take": 0, 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "take_sum": 0, 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "put": 1, 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "put_sum": 311, 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "wait": { 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.592 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.593 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:47.593 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:47.593 INFO:teuthology.orchestra.run.smithi080.stdout: "throttle-write_buf_throttle-0x562ffeef8500": { 2024-04-27T00:13:47.593 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 0, 2024-04-27T00:13:47.593 INFO:teuthology.orchestra.run.smithi080.stdout: "max": 3758096384, 2024-04-27T00:13:47.593 INFO:teuthology.orchestra.run.smithi080.stdout: "get_started": 0, 2024-04-27T00:13:47.593 INFO:teuthology.orchestra.run.smithi080.stdout: "get": 2483, 2024-04-27T00:13:47.593 INFO:teuthology.orchestra.run.smithi080.stdout: "get_sum": 7939833, 2024-04-27T00:13:47.593 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:47.593 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_success": 2483, 2024-04-27T00:13:47.593 INFO:teuthology.orchestra.run.smithi080.stdout: "take": 0, 2024-04-27T00:13:47.593 INFO:teuthology.orchestra.run.smithi080.stdout: "take_sum": 0, 2024-04-27T00:13:47.594 INFO:teuthology.orchestra.run.smithi080.stdout: "put": 2465, 2024-04-27T00:13:47.594 INFO:teuthology.orchestra.run.smithi080.stdout: "put_sum": 7939833, 2024-04-27T00:13:47.594 INFO:teuthology.orchestra.run.smithi080.stdout: "wait": { 2024-04-27T00:13:47.594 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:47.594 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:47.594 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:47.594 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:47.594 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:47.594 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-04-27T00:13:48.407 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-27T00:13:48.408 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-27T00:13:48.408 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=7 2024-04-27T00:13:48.408 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-04-27T00:13:48.408 DEBUG:tasks.check_counter:Getting stats from e 2024-04-27T00:13:48.408 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-27T00:13:48.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:48 smithi080 ceph-mon[32362]: pgmap v775: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-04-27T00:13:48.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:48 smithi103 ceph-mon[32952]: pgmap v775: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-04-27T00:13:48.657 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T00:13:48.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:48 smithi040 ceph-mon[28095]: pgmap v775: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-04-27T00:13:50.299 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-04-27T00:13:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "AsyncMessenger::Worker-0": { 2024-04-27T00:13:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_messages": 250, 2024-04-27T00:13:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages": 248, 2024-04-27T00:13:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_bytes": 351168, 2024-04-27T00:13:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_bytes": 153200, 2024-04-27T00:13:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_created_connections": 4, 2024-04-27T00:13:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_active_connections": 0, 2024-04-27T00:13:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_total_time": 0.041202101, 2024-04-27T00:13:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_send_time": 0.016287214, 2024-04-27T00:13:50.300 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_recv_time": 0.014730634, 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_fast_dispatch_time": 0.006468767, 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 247, 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.013297264, 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000053835 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_encrypted_bytes": 351168, 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_encrypted_bytes": 153200 2024-04-27T00:13:50.301 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "AsyncMessenger::Worker-1": { 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_messages": 3, 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages": 192, 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_bytes": 480, 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_bytes": 269888, 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_created_connections": 5, 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_active_connections": 2, 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_total_time": 0.012428508, 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_send_time": 0.008615244, 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_recv_time": 0.001363544, 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 190, 2024-04-27T00:13:50.302 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.017756719, 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000093456 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_encrypted_bytes": 269888 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "AsyncMessenger::Worker-2": { 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_messages": 4, 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages": 4, 2024-04-27T00:13:50.303 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_bytes": 2992, 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_bytes": 448, 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_created_connections": 2, 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_active_connections": 1, 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_total_time": 0.000498869, 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_send_time": 0.000028318, 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_recv_time": 0.000428322, 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 3, 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000691475, 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000230491 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:50.304 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_encrypted_bytes": 448 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "cct": { 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "total_workers": 0, 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "unhealthy_workers": 0 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "mempool": { 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "bloom_filter_bytes": 0, 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "bloom_filter_items": 0, 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_alloc_bytes": 0, 2024-04-27T00:13:50.305 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_alloc_items": 0, 2024-04-27T00:13:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_data_bytes": 0, 2024-04-27T00:13:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_data_items": 0, 2024-04-27T00:13:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-27T00:13:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_onode_items": 0, 2024-04-27T00:13:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-27T00:13:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_meta_items": 0, 2024-04-27T00:13:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_other_bytes": 0, 2024-04-27T00:13:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_other_items": 0, 2024-04-27T00:13:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-27T00:13:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_buffer_items": 0, 2024-04-27T00:13:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_bytes": 0, 2024-04-27T00:13:50.306 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_items": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_blob_bytes": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_blob_items": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_shared_blob_items": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_inline_bl_items": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_bytes": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_items": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_txc_bytes": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_txc_items": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_deferred_items": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_bytes": 0, 2024-04-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_items": 0, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_bytes": 0, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_items": 0, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_reader_bytes": 0, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_reader_items": 0, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_writer_bytes": 0, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_writer_items": 0, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_anon_bytes": 147915, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_anon_items": 28, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_meta_bytes": 0, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_meta_items": 0, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_bytes": 0, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_items": 0, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mapbl_bytes": 0, 2024-04-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mapbl_items": 0, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pglog_bytes": 0, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pglog_items": 0, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_bytes": 0, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_items": 0, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_mapping_bytes": 0, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_mapping_items": 0, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "pgmap_bytes": 0, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "pgmap_items": 0, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_co_bytes": 928, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_co_items": 2, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_1_bytes": 0, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_1_items": 0, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_2_bytes": 0, 2024-04-27T00:13:50.309 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_2_items": 0 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 0, 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "max": 104857600, 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "get_started": 0, 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "get": 253, 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "get_sum": 315872, 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_success": 253, 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "take": 0, 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "take_sum": 0, 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "put": 253, 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "put_sum": 315872, 2024-04-27T00:13:50.310 INFO:teuthology.orchestra.run.smithi080.stdout: "wait": { 2024-04-27T00:13:50.311 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:50.311 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:50.311 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:50.311 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:50.311 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:50.311 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-04-27T00:13:50.437 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:50 smithi040 ceph-mon[28095]: pgmap v776: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:13:50.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:50 smithi080 ceph-mon[32362]: pgmap v776: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:13:50.599 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:50 smithi103 ceph-mon[32952]: pgmap v776: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:13:50.972 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-04-27T00:13:50.972 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-04-27T00:13:50.972 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.e 2024-04-27T00:13:50.973 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.e 2024-04-27T00:13:50.973 DEBUG:tasks.check_counter:Getting stats from h 2024-04-27T00:13:50.973 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-27T00:13:51.210 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T00:13:52.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:52 smithi080 ceph-mon[32362]: pgmap v777: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-27T00:13:52.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:52 smithi103 ceph-mon[32952]: pgmap v777: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-27T00:13:52.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:52 smithi040 ceph-mon[28095]: pgmap v777: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-27T00:13:52.972 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-04-27T00:13:52.972 INFO:teuthology.orchestra.run.smithi080.stdout: "AsyncMessenger::Worker-0": { 2024-04-27T00:13:52.972 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_messages": 1129, 2024-04-27T00:13:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages": 1124, 2024-04-27T00:13:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_bytes": 665168, 2024-04-27T00:13:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_bytes": 429610, 2024-04-27T00:13:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_created_connections": 5, 2024-04-27T00:13:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_active_connections": 1, 2024-04-27T00:13:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_total_time": 0.136054109, 2024-04-27T00:13:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_send_time": 0.055909741, 2024-04-27T00:13:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_recv_time": 0.033919052, 2024-04-27T00:13:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_fast_dispatch_time": 0.032785770, 2024-04-27T00:13:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 1124, 2024-04-27T00:13:52.973 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.067841871, 2024-04-27T00:13:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000060357 2024-04-27T00:13:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_encrypted_bytes": 373856, 2024-04-27T00:13:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_encrypted_bytes": 153904 2024-04-27T00:13:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.974 INFO:teuthology.orchestra.run.smithi080.stdout: "AsyncMessenger::Worker-1": { 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_messages": 5, 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages": 193, 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_bytes": 1036, 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_bytes": 570154, 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_created_connections": 6, 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_active_connections": 3, 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_total_time": 0.011592432, 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_send_time": 0.008151907, 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_recv_time": 0.001447248, 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_fast_dispatch_time": 0.000043483, 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 191, 2024-04-27T00:13:52.975 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.015846724, 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000082967 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_encrypted_bytes": 569456 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "AsyncMessenger::Worker-2": { 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_messages": 872, 2024-04-27T00:13:52.976 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages": 874, 2024-04-27T00:13:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_bytes": 230008, 2024-04-27T00:13:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_bytes": 277303, 2024-04-27T00:13:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_created_connections": 4, 2024-04-27T00:13:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_active_connections": 3, 2024-04-27T00:13:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_total_time": 0.069879085, 2024-04-27T00:13:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_send_time": 0.019333549, 2024-04-27T00:13:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_recv_time": 0.020069739, 2024-04-27T00:13:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_fast_dispatch_time": 0.024161930, 2024-04-27T00:13:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 872, 2024-04-27T00:13:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.024236883, 2024-04-27T00:13:52.977 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000027794 2024-04-27T00:13:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-27T00:13:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_encrypted_bytes": 0 2024-04-27T00:13:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: "cct": { 2024-04-27T00:13:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: "total_workers": 1, 2024-04-27T00:13:52.978 INFO:teuthology.orchestra.run.smithi080.stdout: "unhealthy_workers": 0 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "finisher-MDSRank": { 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "queue_len": 0, 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "complete_latency": { 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 3475, 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.099039176, 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000028500 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "finisher-PurgeQueue": { 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "queue_len": 0, 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "complete_latency": { 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.979 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "mds": { 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "request": 0, 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "reply": 0, 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "reply_latency": { 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "slow_reply": 0, 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "forward": 0, 2024-04-27T00:13:52.980 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_fetch_complete": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_fetch_keys": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_commit": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_split": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_merge": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes_top": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes_bottom": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes_pin_tail": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes_pinned": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes_expired": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "inodes_with_caps": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "caps": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "subtrees": 1, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse_hit": 0, 2024-04-27T00:13:52.981 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse_forward": 0, 2024-04-27T00:13:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse_discover": 0, 2024-04-27T00:13:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse_dir_fetch": 0, 2024-04-27T00:13:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse_remote_ino": 0, 2024-04-27T00:13:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "traverse_lock": 0, 2024-04-27T00:13:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "load_cent": 0, 2024-04-27T00:13:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "q": 0, 2024-04-27T00:13:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "exported": 0, 2024-04-27T00:13:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "exported_inodes": 0, 2024-04-27T00:13:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "imported": 0, 2024-04-27T00:13:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "imported_inodes": 0, 2024-04-27T00:13:52.982 INFO:teuthology.orchestra.run.smithi080.stdout: "openino_dir_fetch": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "openino_backtrace_fetch": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "openino_peer_discover": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "root_rfiles": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "root_rbytes": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "root_rsnaps": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_backtrace_fetch": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_set_tag": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_backtrace_repaired": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_inotable_repaired": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_dir_inodes": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_dir_base_inodes": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_dirfrag_rstats": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "scrub_file_inodes": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_inode_file_caps": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_cap_op_revoke": 0, 2024-04-27T00:13:52.983 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_cap_op_grant": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_cap_op_trunc": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_client_caps": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_client_caps_dirty": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_client_cap_release": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "process_request_cap_release": 0 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_cache": { 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_update": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_update_receipt": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_try_discover": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_send_discover": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "dir_handle_discover": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "num_strays": 0, 2024-04-27T00:13:52.984 INFO:teuthology.orchestra.run.smithi080.stdout: "num_strays_delayed": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "num_strays_enqueuing": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "strays_created": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "strays_enqueued": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "strays_reintegrated": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "strays_migrated": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "num_recovering_processing": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "num_recovering_enqueued": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "num_recovering_prioritized": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "recovery_started": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "recovery_completed": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_quiesce_path": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_quiesce_inode": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_enqueue_scrub": 0, 2024-04-27T00:13:52.985 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_exportdir": 0, 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_flush": 0, 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_fragmentdir": 0, 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_fragstats": 0, 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "ireq_inodestats": 0 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_log": { 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "evlrg": 0, 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "evadd": 0, 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "evex": 0, 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "evtrm": 0, 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "ev": 3, 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "evexg": 0, 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "evexd": 0, 2024-04-27T00:13:52.986 INFO:teuthology.orchestra.run.smithi080.stdout: "segadd": 0, 2024-04-27T00:13:52.987 INFO:teuthology.orchestra.run.smithi080.stdout: "segex": 0, 2024-04-27T00:13:52.987 INFO:teuthology.orchestra.run.smithi080.stdout: "segtrm": 0, 2024-04-27T00:13:52.987 INFO:teuthology.orchestra.run.smithi080.stdout: "seg": 2, 2024-04-27T00:13:52.987 INFO:teuthology.orchestra.run.smithi080.stdout: "segmjr": 2, 2024-04-27T00:13:52.987 INFO:teuthology.orchestra.run.smithi080.stdout: "segexg": 0, 2024-04-27T00:13:52.987 INFO:teuthology.orchestra.run.smithi080.stdout: "segexd": 0, 2024-04-27T00:13:52.987 INFO:teuthology.orchestra.run.smithi080.stdout: "expos": 4194304, 2024-04-27T00:13:52.987 INFO:teuthology.orchestra.run.smithi080.stdout: "wrpos": 4195283, 2024-04-27T00:13:52.987 INFO:teuthology.orchestra.run.smithi080.stdout: "rdpos": 4194892, 2024-04-27T00:13:52.987 INFO:teuthology.orchestra.run.smithi080.stdout: "jlat": { 2024-04-27T00:13:52.987 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.988 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.988 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.988 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.988 INFO:teuthology.orchestra.run.smithi080.stdout: "replayed": 3 2024-04-27T00:13:52.988 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.988 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_mem": { 2024-04-27T00:13:52.988 INFO:teuthology.orchestra.run.smithi080.stdout: "ino": 3, 2024-04-27T00:13:52.988 INFO:teuthology.orchestra.run.smithi080.stdout: "ino+": 3, 2024-04-27T00:13:52.988 INFO:teuthology.orchestra.run.smithi080.stdout: "ino-": 0, 2024-04-27T00:13:52.988 INFO:teuthology.orchestra.run.smithi080.stdout: "dir": 1, 2024-04-27T00:13:52.988 INFO:teuthology.orchestra.run.smithi080.stdout: "dir+": 1, 2024-04-27T00:13:52.989 INFO:teuthology.orchestra.run.smithi080.stdout: "dir-": 0, 2024-04-27T00:13:52.989 INFO:teuthology.orchestra.run.smithi080.stdout: "dn": 0, 2024-04-27T00:13:52.989 INFO:teuthology.orchestra.run.smithi080.stdout: "dn+": 0, 2024-04-27T00:13:52.989 INFO:teuthology.orchestra.run.smithi080.stdout: "dn-": 0, 2024-04-27T00:13:52.989 INFO:teuthology.orchestra.run.smithi080.stdout: "cap": 0, 2024-04-27T00:13:52.989 INFO:teuthology.orchestra.run.smithi080.stdout: "cap+": 0, 2024-04-27T00:13:52.989 INFO:teuthology.orchestra.run.smithi080.stdout: "cap-": 0, 2024-04-27T00:13:52.989 INFO:teuthology.orchestra.run.smithi080.stdout: "rss": 49188, 2024-04-27T00:13:52.989 INFO:teuthology.orchestra.run.smithi080.stdout: "heap": 215416 2024-04-27T00:13:52.989 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.989 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_server": { 2024-04-27T00:13:52.989 INFO:teuthology.orchestra.run.smithi080.stdout: "dispatch_client_request": 0, 2024-04-27T00:13:52.990 INFO:teuthology.orchestra.run.smithi080.stdout: "dispatch_server_request": 0, 2024-04-27T00:13:52.990 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_client_request": 0, 2024-04-27T00:13:52.990 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_client_session": 0, 2024-04-27T00:13:52.990 INFO:teuthology.orchestra.run.smithi080.stdout: "handle_peer_request": 0, 2024-04-27T00:13:52.990 INFO:teuthology.orchestra.run.smithi080.stdout: "req_create_latency": { 2024-04-27T00:13:52.990 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.990 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.990 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.990 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.990 INFO:teuthology.orchestra.run.smithi080.stdout: "req_getattr_latency": { 2024-04-27T00:13:52.990 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.990 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: "req_getfilelock_latency": { 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: "req_link_latency": { 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lookup_latency": { 2024-04-27T00:13:52.991 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lookuphash_latency": { 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lookupino_latency": { 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.992 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lookupname_latency": { 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lookupparent_latency": { 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lookupsnap_latency": { 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.993 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.994 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.994 INFO:teuthology.orchestra.run.smithi080.stdout: "req_lssnap_latency": { 2024-04-27T00:13:52.994 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.994 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.994 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.994 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.994 INFO:teuthology.orchestra.run.smithi080.stdout: "req_mkdir_latency": { 2024-04-27T00:13:52.994 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.995 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.995 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.995 INFO:teuthology.orchestra.run.smithi080.stdout: "req_mknod_latency": { 2024-04-27T00:13:52.995 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.995 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.995 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.995 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.995 INFO:teuthology.orchestra.run.smithi080.stdout: "req_mksnap_latency": { 2024-04-27T00:13:52.995 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.995 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: "req_open_latency": { 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: "req_readdir_latency": { 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.996 INFO:teuthology.orchestra.run.smithi080.stdout: "req_rename_latency": { 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: "req_renamesnap_latency": { 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: "req_snapdiff_latency": { 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.997 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: "req_rmdir_latency": { 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: "req_rmsnap_latency": { 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: "req_rmxattr_latency": { 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.998 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: "req_setattr_latency": { 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: "req_setdirlayout_latency": { 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:52.999 INFO:teuthology.orchestra.run.smithi080.stdout: "req_setfilelock_latency": { 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: "req_setlayout_latency": { 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: "req_setxattr_latency": { 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.000 INFO:teuthology.orchestra.run.smithi080.stdout: "req_symlink_latency": { 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "req_unlink_latency": { 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "cap_revoke_eviction": 0, 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "cap_acquisition_throttle": 0, 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "req_getvxattr_latency": { 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:53.001 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_sessions": { 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "session_count": 1, 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "session_add": 1, 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "session_remove": 0, 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "sessions_open": 1, 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "sessions_stale": 0, 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "total_load": 0, 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "average_load": 0, 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "avg_session_uptime": 2274, 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "mdthresh_evicted": 0 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "mempool": { 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "bloom_filter_bytes": 0, 2024-04-27T00:13:53.002 INFO:teuthology.orchestra.run.smithi080.stdout: "bloom_filter_items": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_alloc_bytes": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_alloc_items": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_data_bytes": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_data_items": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_onode_items": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_meta_items": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_other_bytes": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_other_items": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_buffer_items": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_bytes": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_items": 0, 2024-04-27T00:13:53.003 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_blob_bytes": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_blob_items": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_shared_blob_items": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_inline_bl_items": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_bytes": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_items": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_txc_bytes": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_txc_items": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_deferred_items": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_bytes": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_items": 0, 2024-04-27T00:13:53.004 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_bytes": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_items": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_reader_bytes": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_reader_items": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_writer_bytes": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_writer_items": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_anon_bytes": 181265, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_anon_items": 45, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_meta_bytes": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_meta_items": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_bytes": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_items": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mapbl_bytes": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mapbl_items": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pglog_bytes": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pglog_items": 0, 2024-04-27T00:13:53.005 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_bytes": 8896, 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_items": 171, 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_mapping_bytes": 0, 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_mapping_items": 0, 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "pgmap_bytes": 0, 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "pgmap_items": 0, 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_co_bytes": 8512, 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_co_items": 13, 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_1_bytes": 0, 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_1_items": 0, 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_2_bytes": 0, 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_2_items": 0 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "objecter": { 2024-04-27T00:13:53.006 INFO:teuthology.orchestra.run.smithi080.stdout: "op_active": 0, 2024-04-27T00:13:53.007 INFO:teuthology.orchestra.run.smithi080.stdout: "op_laggy": 0, 2024-04-27T00:13:53.007 INFO:teuthology.orchestra.run.smithi080.stdout: "op_send": 1741, 2024-04-27T00:13:53.007 INFO:teuthology.orchestra.run.smithi080.stdout: "op_send_bytes": 21, 2024-04-27T00:13:53.007 INFO:teuthology.orchestra.run.smithi080.stdout: "op_resend": 3, 2024-04-27T00:13:53.007 INFO:teuthology.orchestra.run.smithi080.stdout: "op_reply": 1741, 2024-04-27T00:13:53.007 INFO:teuthology.orchestra.run.smithi080.stdout: "op_latency": { 2024-04-27T00:13:53.007 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 1741, 2024-04-27T00:13:53.007 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.875979714, 2024-04-27T00:13:53.007 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000503147 2024-04-27T00:13:53.007 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.007 INFO:teuthology.orchestra.run.smithi080.stdout: "op_inflight": 0, 2024-04-27T00:13:53.007 INFO:teuthology.orchestra.run.smithi080.stdout: "oplen_avg": { 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 1741, 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 1742 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: "op": 1741, 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: "op_r": 1741, 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: "op_w": 0, 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: "op_rmw": 0, 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: "op_pg": 0, 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_stat": 867, 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_create": 0, 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_read": 872, 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_write": 0, 2024-04-27T00:13:53.008 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_writefull": 0, 2024-04-27T00:13:53.009 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_writesame": 0, 2024-04-27T00:13:53.009 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_append": 0, 2024-04-27T00:13:53.009 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_zero": 0, 2024-04-27T00:13:53.009 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_truncate": 0, 2024-04-27T00:13:53.009 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_delete": 0, 2024-04-27T00:13:53.009 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_mapext": 0, 2024-04-27T00:13:53.009 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_sparse_read": 0, 2024-04-27T00:13:53.009 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_clonerange": 0, 2024-04-27T00:13:53.009 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_getxattr": 1, 2024-04-27T00:13:53.009 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_setxattr": 0, 2024-04-27T00:13:53.009 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_cmpxattr": 0, 2024-04-27T00:13:53.010 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_rmxattr": 0, 2024-04-27T00:13:53.010 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_resetxattrs": 0, 2024-04-27T00:13:53.010 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_call": 0, 2024-04-27T00:13:53.010 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_watch": 0, 2024-04-27T00:13:53.010 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_notify": 0, 2024-04-27T00:13:53.010 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_src_cmpxattr": 0, 2024-04-27T00:13:53.010 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_pgls": 0, 2024-04-27T00:13:53.010 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_pgls_filter": 0, 2024-04-27T00:13:53.010 INFO:teuthology.orchestra.run.smithi080.stdout: "osdop_other": 0, 2024-04-27T00:13:53.010 INFO:teuthology.orchestra.run.smithi080.stdout: "linger_active": 0, 2024-04-27T00:13:53.010 INFO:teuthology.orchestra.run.smithi080.stdout: "linger_send": 0, 2024-04-27T00:13:53.010 INFO:teuthology.orchestra.run.smithi080.stdout: "linger_resend": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "linger_ping": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "poolop_active": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "poolop_send": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "poolop_resend": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "poolstat_active": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "poolstat_send": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "poolstat_resend": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "statfs_active": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "statfs_send": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "statfs_resend": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "command_active": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "command_send": 0, 2024-04-27T00:13:53.011 INFO:teuthology.orchestra.run.smithi080.stdout: "command_resend": 0, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "map_epoch": 94, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "map_full": 0, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "map_inc": 4, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_sessions": 4, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_session_open": 4, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_session_close": 0, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_laggy": 0, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_wr": 0, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_rd": 2, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_del": 0 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "oft": { 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_total_objs": 0, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_total_kv_pairs": 0, 2024-04-27T00:13:53.012 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_total_updates": 0, 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: "omap_total_removes": 0 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: "osd-slow-ops": { 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: "slow_ops_count": 0 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: "purge_queue": { 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_executing_ops": 0, 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_executing_ops_high_water": 0, 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_executing": 0, 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_executing_high_water": 0, 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_executed_ops": 0, 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_executed": 0, 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: "pq_item_in_journal": 0 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.013 INFO:teuthology.orchestra.run.smithi080.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 0, 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "max": 104857600, 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "get_started": 0, 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "get": 2002, 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "get_sum": 699678, 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_success": 2002, 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "take": 0, 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "take_sum": 0, 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "put": 2002, 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "put_sum": 699678, 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "wait": { 2024-04-27T00:13:53.014 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: "throttle-objecter_bytes": { 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 0, 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: "max": 104857600, 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: "get_started": 0, 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: "get": 0, 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: "get_sum": 0, 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: "take": 1741, 2024-04-27T00:13:53.015 INFO:teuthology.orchestra.run.smithi080.stdout: "take_sum": 984, 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "put": 1741, 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "put_sum": 984, 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "wait": { 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "throttle-objecter_ops": { 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 0, 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "max": 1024, 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "get_started": 0, 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "get": 0, 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "get_sum": 0, 2024-04-27T00:13:53.016 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "take": 1741, 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "take_sum": 1741, 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "put": 1741, 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "put_sum": 1741, 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "wait": { 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "throttle-write_buf_throttle": { 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 0, 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "max": 3758096384, 2024-04-27T00:13:53.017 INFO:teuthology.orchestra.run.smithi080.stdout: "get_started": 0, 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: "get": 0, 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: "get_sum": 0, 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: "take": 0, 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: "take_sum": 0, 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: "put": 0, 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: "put_sum": 0, 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: "wait": { 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:53.018 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "throttle-write_buf_throttle-0x560610a266e0": { 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 0, 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "max": 3758096384, 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "get_started": 0, 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "get": 0, 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "get_sum": 0, 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "take": 0, 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "take_sum": 0, 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "put": 0, 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "put_sum": 0, 2024-04-27T00:13:53.019 INFO:teuthology.orchestra.run.smithi080.stdout: "wait": { 2024-04-27T00:13:53.020 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:53.020 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:53.020 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:53.020 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:53.020 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:53.020 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-04-27T00:13:53.757 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-27T00:13:53.757 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-27T00:13:53.757 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-04-27T00:13:53.757 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-04-27T00:13:53.757 DEBUG:tasks.check_counter:Getting stats from k 2024-04-27T00:13:53.758 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-27T00:13:53.993 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T00:13:54.403 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:54 smithi080 ceph-mon[32362]: pgmap v778: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-27T00:13:54.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:54 smithi103 ceph-mon[32952]: pgmap v778: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-27T00:13:54.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:54 smithi040 ceph-mon[28095]: pgmap v778: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-27T00:13:55.747 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-04-27T00:13:55.747 INFO:teuthology.orchestra.run.smithi080.stdout: "AsyncMessenger::Worker-0": { 2024-04-27T00:13:55.747 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_messages": 4, 2024-04-27T00:13:55.747 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages": 4, 2024-04-27T00:13:55.747 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_bytes": 2992, 2024-04-27T00:13:55.747 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_bytes": 448, 2024-04-27T00:13:55.747 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_created_connections": 4, 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_active_connections": 0, 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_total_time": 0.001317730, 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_send_time": 0.000031133, 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_recv_time": 0.000479591, 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 3, 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000714125, 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000238041 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:55.748 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_encrypted_bytes": 448 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "AsyncMessenger::Worker-1": { 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_messages": 2, 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages": 193, 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_bytes": 320, 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_bytes": 191248, 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_created_connections": 4, 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_active_connections": 2, 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_total_time": 0.012684798, 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_send_time": 0.009120757, 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_recv_time": 0.001056212, 2024-04-27T00:13:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 191, 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.015242568, 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000079804 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_encrypted_bytes": 191248 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: "AsyncMessenger::Worker-2": { 2024-04-27T00:13:55.751 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_messages": 254, 2024-04-27T00:13:55.753 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages": 252, 2024-04-27T00:13:55.753 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_bytes": 352256, 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_bytes": 155760, 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_created_connections": 2, 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_active_connections": 1, 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_total_time": 0.040375410, 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_send_time": 0.016584244, 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_recv_time": 0.014178668, 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_running_fast_dispatch_time": 0.006693415, 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 251, 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.014426691, 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000057476 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:55.754 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_recv_encrypted_bytes": 352256, 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: "msgr_send_encrypted_bytes": 155760 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: "cct": { 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: "total_workers": 0, 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: "unhealthy_workers": 0 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: "mempool": { 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: "bloom_filter_bytes": 0, 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: "bloom_filter_items": 0, 2024-04-27T00:13:55.755 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_alloc_bytes": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_alloc_items": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_data_bytes": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_data_items": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_onode_items": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_meta_items": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_other_bytes": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_other_items": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_cache_buffer_items": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_bytes": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_extent_items": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_blob_bytes": 0, 2024-04-27T00:13:55.756 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_blob_items": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_shared_blob_items": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_inline_bl_items": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_bytes": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_fsck_items": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_txc_bytes": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_txc_items": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_deferred_items": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_bytes": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluestore_writing_items": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_bytes": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_items": 0, 2024-04-27T00:13:55.757 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_reader_bytes": 0, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_reader_items": 0, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_writer_bytes": 0, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "bluefs_file_writer_items": 0, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_anon_bytes": 147898, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_anon_items": 27, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_meta_bytes": 0, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "buffer_meta_items": 0, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_bytes": 0, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_items": 0, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mapbl_bytes": 0, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_mapbl_items": 0, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pglog_bytes": 0, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "osd_pglog_items": 0, 2024-04-27T00:13:55.758 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_bytes": 0, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_items": 0, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_mapping_bytes": 0, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "osdmap_mapping_items": 0, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "pgmap_bytes": 0, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "pgmap_items": 0, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_co_bytes": 928, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "mds_co_items": 2, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_1_bytes": 0, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_1_items": 0, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_2_bytes": 0, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "unittest_2_items": 0 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: }, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "val": 0, 2024-04-27T00:13:55.759 INFO:teuthology.orchestra.run.smithi080.stdout: "max": 104857600, 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "get_started": 0, 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "get": 256, 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "get_sum": 316365, 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "get_or_fail_success": 256, 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "take": 0, 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "take_sum": 0, 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "put": 256, 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "put_sum": 316365, 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "wait": { 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "avgcount": 0, 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "sum": 0.000000000, 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: "avgtime": 0.000000000 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:55.760 INFO:teuthology.orchestra.run.smithi080.stdout: } 2024-04-27T00:13:55.761 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-04-27T00:13:56.410 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:56 smithi080 ceph-mon[32362]: pgmap v779: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:13:56.438 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-27T00:13:56.438 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-27T00:13:56.438 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-04-27T00:13:56.438 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-04-27T00:13:56.439 DEBUG:tasks.check_counter:Getting stats from c 2024-04-27T00:13:56.439 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-27T00:13:56.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:56 smithi103 ceph-mon[32952]: pgmap v779: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:13:56.690 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-27T00:13:56.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:56 smithi040 ceph-mon[28095]: pgmap v779: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-27T00:13:58.381 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:13:58 smithi103 ceph-mon[32952]: pgmap v780: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-27T00:13:58.508 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-27T00:13:58.508 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-04-27T00:13:58.508 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 2020, 2024-04-27T00:13:58.508 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 2017, 2024-04-27T00:13:58.508 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 5086574, 2024-04-27T00:13:58.508 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 714104, 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 5, 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 1, 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.223733084, 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.072928973, 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.066459903, 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.064873744, 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2016, 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.084475587, 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000041902 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.509 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.510 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.510 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 371904, 2024-04-27T00:13:58.510 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 155056 2024-04-27T00:13:58.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.510 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-04-27T00:13:58.510 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 325, 2024-04-27T00:13:58.510 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 514, 2024-04-27T00:13:58.510 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 80536, 2024-04-27T00:13:58.510 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 507082, 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 5, 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 4, 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.043566970, 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.019761670, 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.007972394, 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.010974423, 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 512, 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.030422150, 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000059418 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.511 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 403984 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 1171, 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 1171, 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 4040829, 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 371561, 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 4, 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 3, 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.106511752, 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.039501747, 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.026258354, 2024-04-27T00:13:58.512 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.031401610, 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1170, 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.042153003, 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000036028 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 448 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 1, 2024-04-27T00:13:58.513 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-MDSRank": { 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 5514, 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.152012774, 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000027568 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-PurgeQueue": { 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.514 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: "request": 0, 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: "reply": 0, 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: "reply_latency": { 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_reply": 0, 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: "forward": 0, 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_complete": 0, 2024-04-27T00:13:58.515 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_keys": 0, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_commit": 0, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_split": 0, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_merge": 0, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes": 17, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_top": 4, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_bottom": 13, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pin_tail": 0, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pinned": 10, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_expired": 0, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_with_caps": 0, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": 0, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "subtrees": 2, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse": 0, 2024-04-27T00:13:58.516 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_hit": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_forward": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_discover": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_dir_fetch": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_remote_ino": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_lock": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "load_cent": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "q": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "exported": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "exported_inodes": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "imported": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "imported_inodes": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_dir_fetch": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_backtrace_fetch": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_peer_discover": 0, 2024-04-27T00:13:58.517 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rfiles": 4, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rbytes": 234, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rsnaps": 0, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_fetch": 0, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_set_tag": 0, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_repaired": 0, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_inotable_repaired": 0, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_inodes": 0, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_base_inodes": 0, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dirfrag_rstats": 0, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_file_inodes": 0, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_inode_file_caps": 0, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_revoke": 0, 2024-04-27T00:13:58.518 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_grant": 0, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_trunc": 0, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps": 0, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps_dirty": 0, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_cap_release": 0, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "process_request_cap_release": 0 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache": { 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update": 0, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update_receipt": 0, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_try_discover": 0, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_send_discover": 0, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_handle_discover": 0, 2024-04-27T00:13:58.519 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays": 0, 2024-04-27T00:13:58.520 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_delayed": 0, 2024-04-27T00:13:58.520 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_enqueuing": 0, 2024-04-27T00:13:58.520 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_created": 0, 2024-04-27T00:13:58.520 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_enqueued": 0, 2024-04-27T00:13:58.520 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_reintegrated": 0, 2024-04-27T00:13:58.520 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_migrated": 0, 2024-04-27T00:13:58.520 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_processing": 0, 2024-04-27T00:13:58.520 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_enqueued": 0, 2024-04-27T00:13:58.520 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_prioritized": 0, 2024-04-27T00:13:58.520 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_started": 0, 2024-04-27T00:13:58.520 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_completed": 0, 2024-04-27T00:13:58.521 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_path": 0, 2024-04-27T00:13:58.521 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_inode": 0, 2024-04-27T00:13:58.521 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_enqueue_scrub": 0, 2024-04-27T00:13:58.521 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_exportdir": 0, 2024-04-27T00:13:58.521 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_flush": 0, 2024-04-27T00:13:58.521 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragmentdir": 0, 2024-04-27T00:13:58.521 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragstats": 0, 2024-04-27T00:13:58.521 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_inodestats": 0 2024-04-27T00:13:58.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.521 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log": { 2024-04-27T00:13:58.521 INFO:teuthology.orchestra.run.smithi103.stdout: "evlrg": 0, 2024-04-27T00:13:58.521 INFO:teuthology.orchestra.run.smithi103.stdout: "evadd": 0, 2024-04-27T00:13:58.522 INFO:teuthology.orchestra.run.smithi103.stdout: "evex": 0, 2024-04-27T00:13:58.522 INFO:teuthology.orchestra.run.smithi103.stdout: "evtrm": 0, 2024-04-27T00:13:58.522 INFO:teuthology.orchestra.run.smithi103.stdout: "ev": 2483, 2024-04-27T00:13:58.522 INFO:teuthology.orchestra.run.smithi103.stdout: "evexg": 0, 2024-04-27T00:13:58.522 INFO:teuthology.orchestra.run.smithi103.stdout: "evexd": 0, 2024-04-27T00:13:58.522 INFO:teuthology.orchestra.run.smithi103.stdout: "segadd": 0, 2024-04-27T00:13:58.522 INFO:teuthology.orchestra.run.smithi103.stdout: "segex": 0, 2024-04-27T00:13:58.522 INFO:teuthology.orchestra.run.smithi103.stdout: "segtrm": 0, 2024-04-27T00:13:58.522 INFO:teuthology.orchestra.run.smithi103.stdout: "seg": 2, 2024-04-27T00:13:58.522 INFO:teuthology.orchestra.run.smithi103.stdout: "segmjr": 54, 2024-04-27T00:13:58.522 INFO:teuthology.orchestra.run.smithi103.stdout: "segexg": 0, 2024-04-27T00:13:58.522 INFO:teuthology.orchestra.run.smithi103.stdout: "segexd": 0, 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "expos": 11974663, 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "wrpos": 12134137, 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "rdpos": 12130715, 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "jlat": { 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "replayed": 2483 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_mem": { 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 14, 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "ino+": 17, 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "ino-": 3, 2024-04-27T00:13:58.523 INFO:teuthology.orchestra.run.smithi103.stdout: "dir": 9, 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "dir+": 11, 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "dir-": 2, 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "dn": 17, 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "dn+": 19, 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "dn-": 2, 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "cap": 0, 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "cap+": 0, 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "cap-": 0, 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "rss": 51140, 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "heap": 215416 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_server": { 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_client_request": 0, 2024-04-27T00:13:58.524 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_server_request": 0, 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_request": 0, 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_session": 0, 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_peer_request": 0, 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: "req_create_latency": { 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getattr_latency": { 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.525 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getfilelock_latency": { 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: "req_link_latency": { 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookup_latency": { 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookuphash_latency": { 2024-04-27T00:13:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupino_latency": { 2024-04-27T00:13:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupname_latency": { 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupparent_latency": { 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupsnap_latency": { 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lssnap_latency": { 2024-04-27T00:13:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mkdir_latency": { 2024-04-27T00:13:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mknod_latency": { 2024-04-27T00:13:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mksnap_latency": { 2024-04-27T00:13:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "req_open_latency": { 2024-04-27T00:13:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "req_readdir_latency": { 2024-04-27T00:13:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rename_latency": { 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "req_renamesnap_latency": { 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "req_snapdiff_latency": { 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmdir_latency": { 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmsnap_latency": { 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmxattr_latency": { 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setattr_latency": { 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setdirlayout_latency": { 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setfilelock_latency": { 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setlayout_latency": { 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setxattr_latency": { 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "req_symlink_latency": { 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "req_unlink_latency": { 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_revoke_eviction": 0, 2024-04-27T00:13:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_acquisition_throttle": 0, 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getvxattr_latency": { 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sessions": { 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "session_count": 2, 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "session_add": 3, 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "session_remove": 1, 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_open": 2, 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_stale": 0, 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "total_load": 0, 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "average_load": 0, 2024-04-27T00:13:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "avg_session_uptime": 2246, 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "mdthresh_evicted": 0 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-04-27T00:13:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-04-27T00:13:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 189361, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 50, 2024-04-27T00:13:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-04-27T00:13:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-04-27T00:13:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-04-27T00:13:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-04-27T00:13:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-04-27T00:13:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-04-27T00:13:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-04-27T00:13:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-04-27T00:13:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 8896, 2024-04-27T00:13:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 171, 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 55106, 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 263, 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter": { 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "op_active": 0, 2024-04-27T00:13:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "op_laggy": 0, 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send": 3249, 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send_bytes": 26, 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "op_resend": 4, 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "op_reply": 3249, 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "op_latency": { 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3249, 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 1.510972402, 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000465057 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "op_inflight": 0, 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "oplen_avg": { 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3249, 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 3250 2024-04-27T00:13:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "op": 3249, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "op_r": 3249, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "op_w": 0, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "op_rmw": 0, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "op_pg": 0, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_stat": 1744, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_create": 0, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_read": 1502, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_write": 0, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writefull": 0, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writesame": 0, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_append": 0, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_zero": 0, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_truncate": 0, 2024-04-27T00:13:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_delete": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_mapext": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_sparse_read": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_clonerange": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_getxattr": 2, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_setxattr": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_cmpxattr": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_rmxattr": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_resetxattrs": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_call": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_watch": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_notify": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_src_cmpxattr": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls_filter": 0, 2024-04-27T00:13:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_other": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_active": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_send": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_resend": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_ping": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_active": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_send": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_resend": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_active": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_send": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_resend": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_active": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_send": 0, 2024-04-27T00:13:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_resend": 0, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "command_active": 0, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "command_send": 0, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "command_resend": 0, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "map_epoch": 94, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "map_full": 0, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "map_inc": 4, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_sessions": 5, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_open": 5, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_close": 0, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_laggy": 0, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_wr": 0, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_rd": 2, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_del": 0 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "oft": { 2024-04-27T00:13:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_objs": 0, 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_kv_pairs": 0, 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_updates": 0, 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_removes": 0 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "osd-slow-ops": { 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_ops_count": 0 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "purge_queue": { 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops": 0, 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops_high_water": 0, 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing": 0, 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_high_water": 0, 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed_ops": 0, 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed": 0, 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_item_in_journal": 0 2024-04-27T00:13:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 3512, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 8875389, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 3512, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 3512, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 8875389, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_bytes": { 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 3249, 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 7939843, 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 3249, 2024-04-27T00:13:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 7939843, 2024-04-27T00:13:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:13:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:13:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_ops": { 2024-04-27T00:13:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:13:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 1024, 2024-04-27T00:13:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:13:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 3249, 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 3249, 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 3249, 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 3249, 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:13:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle": { 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 0, 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 0, 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:13:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle-0x5646ba7906e0": { 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-04-27T00:13:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-04-27T00:13:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-04-27T00:13:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 0, 2024-04-27T00:13:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 0, 2024-04-27T00:13:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:13:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:13:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:13:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:13:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:13:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:13:58.555 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-27T00:13:58.555 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:13:58 smithi080 ceph-mon[32362]: pgmap v780: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-27T00:13:58.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:13:58 smithi040 ceph-mon[28095]: pgmap v780: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-27T00:13:59.365 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-27T00:13:59.365 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-27T00:13:59.365 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-04-27T00:13:59.365 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-04-27T00:13:59.365 DEBUG:tasks.check_counter:Getting stats from f 2024-04-27T00:13:59.365 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-27T00:13:59.613 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-27T00:14:00.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:00 smithi080 ceph-mon[32362]: pgmap v781: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:14:00.592 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:00 smithi103 ceph-mon[32952]: pgmap v781: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:14:00.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:00 smithi040 ceph-mon[28095]: pgmap v781: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:14:01.358 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-27T00:14:01.358 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-04-27T00:14:01.358 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 880, 2024-04-27T00:14:01.358 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 882, 2024-04-27T00:14:01.358 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 232440, 2024-04-27T00:14:01.358 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 279904, 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 6, 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 2, 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.090825288, 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.025641408, 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.026335183, 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.029907918, 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 880, 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.033965477, 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000038597 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.359 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 0 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 741, 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 930, 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 184058, 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 807081, 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 7, 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 4, 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.092968669, 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.028917136, 2024-04-27T00:14:01.360 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.022510368, 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.032848293, 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 929, 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.044708668, 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000048125 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 573664 2024-04-27T00:14:01.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 1137, 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 1134, 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 665952, 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 434108, 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 3, 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 2, 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.147803642, 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.050056888, 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.042573017, 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.041295569, 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:14:01.362 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1133, 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.073856521, 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000065186 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 372288, 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 156176 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 1, 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-04-27T00:14:01.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-MDSRank": { 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 4200, 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.128081485, 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000030495 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-PurgeQueue": { 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:01.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "request": 0, 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "reply": 0, 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "reply_latency": { 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_reply": 0, 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "forward": 0, 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_complete": 0, 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_keys": 0, 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_commit": 0, 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_split": 0, 2024-04-27T00:14:01.365 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_merge": 0, 2024-04-27T00:14:01.366 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes": 0, 2024-04-27T00:14:01.366 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_top": 0, 2024-04-27T00:14:01.366 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_bottom": 0, 2024-04-27T00:14:01.366 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pin_tail": 0, 2024-04-27T00:14:01.366 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pinned": 0, 2024-04-27T00:14:01.366 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_expired": 0, 2024-04-27T00:14:01.366 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_with_caps": 0, 2024-04-27T00:14:01.366 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": 0, 2024-04-27T00:14:01.366 INFO:teuthology.orchestra.run.smithi103.stdout: "subtrees": 1, 2024-04-27T00:14:01.366 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse": 0, 2024-04-27T00:14:01.366 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_hit": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_forward": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_discover": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_dir_fetch": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_remote_ino": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_lock": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "load_cent": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "q": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "exported": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "exported_inodes": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "imported": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "imported_inodes": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_dir_fetch": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_backtrace_fetch": 0, 2024-04-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_peer_discover": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rfiles": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rbytes": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rsnaps": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_fetch": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_set_tag": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_repaired": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_inotable_repaired": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_inodes": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_base_inodes": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dirfrag_rstats": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_file_inodes": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_inode_file_caps": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_revoke": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_grant": 0, 2024-04-27T00:14:01.368 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_trunc": 0, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps": 0, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps_dirty": 0, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_cap_release": 0, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "process_request_cap_release": 0 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache": { 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update": 0, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update_receipt": 0, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_try_discover": 0, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_send_discover": 0, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_handle_discover": 0, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays": 0, 2024-04-27T00:14:01.369 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_delayed": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_enqueuing": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_created": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_enqueued": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_reintegrated": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_migrated": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_processing": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_enqueued": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_prioritized": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_started": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_completed": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_path": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_inode": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_enqueue_scrub": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_exportdir": 0, 2024-04-27T00:14:01.370 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_flush": 0, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragmentdir": 0, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragstats": 0, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_inodestats": 0 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log": { 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "evlrg": 0, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "evadd": 0, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "evex": 0, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "evtrm": 0, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "ev": 4, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "evexg": 0, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "evexd": 0, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "segadd": 0, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "segex": 0, 2024-04-27T00:14:01.371 INFO:teuthology.orchestra.run.smithi103.stdout: "segtrm": 0, 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "seg": 2, 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "segmjr": 2, 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "segexg": 0, 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "segexd": 0, 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "expos": 4194304, 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "wrpos": 4195593, 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "rdpos": 4195283, 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "jlat": { 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "replayed": 4 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.372 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_mem": { 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 3, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "ino+": 3, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "ino-": 0, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "dir": 1, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "dir+": 1, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "dir-": 0, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "dn": 0, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "dn+": 0, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "dn-": 0, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "cap": 0, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "cap+": 0, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "cap-": 0, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "rss": 49236, 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: "heap": 215416 2024-04-27T00:14:01.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_server": { 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_client_request": 0, 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_server_request": 0, 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_request": 0, 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_session": 0, 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_peer_request": 0, 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: "req_create_latency": { 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getattr_latency": { 2024-04-27T00:14:01.374 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getfilelock_latency": { 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "req_link_latency": { 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookup_latency": { 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.375 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookuphash_latency": { 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupino_latency": { 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupname_latency": { 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.376 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.377 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupparent_latency": { 2024-04-27T00:14:01.377 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.377 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.377 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.377 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupsnap_latency": { 2024-04-27T00:14:01.377 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.377 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.377 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.378 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lssnap_latency": { 2024-04-27T00:14:01.378 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.378 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.378 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.378 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mkdir_latency": { 2024-04-27T00:14:01.378 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.379 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.379 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.379 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mknod_latency": { 2024-04-27T00:14:01.379 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.379 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.379 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.379 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mksnap_latency": { 2024-04-27T00:14:01.379 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.379 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.379 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: "req_open_latency": { 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: "req_readdir_latency": { 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rename_latency": { 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.380 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: "req_renamesnap_latency": { 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: "req_snapdiff_latency": { 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmdir_latency": { 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.381 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.382 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmsnap_latency": { 2024-04-27T00:14:01.382 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.382 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.382 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.382 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmxattr_latency": { 2024-04-27T00:14:01.382 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.382 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.382 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.382 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setattr_latency": { 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setdirlayout_latency": { 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setfilelock_latency": { 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.383 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setlayout_latency": { 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setxattr_latency": { 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "req_symlink_latency": { 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "req_unlink_latency": { 2024-04-27T00:14:01.384 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_revoke_eviction": 0, 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_acquisition_throttle": 0, 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getvxattr_latency": { 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sessions": { 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: "session_count": 2, 2024-04-27T00:14:01.385 INFO:teuthology.orchestra.run.smithi103.stdout: "session_add": 2, 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "session_remove": 0, 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_open": 2, 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_stale": 0, 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "total_load": 0, 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "average_load": 0, 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "avg_session_uptime": 2248, 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "mdthresh_evicted": 0 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-04-27T00:14:01.386 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-04-27T00:14:01.387 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-04-27T00:14:01.388 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 190129, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 51, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 8896, 2024-04-27T00:14:01.389 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 171, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 8512, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 13, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter": { 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "op_active": 0, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "op_laggy": 0, 2024-04-27T00:14:01.390 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send": 2488, 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send_bytes": 21, 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "op_resend": 3, 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "op_reply": 2488, 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "op_latency": { 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2488, 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 2.181960176, 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000876993 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "op_inflight": 0, 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "oplen_avg": { 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2488, 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 2489 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "op": 2488, 2024-04-27T00:14:01.391 INFO:teuthology.orchestra.run.smithi103.stdout: "op_r": 2488, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "op_w": 0, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "op_rmw": 0, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "op_pg": 0, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_stat": 1606, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_create": 0, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_read": 880, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_write": 0, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writefull": 0, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writesame": 0, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_append": 0, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_zero": 0, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_truncate": 0, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_delete": 0, 2024-04-27T00:14:01.392 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_mapext": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_sparse_read": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_clonerange": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_getxattr": 1, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_setxattr": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_cmpxattr": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_rmxattr": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_resetxattrs": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_call": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_watch": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_notify": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_src_cmpxattr": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls_filter": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_other": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_active": 0, 2024-04-27T00:14:01.393 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_send": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_resend": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_ping": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_active": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_send": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_resend": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_active": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_send": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_resend": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_active": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_send": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_resend": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "command_active": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "command_send": 0, 2024-04-27T00:14:01.394 INFO:teuthology.orchestra.run.smithi103.stdout: "command_resend": 0, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "map_epoch": 94, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "map_full": 0, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "map_inc": 4, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_sessions": 5, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_open": 5, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_close": 0, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_laggy": 0, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_wr": 0, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_rd": 2, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_del": 0 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "oft": { 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_objs": 0, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_kv_pairs": 0, 2024-04-27T00:14:01.395 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_updates": 0, 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_removes": 0 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: "osd-slow-ops": { 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_ops_count": 0 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: "purge_queue": { 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops": 0, 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops_high_water": 0, 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing": 0, 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_high_water": 0, 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed_ops": 0, 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed": 0, 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_item_in_journal": 0 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.396 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 2754, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 817976, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 2754, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 2754, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 817976, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.397 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_bytes": { 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 2488, 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 1294, 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 2488, 2024-04-27T00:14:01.398 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 1294, 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_ops": { 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 1024, 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-04-27T00:14:01.399 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 2488, 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 2488, 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 2488, 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 2488, 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle": { 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:14:01.400 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 0, 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 0, 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle-0x55feef47c6e0": { 2024-04-27T00:14:01.401 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 0, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 0, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:01.402 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:01.403 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:01.403 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:01.403 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-27T00:14:01.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:01 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:14:01.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:01 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:14:01.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:01 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:14:02.107 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-27T00:14:02.107 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-27T00:14:02.107 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-27T00:14:02.107 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-04-27T00:14:02.107 DEBUG:tasks.check_counter:Getting stats from i 2024-04-27T00:14:02.107 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-27T00:14:02.346 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-27T00:14:02.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:02 smithi080 ceph-mon[32362]: pgmap v782: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:14:02.589 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:02 smithi103 ceph-mon[32952]: pgmap v782: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:14:02.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:02 smithi040 ceph-mon[28095]: pgmap v782: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:14:03.490 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:14:03.490 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:14:03.490 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:03 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:14:03.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:14:03.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:14:03.549 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:03 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:14:03.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:14:03.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:14:03.716 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:03 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' 2024-04-27T00:14:04.046 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-27T00:14:04.046 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-04-27T00:14:04.047 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 1289, 2024-04-27T00:14:04.047 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 915, 2024-04-27T00:14:04.047 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 316701, 2024-04-27T00:14:04.047 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 456275, 2024-04-27T00:14:04.047 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 9, 2024-04-27T00:14:04.047 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 4, 2024-04-27T00:14:04.047 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.093002691, 2024-04-27T00:14:04.047 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.037424349, 2024-04-27T00:14:04.047 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.055932840, 2024-04-27T00:14:04.047 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.005628977, 2024-04-27T00:14:04.047 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 913, 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.061213314, 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000067046 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2121, 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000522820, 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000246 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 0 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 34, 2024-04-27T00:14:04.048 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 223, 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 9498, 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 635306, 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 14, 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 7, 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.025092737, 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.011765671, 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.010378491, 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.000490768, 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 220, 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.026443865, 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000120199 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:14:04.049 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 29, 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000006906, 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000238 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 5136, 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 575664 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 318, 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 310, 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 389658, 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 165917, 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 7, 2024-04-27T00:14:04.050 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 4, 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.043436861, 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.017414403, 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.029774075, 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.006473833, 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 310, 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.020123716, 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000064915 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 29, 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000004851, 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000167 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.051 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 387968, 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 162848 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 1, 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-MDSRank": { 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 53, 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.001646980, 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000031075 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.052 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-PurgeQueue": { 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2, 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000008083, 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000004041 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: "request": 6, 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: "reply": 3, 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: "reply_latency": { 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3, 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000824624, 2024-04-27T00:14:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000274874 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_reply": 0, 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: "forward": 3, 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_complete": 0, 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_keys": 0, 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_commit": 11, 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_split": 0, 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_merge": 0, 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes": 16, 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_top": 0, 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_bottom": 16, 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pin_tail": 0, 2024-04-27T00:14:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pinned": 13, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_expired": 4, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_with_caps": 2, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": 2, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "subtrees": 3, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse": 11, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_hit": 5, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_forward": 1, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_discover": 3, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_dir_fetch": 0, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_remote_ino": 0, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_lock": 0, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "load_cent": 0, 2024-04-27T00:14:04.055 INFO:teuthology.orchestra.run.smithi103.stdout: "q": 0, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "exported": 0, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "exported_inodes": 0, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "imported": 0, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "imported_inodes": 0, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_dir_fetch": 0, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_backtrace_fetch": 0, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_peer_discover": 0, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rfiles": 5, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rbytes": 298, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rsnaps": 0, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_fetch": 0, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_set_tag": 0, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_repaired": 0, 2024-04-27T00:14:04.056 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_inotable_repaired": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_inodes": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_base_inodes": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dirfrag_rstats": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_file_inodes": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_inode_file_caps": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_revoke": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_grant": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_trunc": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps_dirty": 0, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_cap_release": 1, 2024-04-27T00:14:04.057 INFO:teuthology.orchestra.run.smithi103.stdout: "process_request_cap_release": 2 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache": { 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update": 0, 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update_receipt": 9, 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_try_discover": 6, 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_send_discover": 7, 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_handle_discover": 0, 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays": 0, 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_delayed": 0, 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_enqueuing": 0, 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_created": 0, 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_enqueued": 0, 2024-04-27T00:14:04.058 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_reintegrated": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_migrated": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_processing": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_enqueued": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_prioritized": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_started": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_completed": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_path": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_inode": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_enqueue_scrub": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_exportdir": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_flush": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragmentdir": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragstats": 0, 2024-04-27T00:14:04.059 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_inodestats": 0 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log": { 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "evlrg": 0, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "evadd": 4, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "evex": 0, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "evtrm": 0, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "ev": 4, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "evexg": 0, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "evexd": 0, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "segadd": 2, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "segex": 0, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "segtrm": 0, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "seg": 2, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "segmjr": 2, 2024-04-27T00:14:04.060 INFO:teuthology.orchestra.run.smithi103.stdout: "segexg": 0, 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "segexd": 0, 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "expos": 4194304, 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "wrpos": 4195593, 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "rdpos": 0, 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "jlat": { 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3, 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.005245329, 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.001748443 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "replayed": 0 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_mem": { 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 20, 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "ino+": 22, 2024-04-27T00:14:04.061 INFO:teuthology.orchestra.run.smithi103.stdout: "ino-": 2, 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "dir": 19, 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "dir+": 21, 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "dir-": 2, 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "dn": 16, 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "dn+": 20, 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "dn-": 4, 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "cap": 2, 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "cap+": 3, 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "cap-": 1, 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "rss": 54716, 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "heap": 223608 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_server": { 2024-04-27T00:14:04.062 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_client_request": 6, 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_server_request": 0, 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_request": 6, 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_session": 21, 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_peer_request": 0, 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: "req_create_latency": { 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getattr_latency": { 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1, 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000268512, 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000268512 2024-04-27T00:14:04.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getfilelock_latency": { 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "req_link_latency": { 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookup_latency": { 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.064 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookuphash_latency": { 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupino_latency": { 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2, 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000556112, 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000278056 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupname_latency": { 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.065 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupparent_latency": { 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupsnap_latency": { 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lssnap_latency": { 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.066 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.067 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.067 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mkdir_latency": { 2024-04-27T00:14:04.067 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mknod_latency": { 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mksnap_latency": { 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: "req_open_latency": { 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: "req_readdir_latency": { 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rename_latency": { 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.069 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: "req_renamesnap_latency": { 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: "req_snapdiff_latency": { 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmdir_latency": { 2024-04-27T00:14:04.070 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmsnap_latency": { 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmxattr_latency": { 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setattr_latency": { 2024-04-27T00:14:04.071 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setdirlayout_latency": { 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setfilelock_latency": { 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.072 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setlayout_latency": { 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setxattr_latency": { 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: "req_symlink_latency": { 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.073 INFO:teuthology.orchestra.run.smithi103.stdout: "req_unlink_latency": { 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_revoke_eviction": 0, 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_acquisition_throttle": 0, 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getvxattr_latency": { 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sessions": { 2024-04-27T00:14:04.074 INFO:teuthology.orchestra.run.smithi103.stdout: "session_count": 2, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "session_add": 2, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "session_remove": 0, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_open": 2, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_stale": 0, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "total_load": 2, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "average_load": 1, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "avg_session_uptime": 807, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "mdthresh_evicted": 0 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-04-27T00:14:04.075 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-04-27T00:14:04.076 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-04-27T00:14:04.077 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 264631, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 98, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-04-27T00:14:04.078 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 8896, 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 171, 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 81693, 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 330, 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter": { 2024-04-27T00:14:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "op_active": 0, 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "op_laggy": 0, 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send": 39, 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send_bytes": 11712, 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "op_resend": 33, 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "op_reply": 39, 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "op_latency": { 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 39, 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.344993743, 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.008845993 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "op_inflight": 0, 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "oplen_avg": { 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 39, 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 62 2024-04-27T00:14:04.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "op": 39, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "op_r": 0, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "op_w": 39, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "op_rmw": 0, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "op_pg": 0, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_stat": 0, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_create": 10, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_read": 0, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_write": 3, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writefull": 6, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writesame": 0, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_append": 0, 2024-04-27T00:14:04.081 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_zero": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_truncate": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_delete": 6, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_mapext": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_sparse_read": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_clonerange": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_getxattr": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_setxattr": 20, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_cmpxattr": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_rmxattr": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_resetxattrs": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_call": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_watch": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_notify": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_src_cmpxattr": 0, 2024-04-27T00:14:04.082 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls_filter": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_other": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_active": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_send": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_resend": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_ping": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_active": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_send": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_resend": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_active": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_send": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_resend": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_active": 0, 2024-04-27T00:14:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_send": 0, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_resend": 0, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "command_active": 0, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "command_send": 0, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "command_resend": 0, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "map_epoch": 94, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "map_full": 0, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "map_inc": 4, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_sessions": 9, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_open": 9, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_close": 0, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_laggy": 0, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_wr": 16, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_rd": 0, 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_del": 1 2024-04-27T00:14:04.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "oft": { 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_objs": 1, 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_kv_pairs": 0, 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_updates": 0, 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_removes": 0 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "osd-slow-ops": { 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_ops_count": 0 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "purge_queue": { 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops": 0, 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops_high_water": 0, 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing": 0, 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_high_water": 0, 2024-04-27T00:14:04.085 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed_ops": 0, 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed": 0, 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_item_in_journal": 0 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 1637, 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 549309, 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 1637, 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-04-27T00:14:04.086 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 1637, 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 549309, 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_bytes": { 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-04-27T00:14:04.087 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 39, 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 11712, 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 39, 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 11712, 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_ops": { 2024-04-27T00:14:04.088 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:14:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 1024, 2024-04-27T00:14:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:14:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-04-27T00:14:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-04-27T00:14:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:14:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-04-27T00:14:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 39, 2024-04-27T00:14:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 39, 2024-04-27T00:14:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 39, 2024-04-27T00:14:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 39, 2024-04-27T00:14:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:14:04.089 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle": { 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-04-27T00:14:04.090 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 0, 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 0, 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle-0x562488f84500": { 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 4, 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 1289, 2024-04-27T00:14:04.091 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:14:04.092 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 4, 2024-04-27T00:14:04.092 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-04-27T00:14:04.092 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-04-27T00:14:04.092 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 3, 2024-04-27T00:14:04.092 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 1289, 2024-04-27T00:14:04.092 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:14:04.092 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:04.092 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:04.092 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:04.092 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:04.092 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:04.092 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-27T00:14:04.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:04 smithi080 ceph-mon[32362]: pgmap v783: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-27T00:14:04.818 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:04 smithi103 ceph-mon[32952]: pgmap v783: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-27T00:14:04.846 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-27T00:14:04.846 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-27T00:14:04.846 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-04-27T00:14:04.846 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=9 2024-04-27T00:14:04.846 DEBUG:tasks.check_counter:Getting stats from l 2024-04-27T00:14:04.846 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-27T00:14:04.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:04 smithi040 ceph-mon[28095]: pgmap v783: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-27T00:14:05.087 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/mon.c/config 2024-04-27T00:14:06.732 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:06 smithi103 ceph-mon[32952]: pgmap v784: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:14:06.760 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-27T00:14:06.760 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-04-27T00:14:06.760 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 0, 2024-04-27T00:14:06.760 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 2, 2024-04-27T00:14:06.760 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 0, 2024-04-27T00:14:06.760 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 0, 2024-04-27T00:14:06.760 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 4, 2024-04-27T00:14:06.760 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 0, 2024-04-27T00:14:06.760 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.000911457, 2024-04-27T00:14:06.760 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.000000000, 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.000105528, 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-27T00:14:06.761 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 0 2024-04-27T00:14:06.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:06.762 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-04-27T00:14:06.762 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 7, 2024-04-27T00:14:06.762 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 200, 2024-04-27T00:14:06.762 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 3472, 2024-04-27T00:14:06.762 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 271216, 2024-04-27T00:14:06.762 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 5, 2024-04-27T00:14:06.762 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 2, 2024-04-27T00:14:06.762 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.013932047, 2024-04-27T00:14:06.762 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.009733611, 2024-04-27T00:14:06.762 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.001812761, 2024-04-27T00:14:06.762 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 198, 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.017546659, 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000088619 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 271216 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-04-27T00:14:06.763 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 258, 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 256, 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 353408, 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 157840, 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 2, 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 1, 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.041405776, 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.016669528, 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.014932894, 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.006867164, 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 255, 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.013585421, 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000053276 2024-04-27T00:14:06.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 353408, 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 157840 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 0, 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-04-27T00:14:06.765 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-04-27T00:14:06.766 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-27T00:14:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 147898, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 27, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-04-27T00:14:06.768 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 928, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 2, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-04-27T00:14:06.769 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-04-27T00:14:06.770 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-04-27T00:14:06.770 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-27T00:14:06.770 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-27T00:14:06.770 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-04-27T00:14:06.770 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-04-27T00:14:06.770 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-04-27T00:14:06.770 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 261, 2024-04-27T00:14:06.770 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 316976, 2024-04-27T00:14:06.770 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-04-27T00:14:06.770 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 261, 2024-04-27T00:14:06.770 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-04-27T00:14:06.770 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-04-27T00:14:06.771 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 261, 2024-04-27T00:14:06.771 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 316976, 2024-04-27T00:14:06.771 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-04-27T00:14:06.771 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-04-27T00:14:06.771 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-04-27T00:14:06.771 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-04-27T00:14:06.771 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:06.771 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-27T00:14:06.771 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-27T00:14:06.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:06 smithi080 ceph-mon[32362]: pgmap v784: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:14:06.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:06 smithi040 ceph-mon[28095]: pgmap v784: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:14:07.617 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-27T00:14:07.617 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-27T00:14:07.617 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-04-27T00:14:07.617 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-04-27T00:14:07.617 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-27T00:14:07.630 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-27T00:14:07.630 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-27T00:14:07.630 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T00:14:07.693 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-04-27T00:14:07.693 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T00:14:07.751 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T00:14:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.975+0000 7facc4a00640 1 -- 172.21.15.40:0/2812217930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc0105b20 msgr2=0x7facc0107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.975+0000 7facc4a00640 1 --2- 172.21.15.40:0/2812217930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc0105b20 0x7facc0107f10 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7faca8009900 tx=0x7faca802a400 comp rx=0 tx=0).stop 2024-04-27T00:14:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.975+0000 7facc4a00640 1 -- 172.21.15.40:0/2812217930 shutdown_connections 2024-04-27T00:14:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.975+0000 7facc4a00640 1 --2- 172.21.15.40:0/2812217930 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7facc0108450 0x7facc010a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.975+0000 7facc4a00640 1 --2- 172.21.15.40:0/2812217930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc0105b20 0x7facc0107f10 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.975+0000 7facc4a00640 1 --2- 172.21.15.40:0/2812217930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facc01031f0 0x7facc01055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.975+0000 7facc4a00640 1 -- 172.21.15.40:0/2812217930 >> 172.21.15.40:0/2812217930 conn(0x7facc00fcbc0 msgr2=0x7facc00ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.975+0000 7facc4a00640 1 -- 172.21.15.40:0/2812217930 shutdown_connections 2024-04-27T00:14:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.975+0000 7facc4a00640 1 -- 172.21.15.40:0/2812217930 wait complete. 2024-04-27T00:14:07.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.976+0000 7facc4a00640 1 Processor -- start 2024-04-27T00:14:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.976+0000 7facc4a00640 1 -- start start 2024-04-27T00:14:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.977+0000 7facc4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc01031f0 0x7facc0198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.977+0000 7facc4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facc0105b20 0x7facc0198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.977+0000 7facc4a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7facc0108450 0x7facc019fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.977+0000 7facc4a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7facc010e2e0 con 0x7facc01031f0 2024-04-27T00:14:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.977+0000 7facc4a00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7facc010e160 con 0x7facc0108450 2024-04-27T00:14:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.977+0000 7facc4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7facc010e460 con 0x7facc0105b20 2024-04-27T00:14:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.977+0000 7facbe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc01031f0 0x7facc0198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.977+0000 7facbe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc01031f0 0x7facc0198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54058/0 (socket says 172.21.15.40:54058) 2024-04-27T00:14:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.977+0000 7facbe800640 1 -- 172.21.15.40:0/1173959455 learned_addr learned my addr 172.21.15.40:0/1173959455 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.977+0000 7facbde00640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facc0105b20 0x7facc0198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.977+0000 7facbf200640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7facc0108450 0x7facc019fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:07.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.978+0000 7facbe800640 1 -- 172.21.15.40:0/1173959455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facc0105b20 msgr2=0x7facc0198ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:07.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.978+0000 7facbe800640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facc0105b20 0x7facc0198ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.978+0000 7facbe800640 1 -- 172.21.15.40:0/1173959455 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7facc0108450 msgr2=0x7facc019fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.978+0000 7facbe800640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7facc0108450 0x7facc019fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.978+0000 7facbe800640 1 -- 172.21.15.40:0/1173959455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc01a0550 con 0x7facc01031f0 2024-04-27T00:14:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.978+0000 7facbe800640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc01031f0 0x7facc0198980 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7facb000d950 tx=0x7facb000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.978+0000 7facb7400640 1 -- 172.21.15.40:0/1173959455 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb0017070 con 0x7facc01031f0 2024-04-27T00:14:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.978+0000 7facb7400640 1 -- 172.21.15.40:0/1173959455 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facb0009c30 con 0x7facc01031f0 2024-04-27T00:14:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.978+0000 7facc4a00640 1 -- 172.21.15.40:0/1173959455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc01a0840 con 0x7facc01031f0 2024-04-27T00:14:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.978+0000 7facb7400640 1 -- 172.21.15.40:0/1173959455 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb000f620 con 0x7facc01031f0 2024-04-27T00:14:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.979+0000 7facc4a00640 1 -- 172.21.15.40:0/1173959455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc01a0db0 con 0x7facc01031f0 2024-04-27T00:14:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.980+0000 7facc4a00640 1 -- 172.21.15.40:0/1173959455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facc00fe6b0 con 0x7facc01031f0 2024-04-27T00:14:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.983+0000 7facb7400640 1 -- 172.21.15.40:0/1173959455 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7facb0010460 con 0x7facc01031f0 2024-04-27T00:14:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.984+0000 7facb7400640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7faca40a3cc0 0x7faca40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:07.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.984+0000 7facb7400640 1 -- 172.21.15.40:0/1173959455 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7facb00d0430 con 0x7facc01031f0 2024-04-27T00:14:07.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.984+0000 7facbde00640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7faca40a3cc0 0x7faca40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:07.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.985+0000 7facbde00640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7faca40a3cc0 0x7faca40a6180 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7faca8005ab0 tx=0x7faca8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:07.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:07.989+0000 7facb7400640 1 -- 172.21.15.40:0/1173959455 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7facb0009de0 con 0x7facc01031f0 2024-04-27T00:14:08.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.197+0000 7facc4a00640 1 -- 172.21.15.40:0/1173959455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7facc0063530 con 0x7facc01031f0 2024-04-27T00:14:08.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.200+0000 7facb7400640 1 -- 172.21.15.40:0/1173959455 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24727 (secure 0 0 0) 0x7facb009c2c0 con 0x7facc01031f0 2024-04-27T00:14:08.201 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:14:08.202 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":94,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-27T00:01:46.569457+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9},{"from":5,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T00:14:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.203+0000 7facc4a00640 1 -- 172.21.15.40:0/1173959455 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7faca40a3cc0 msgr2=0x7faca40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.203+0000 7facc4a00640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7faca40a3cc0 0x7faca40a6180 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7faca8005ab0 tx=0x7faca8005a20 comp rx=0 tx=0).stop 2024-04-27T00:14:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.203+0000 7facc4a00640 1 -- 172.21.15.40:0/1173959455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc01031f0 msgr2=0x7facc0198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.203+0000 7facc4a00640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc01031f0 0x7facc0198980 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7facb000d950 tx=0x7facb000de10 comp rx=0 tx=0).stop 2024-04-27T00:14:08.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.204+0000 7facc4a00640 1 -- 172.21.15.40:0/1173959455 shutdown_connections 2024-04-27T00:14:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.204+0000 7facc4a00640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7facc0108450 0x7facc019fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.204+0000 7facc4a00640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7faca40a3cc0 0x7faca40a6180 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.204+0000 7facc4a00640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facc0105b20 0x7facc0198ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.204+0000 7facc4a00640 1 --2- 172.21.15.40:0/1173959455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc01031f0 0x7facc0198980 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.204+0000 7facc4a00640 1 -- 172.21.15.40:0/1173959455 >> 172.21.15.40:0/1173959455 conn(0x7facc00fcbc0 msgr2=0x7facc0103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.204+0000 7facc4a00640 1 -- 172.21.15.40:0/1173959455 shutdown_connections 2024-04-27T00:14:08.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.204+0000 7facc4a00640 1 -- 172.21.15.40:0/1173959455 wait complete. 2024-04-27T00:14:08.226 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:53:32.708818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:58:48.506601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:58:49.540141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:58:53.179820+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-27T00:14:08.227 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-27T00:14:08.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 -- 172.21.15.40:0/3504203332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8148101770 msgr2=0x7f8148101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:08.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 --2- 172.21.15.40:0/3504203332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8148101770 0x7f8148101b90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f813c009900 tx=0x7f813c02a410 comp rx=0 tx=0).stop 2024-04-27T00:14:08.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 -- 172.21.15.40:0/3504203332 shutdown_connections 2024-04-27T00:14:08.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 --2- 172.21.15.40:0/3504203332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8148103320 0x7f8148109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 --2- 172.21.15.40:0/3504203332 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8148102960 0x7f8148102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 --2- 172.21.15.40:0/3504203332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8148101770 0x7f8148101b90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 -- 172.21.15.40:0/3504203332 >> 172.21.15.40:0/3504203332 conn(0x7f81480fcce0 msgr2=0x7f81480ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 -- 172.21.15.40:0/3504203332 shutdown_connections 2024-04-27T00:14:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 -- 172.21.15.40:0/3504203332 wait complete. 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 Processor -- start 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 -- start start 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8148101770 0x7f814819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8148102960 0x7f814819b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.439+0000 7f814e000640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8148103320 0x7f81481a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814e000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f814810e3e0 con 0x7f8148101770 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814e000640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f814810e260 con 0x7f8148103320 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814e000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f814810e560 con 0x7f8148102960 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8148101770 0x7f814819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8148101770 0x7f814819ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54066/0 (socket says 172.21.15.40:54066) 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814cc00640 1 -- 172.21.15.40:0/2484028195 learned_addr learned my addr 172.21.15.40:0/2484028195 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f8147e00640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8148102960 0x7f814819b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814d600640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8148103320 0x7f81481a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814cc00640 1 -- 172.21.15.40:0/2484028195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8148102960 msgr2=0x7f814819b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814cc00640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8148102960 0x7f814819b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814cc00640 1 -- 172.21.15.40:0/2484028195 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8148103320 msgr2=0x7f81481a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814cc00640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8148103320 0x7f81481a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814cc00640 1 -- 172.21.15.40:0/2484028195 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81481a28c0 con 0x7f8148101770 2024-04-27T00:14:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.440+0000 7f814d600640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8148103320 0x7f81481a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-27T00:14:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.441+0000 7f814cc00640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8148101770 0x7f814819ac80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f813c031500 tx=0x7f813c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.441+0000 7f8145600640 1 -- 172.21.15.40:0/2484028195 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813c03f070 con 0x7f8148101770 2024-04-27T00:14:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.441+0000 7f8145600640 1 -- 172.21.15.40:0/2484028195 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f813c031c50 con 0x7f8148101770 2024-04-27T00:14:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.441+0000 7f8145600640 1 -- 172.21.15.40:0/2484028195 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813c035030 con 0x7f8148101770 2024-04-27T00:14:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.441+0000 7f814e000640 1 -- 172.21.15.40:0/2484028195 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81481a2b50 con 0x7f8148101770 2024-04-27T00:14:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.441+0000 7f814e000640 1 -- 172.21.15.40:0/2484028195 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81481a2ff0 con 0x7f8148101770 2024-04-27T00:14:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.442+0000 7f814e000640 1 -- 172.21.15.40:0/2484028195 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81481a3220 con 0x7f8148101770 2024-04-27T00:14:08.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.445+0000 7f8145600640 1 -- 172.21.15.40:0/2484028195 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f813c042070 con 0x7f8148101770 2024-04-27T00:14:08.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.446+0000 7f8145600640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f81300a3c40 0x7f81300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:08.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.446+0000 7f8145600640 1 -- 172.21.15.40:0/2484028195 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f813c0f11f0 con 0x7f8148101770 2024-04-27T00:14:08.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.446+0000 7f8147e00640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f81300a3c40 0x7f81300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:08.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.447+0000 7f8147e00640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f81300a3c40 0x7f81300a6100 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f81340095f0 tx=0x7f8134005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:08.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.451+0000 7f8145600640 1 -- 172.21.15.40:0/2484028195 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f813c0bd670 con 0x7f8148101770 2024-04-27T00:14:08.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.657+0000 7f814e000640 1 -- 172.21.15.40:0/2484028195 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f81481a3510 con 0x7f8148101770 2024-04-27T00:14:08.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.658+0000 7f8145600640 1 -- 172.21.15.40:0/2484028195 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f813c0bcf70 con 0x7f8148101770 2024-04-27T00:14:08.658 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-04-27T00:14:08.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.662+0000 7f814e000640 1 -- 172.21.15.40:0/2484028195 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f81300a3c40 msgr2=0x7f81300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:08.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.662+0000 7f814e000640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f81300a3c40 0x7f81300a6100 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f81340095f0 tx=0x7f8134005f90 comp rx=0 tx=0).stop 2024-04-27T00:14:08.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.662+0000 7f814e000640 1 -- 172.21.15.40:0/2484028195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8148101770 msgr2=0x7f814819ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:08.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.662+0000 7f814e000640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8148101770 0x7f814819ac80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f813c031500 tx=0x7f813c003c10 comp rx=0 tx=0).stop 2024-04-27T00:14:08.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.662+0000 7f814e000640 1 -- 172.21.15.40:0/2484028195 shutdown_connections 2024-04-27T00:14:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.662+0000 7f814e000640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f8148103320 0x7f81481a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.662+0000 7f814e000640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f81300a3c40 0x7f81300a6100 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.662+0000 7f814e000640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8148102960 0x7f814819b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.662+0000 7f814e000640 1 --2- 172.21.15.40:0/2484028195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8148101770 0x7f814819ac80 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.662+0000 7f814e000640 1 -- 172.21.15.40:0/2484028195 >> 172.21.15.40:0/2484028195 conn(0x7f81480fcce0 msgr2=0x7f814810a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.662+0000 7f814e000640 1 -- 172.21.15.40:0/2484028195 shutdown_connections 2024-04-27T00:14:08.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.662+0000 7f814e000640 1 -- 172.21.15.40:0/2484028195 wait complete. 2024-04-27T00:14:08.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-27T00:14:08.754 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:08 smithi040 ceph-mon[28095]: pgmap v785: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:14:08.755 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:08 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1173959455' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:14:08.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:08 smithi080 ceph-mon[32362]: pgmap v785: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:14:08.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:08 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1173959455' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:14:08.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:08 smithi103 ceph-mon[32952]: pgmap v785: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:14:08.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:08 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1173959455' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:14:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.902+0000 7fea2f200640 1 -- 172.21.15.40:0/3900660409 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea30109fc0 msgr2=0x7fea3010a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.902+0000 7fea2f200640 1 --2- 172.21.15.40:0/3900660409 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea30109fc0 0x7fea3010a440 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fea18009900 tx=0x7fea1802a3f0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.902+0000 7fea2f200640 1 -- 172.21.15.40:0/3900660409 shutdown_connections 2024-04-27T00:14:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.902+0000 7fea2f200640 1 --2- 172.21.15.40:0/3900660409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea3010a980 0x7fea301112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.902+0000 7fea2f200640 1 --2- 172.21.15.40:0/3900660409 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea30109fc0 0x7fea3010a440 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.902+0000 7fea2f200640 1 --2- 172.21.15.40:0/3900660409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30108e70 0x7fea30109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.902+0000 7fea2f200640 1 -- 172.21.15.40:0/3900660409 >> 172.21.15.40:0/3900660409 conn(0x7fea30077e00 msgr2=0x7fea3007a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.902+0000 7fea2f200640 1 -- 172.21.15.40:0/3900660409 shutdown_connections 2024-04-27T00:14:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.902+0000 7fea2f200640 1 -- 172.21.15.40:0/3900660409 wait complete. 2024-04-27T00:14:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.903+0000 7fea2f200640 1 Processor -- start 2024-04-27T00:14:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.904+0000 7fea2f200640 1 -- start start 2024-04-27T00:14:08.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.904+0000 7fea2f200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea30108e70 0x7fea3019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:08.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.904+0000 7fea2f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30109fc0 0x7fea3019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:08.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.904+0000 7fea2f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea3010a980 0x7fea301a6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:08.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.904+0000 7fea2f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fea301149d0 con 0x7fea30109fc0 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.904+0000 7fea2f200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fea30114850 con 0x7fea30108e70 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.904+0000 7fea2f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea30114b50 con 0x7fea3010a980 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.905+0000 7fea2de00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea30108e70 0x7fea3019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.905+0000 7fea27e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30109fc0 0x7fea3019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.905+0000 7fea2de00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea30108e70 0x7fea3019f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:39410/0 (socket says 172.21.15.40:39410) 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.905+0000 7fea2de00640 1 -- 172.21.15.40:0/1775148276 learned_addr learned my addr 172.21.15.40:0/1775148276 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.905+0000 7fea2e800640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea3010a980 0x7fea301a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.905+0000 7fea2de00640 1 -- 172.21.15.40:0/1775148276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea3010a980 msgr2=0x7fea301a6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.905+0000 7fea2de00640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea3010a980 0x7fea301a6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.905+0000 7fea2de00640 1 -- 172.21.15.40:0/1775148276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30109fc0 msgr2=0x7fea3019f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.905+0000 7fea2de00640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30109fc0 0x7fea3019f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.905+0000 7fea2de00640 1 -- 172.21.15.40:0/1775148276 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea301a6cf0 con 0x7fea30108e70 2024-04-27T00:14:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.905+0000 7fea2de00640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea30108e70 0x7fea3019f120 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fea2000e9f0 tx=0x7fea2000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.906+0000 7fea26a00640 1 -- 172.21.15.40:0/1775148276 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2000cd80 con 0x7fea30108e70 2024-04-27T00:14:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.906+0000 7fea2f200640 1 -- 172.21.15.40:0/1775148276 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea301a6fe0 con 0x7fea30108e70 2024-04-27T00:14:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.906+0000 7fea26a00640 1 -- 172.21.15.40:0/1775148276 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea200108e0 con 0x7fea30108e70 2024-04-27T00:14:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.906+0000 7fea26a00640 1 -- 172.21.15.40:0/1775148276 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea20018980 con 0x7fea30108e70 2024-04-27T00:14:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.906+0000 7fea2f200640 1 -- 172.21.15.40:0/1775148276 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea301a7450 con 0x7fea30108e70 2024-04-27T00:14:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.907+0000 7fea2f200640 1 -- 172.21.15.40:0/1775148276 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea301102a0 con 0x7fea30108e70 2024-04-27T00:14:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.909+0000 7fea26a00640 1 -- 172.21.15.40:0/1775148276 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fea20010430 con 0x7fea30108e70 2024-04-27T00:14:08.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.911+0000 7fea26a00640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fea140a3cc0 0x7fea140a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:08.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.911+0000 7fea26a00640 1 -- 172.21.15.40:0/1775148276 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fea200d0e60 con 0x7fea30108e70 2024-04-27T00:14:08.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.911+0000 7fea27e00640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fea140a3cc0 0x7fea140a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.912+0000 7fea27e00640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fea140a3cc0 0x7fea140a6180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fea18009230 tx=0x7fea180314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:08.913+0000 7fea26a00640 1 -- 172.21.15.40:0/1775148276 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fea2009cc90 con 0x7fea30108e70 2024-04-27T00:14:09.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.121+0000 7fea2f200640 1 -- 172.21.15.40:0/1775148276 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fea3002db70 con 0x7fea30108e70 2024-04-27T00:14:09.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.122+0000 7fea26a00640 1 -- 172.21.15.40:0/1775148276 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fea200a0f90 con 0x7fea30108e70 2024-04-27T00:14:09.121 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-27T00:14:09.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.125+0000 7fea2f200640 1 -- 172.21.15.40:0/1775148276 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fea140a3cc0 msgr2=0x7fea140a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:09.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.125+0000 7fea2f200640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fea140a3cc0 0x7fea140a6180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fea18009230 tx=0x7fea180314a0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.125+0000 7fea2f200640 1 -- 172.21.15.40:0/1775148276 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea30108e70 msgr2=0x7fea3019f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:09.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.125+0000 7fea2f200640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea30108e70 0x7fea3019f120 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fea2000e9f0 tx=0x7fea2000eeb0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.125+0000 7fea2f200640 1 -- 172.21.15.40:0/1775148276 shutdown_connections 2024-04-27T00:14:09.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.125+0000 7fea2f200640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fea140a3cc0 0x7fea140a6180 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.125+0000 7fea2f200640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea3010a980 0x7fea301a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.125+0000 7fea2f200640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea30109fc0 0x7fea3019f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.125+0000 7fea2f200640 1 --2- 172.21.15.40:0/1775148276 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fea30108e70 0x7fea3019f120 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.125+0000 7fea2f200640 1 -- 172.21.15.40:0/1775148276 >> 172.21.15.40:0/1775148276 conn(0x7fea30077e00 msgr2=0x7fea3010d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:09.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.125+0000 7fea2f200640 1 -- 172.21.15.40:0/1775148276 shutdown_connections 2024-04-27T00:14:09.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.125+0000 7fea2f200640 1 -- 172.21.15.40:0/1775148276 wait complete. 2024-04-27T00:14:09.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-27T00:14:09.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.364+0000 7fd698800640 1 -- 172.21.15.40:0/2224773241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd690106720 msgr2=0x7fd69010cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:09.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.364+0000 7fd698800640 1 --2- 172.21.15.40:0/2224773241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd690106720 0x7fd69010cfb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd68c009be0 tx=0x7fd68c02a6c0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.364+0000 7fd698800640 1 -- 172.21.15.40:0/2224773241 shutdown_connections 2024-04-27T00:14:09.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.364+0000 7fd698800640 1 --2- 172.21.15.40:0/2224773241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd690106720 0x7fd69010cfb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.364+0000 7fd698800640 1 --2- 172.21.15.40:0/2224773241 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd690105d60 0x7fd6901061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.364+0000 7fd698800640 1 --2- 172.21.15.40:0/2224773241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690104b70 0x7fd690104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.364+0000 7fd698800640 1 -- 172.21.15.40:0/2224773241 >> 172.21.15.40:0/2224773241 conn(0x7fd690100120 msgr2=0x7fd690102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:09.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.364+0000 7fd698800640 1 -- 172.21.15.40:0/2224773241 shutdown_connections 2024-04-27T00:14:09.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.364+0000 7fd698800640 1 -- 172.21.15.40:0/2224773241 wait complete. 2024-04-27T00:14:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.365+0000 7fd698800640 1 Processor -- start 2024-04-27T00:14:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.365+0000 7fd698800640 1 -- start start 2024-04-27T00:14:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd698800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd690104b70 0x7fd69019ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd698800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690105d60 0x7fd69019b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd698800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd690106720 0x7fd6901a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd698800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd690110680 con 0x7fd690105d60 2024-04-27T00:14:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd698800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fd690110500 con 0x7fd690104b70 2024-04-27T00:14:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd698800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd690110800 con 0x7fd690106720 2024-04-27T00:14:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd695400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690105d60 0x7fd69019b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd695e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd690104b70 0x7fd69019ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd695400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690105d60 0x7fd69019b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54098/0 (socket says 172.21.15.40:54098) 2024-04-27T00:14:09.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd695e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd690104b70 0x7fd69019ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:39420/0 (socket says 172.21.15.40:39420) 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd695400640 1 -- 172.21.15.40:0/2325469939 learned_addr learned my addr 172.21.15.40:0/2325469939 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd696800640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd690106720 0x7fd6901a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd695400640 1 -- 172.21.15.40:0/2325469939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd690106720 msgr2=0x7fd6901a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd695400640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd690106720 0x7fd6901a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd695400640 1 -- 172.21.15.40:0/2325469939 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd690104b70 msgr2=0x7fd69019ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd695400640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd690104b70 0x7fd69019ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.366+0000 7fd695400640 1 -- 172.21.15.40:0/2325469939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6901a2960 con 0x7fd690105d60 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.367+0000 7fd695400640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690105d60 0x7fd69019b280 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd68000ea60 tx=0x7fd68000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.367+0000 7fd68aa00640 1 -- 172.21.15.40:0/2325469939 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68000cd20 con 0x7fd690105d60 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.367+0000 7fd698800640 1 -- 172.21.15.40:0/2325469939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6901a2c50 con 0x7fd690105d60 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.367+0000 7fd68aa00640 1 -- 172.21.15.40:0/2325469939 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6800035f0 con 0x7fd690105d60 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.367+0000 7fd68aa00640 1 -- 172.21.15.40:0/2325469939 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd680010690 con 0x7fd690105d60 2024-04-27T00:14:09.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.367+0000 7fd698800640 1 -- 172.21.15.40:0/2325469939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6901a31a0 con 0x7fd690105d60 2024-04-27T00:14:09.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.368+0000 7fd698800640 1 -- 172.21.15.40:0/2325469939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd69010bf60 con 0x7fd690105d60 2024-04-27T00:14:09.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.371+0000 7fd68aa00640 1 -- 172.21.15.40:0/2325469939 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd6800037a0 con 0x7fd690105d60 2024-04-27T00:14:09.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.372+0000 7fd68aa00640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd6780a3c40 0x7fd6780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:09.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.372+0000 7fd68aa00640 1 -- 172.21.15.40:0/2325469939 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7fd6800d0be0 con 0x7fd690105d60 2024-04-27T00:14:09.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.372+0000 7fd695e00640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd6780a3c40 0x7fd6780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:09.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.373+0000 7fd695e00640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd6780a3c40 0x7fd6780a6100 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd690105bc0 tx=0x7fd684009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:09.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.376+0000 7fd68aa00640 1 -- 172.21.15.40:0/2325469939 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd68009d060 con 0x7fd690105d60 2024-04-27T00:14:09.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.585+0000 7fd698800640 1 -- 172.21.15.40:0/2325469939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd69002dad0 con 0x7fd690105d60 2024-04-27T00:14:09.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.586+0000 7fd68aa00640 1 -- 172.21.15.40:0/2325469939 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fd68009c960 con 0x7fd690105d60 2024-04-27T00:14:09.586 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-04-27T00:14:09.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.589+0000 7fd698800640 1 -- 172.21.15.40:0/2325469939 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd6780a3c40 msgr2=0x7fd6780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:09.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.589+0000 7fd698800640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd6780a3c40 0x7fd6780a6100 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd690105bc0 tx=0x7fd684009260 comp rx=0 tx=0).stop 2024-04-27T00:14:09.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.589+0000 7fd698800640 1 -- 172.21.15.40:0/2325469939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690105d60 msgr2=0x7fd69019b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:09.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.589+0000 7fd698800640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690105d60 0x7fd69019b280 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd68000ea60 tx=0x7fd68000ee60 comp rx=0 tx=0).stop 2024-04-27T00:14:09.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.589+0000 7fd698800640 1 -- 172.21.15.40:0/2325469939 shutdown_connections 2024-04-27T00:14:09.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.589+0000 7fd698800640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fd6780a3c40 0x7fd6780a6100 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.589+0000 7fd698800640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd690106720 0x7fd6901a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.589+0000 7fd698800640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd690105d60 0x7fd69019b280 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.589+0000 7fd698800640 1 --2- 172.21.15.40:0/2325469939 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fd690104b70 0x7fd69019ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.589+0000 7fd698800640 1 -- 172.21.15.40:0/2325469939 >> 172.21.15.40:0/2325469939 conn(0x7fd690100120 msgr2=0x7fd690109130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:09.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.589+0000 7fd698800640 1 -- 172.21.15.40:0/2325469939 shutdown_connections 2024-04-27T00:14:09.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.589+0000 7fd698800640 1 -- 172.21.15.40:0/2325469939 wait complete. 2024-04-27T00:14:09.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-27T00:14:09.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:09 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2484028195' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:14:09.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:09 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1775148276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:14:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.828+0000 7f1c7dc00640 1 -- 172.21.15.40:0/289851350 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1c78102940 msgr2=0x7f1c78102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.828+0000 7f1c7dc00640 1 --2- 172.21.15.40:0/289851350 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1c78102940 0x7f1c78102dc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1c60009900 tx=0x7f1c6002a3f0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.829+0000 7f1c7dc00640 1 -- 172.21.15.40:0/289851350 shutdown_connections 2024-04-27T00:14:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.829+0000 7f1c7dc00640 1 --2- 172.21.15.40:0/289851350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c78103300 0x7f1c78109b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.829+0000 7f1c7dc00640 1 --2- 172.21.15.40:0/289851350 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1c78102940 0x7f1c78102dc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.829+0000 7f1c7dc00640 1 --2- 172.21.15.40:0/289851350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c78101750 0x7f1c78101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.829+0000 7f1c7dc00640 1 -- 172.21.15.40:0/289851350 >> 172.21.15.40:0/289851350 conn(0x7f1c780fcce0 msgr2=0x7f1c780ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.829+0000 7f1c7dc00640 1 -- 172.21.15.40:0/289851350 shutdown_connections 2024-04-27T00:14:09.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.829+0000 7f1c7dc00640 1 -- 172.21.15.40:0/289851350 wait complete. 2024-04-27T00:14:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.830+0000 7f1c7dc00640 1 Processor -- start 2024-04-27T00:14:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.830+0000 7f1c7dc00640 1 -- start start 2024-04-27T00:14:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c7dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c78101750 0x7f1c781056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c7dc00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1c78102940 0x7f1c78103d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c7dc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c78103300 0x7f1c78104260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c7dc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7810d280 con 0x7f1c78101750 2024-04-27T00:14:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c7dc00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7810d100 con 0x7f1c78102940 2024-04-27T00:14:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c7dc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7810d400 con 0x7f1c78103300 2024-04-27T00:14:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c77e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c78101750 0x7f1c781056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c77e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c78101750 0x7f1c781056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54112/0 (socket says 172.21.15.40:54112) 2024-04-27T00:14:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c77e00640 1 -- 172.21.15.40:0/2944920596 learned_addr learned my addr 172.21.15.40:0/2944920596 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c6fe00640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1c78102940 0x7f1c78103d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c7d200640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c78103300 0x7f1c78104260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c77e00640 1 -- 172.21.15.40:0/2944920596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c78103300 msgr2=0x7f1c78104260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c77e00640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c78103300 0x7f1c78104260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c77e00640 1 -- 172.21.15.40:0/2944920596 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1c78102940 msgr2=0x7f1c78103d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c77e00640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1c78102940 0x7f1c78103d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.831+0000 7f1c77e00640 1 -- 172.21.15.40:0/2944920596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c78104af0 con 0x7f1c78101750 2024-04-27T00:14:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.832+0000 7f1c77e00640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c78101750 0x7f1c781056e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800e9c0 tx=0x7f1c6800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.832+0000 7f1c75600640 1 -- 172.21.15.40:0/2944920596 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6800cd80 con 0x7f1c78101750 2024-04-27T00:14:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.832+0000 7f1c75600640 1 -- 172.21.15.40:0/2944920596 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c680035f0 con 0x7f1c78101750 2024-04-27T00:14:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.832+0000 7f1c75600640 1 -- 172.21.15.40:0/2944920596 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68010690 con 0x7f1c78101750 2024-04-27T00:14:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.832+0000 7f1c7dc00640 1 -- 172.21.15.40:0/2944920596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c781a2bf0 con 0x7f1c78101750 2024-04-27T00:14:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.832+0000 7f1c7dc00640 1 -- 172.21.15.40:0/2944920596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c781a30c0 con 0x7f1c78101750 2024-04-27T00:14:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.833+0000 7f1c7dc00640 1 -- 172.21.15.40:0/2944920596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c78069ac0 con 0x7f1c78101750 2024-04-27T00:14:09.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.835+0000 7f1c75600640 1 -- 172.21.15.40:0/2944920596 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1c68004000 con 0x7f1c78101750 2024-04-27T00:14:09.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.837+0000 7f1c75600640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1c5c0a3c40 0x7f1c5c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:09.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.837+0000 7f1c75600640 1 -- 172.21.15.40:0/2944920596 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f1c680d0ef0 con 0x7f1c78101750 2024-04-27T00:14:09.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.837+0000 7f1c6fe00640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1c5c0a3c40 0x7f1c5c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:09.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.838+0000 7f1c6fe00640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1c5c0a3c40 0x7f1c5c0a6100 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1c60000c00 tx=0x7f1c600314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:09.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:09.841+0000 7f1c75600640 1 -- 172.21.15.40:0/2944920596 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1c6809d370 con 0x7f1c78101750 2024-04-27T00:14:09.846 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:09 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2484028195' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:14:09.846 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:09 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1775148276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:14:09.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:09 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2484028195' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:14:09.966 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:09 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1775148276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:14:10.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.050+0000 7f1c7dc00640 1 -- 172.21.15.40:0/2944920596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1c781a3470 con 0x7f1c78101750 2024-04-27T00:14:10.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.051+0000 7f1c75600640 1 -- 172.21.15.40:0/2944920596 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f1c6809cc70 con 0x7f1c78101750 2024-04-27T00:14:10.051 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-04-27T00:14:10.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.054+0000 7f1c7dc00640 1 -- 172.21.15.40:0/2944920596 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1c5c0a3c40 msgr2=0x7f1c5c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:10.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.054+0000 7f1c7dc00640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1c5c0a3c40 0x7f1c5c0a6100 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1c60000c00 tx=0x7f1c600314a0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.054+0000 7f1c7dc00640 1 -- 172.21.15.40:0/2944920596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c78101750 msgr2=0x7f1c781056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:10.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.054+0000 7f1c7dc00640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c78101750 0x7f1c781056e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800e9c0 tx=0x7f1c6800ee80 comp rx=0 tx=0).stop 2024-04-27T00:14:10.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.054+0000 7f1c7dc00640 1 -- 172.21.15.40:0/2944920596 shutdown_connections 2024-04-27T00:14:10.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.054+0000 7f1c7dc00640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1c5c0a3c40 0x7f1c5c0a6100 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.054+0000 7f1c7dc00640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c78103300 0x7f1c78104260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.054+0000 7f1c7dc00640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1c78102940 0x7f1c78103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.054+0000 7f1c7dc00640 1 --2- 172.21.15.40:0/2944920596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c78101750 0x7f1c781056e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.054+0000 7f1c7dc00640 1 -- 172.21.15.40:0/2944920596 >> 172.21.15.40:0/2944920596 conn(0x7f1c780fcce0 msgr2=0x7f1c780fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:10.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.054+0000 7f1c7dc00640 1 -- 172.21.15.40:0/2944920596 shutdown_connections 2024-04-27T00:14:10.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.054+0000 7f1c7dc00640 1 -- 172.21.15.40:0/2944920596 wait complete. 2024-04-27T00:14:10.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-27T00:14:10.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.283+0000 7f7505400640 1 -- 172.21.15.40:0/2244767365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f750010a940 msgr2=0x7f75001111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:10.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.283+0000 7f7505400640 1 --2- 172.21.15.40:0/2244767365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f750010a940 0x7f75001111d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f74f8009ca0 tx=0x7f74f802a8c0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.283+0000 7f7505400640 1 -- 172.21.15.40:0/2244767365 shutdown_connections 2024-04-27T00:14:10.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.283+0000 7f7505400640 1 --2- 172.21.15.40:0/2244767365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f750010a940 0x7f75001111d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.283+0000 7f7505400640 1 --2- 172.21.15.40:0/2244767365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7500109f80 0x7f750010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.283+0000 7f7505400640 1 --2- 172.21.15.40:0/2244767365 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7500108d90 0x7f75001091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.283+0000 7f7505400640 1 -- 172.21.15.40:0/2244767365 >> 172.21.15.40:0/2244767365 conn(0x7f7500077d30 msgr2=0x7f750007a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:10.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.283+0000 7f7505400640 1 -- 172.21.15.40:0/2244767365 shutdown_connections 2024-04-27T00:14:10.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.283+0000 7f7505400640 1 -- 172.21.15.40:0/2244767365 wait complete. 2024-04-27T00:14:10.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.284+0000 7f7505400640 1 Processor -- start 2024-04-27T00:14:10.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.284+0000 7f7505400640 1 -- start start 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.284+0000 7f7505400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7500108d90 0x7f750019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.284+0000 7f7505400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7500109f80 0x7f750019f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.284+0000 7f7505400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f750010a940 0x7f75001a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f7505400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f75001148f0 con 0x7f7500108d90 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f7505400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f7500114770 con 0x7f7500109f80 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f7505400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7500114a70 con 0x7f750010a940 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f74ff400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7500109f80 0x7f750019f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f74ffe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7500108d90 0x7f750019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f74ff400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7500109f80 0x7f750019f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:36708/0 (socket says 172.21.15.40:36708) 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f74ff400640 1 -- 172.21.15.40:0/3901773334 learned_addr learned my addr 172.21.15.40:0/3901773334 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f7504a00640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f750010a940 0x7f75001a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f74ff400640 1 -- 172.21.15.40:0/3901773334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f750010a940 msgr2=0x7f75001a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f74ff400640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f750010a940 0x7f75001a6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f74ff400640 1 -- 172.21.15.40:0/3901773334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7500108d90 msgr2=0x7f750019eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:10.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f74ff400640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7500108d90 0x7f750019eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.285+0000 7f74ff400640 1 -- 172.21.15.40:0/3901773334 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75001a6b00 con 0x7f7500109f80 2024-04-27T00:14:10.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.286+0000 7f74ff400640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7500109f80 0x7f750019f420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00e9f0 tx=0x7f74ec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:10.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.286+0000 7f74fcc00640 1 -- 172.21.15.40:0/3901773334 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ec00cd80 con 0x7f7500109f80 2024-04-27T00:14:10.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.286+0000 7f7505400640 1 -- 172.21.15.40:0/3901773334 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75001a6df0 con 0x7f7500109f80 2024-04-27T00:14:10.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.286+0000 7f74fcc00640 1 -- 172.21.15.40:0/3901773334 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74ec0108e0 con 0x7f7500109f80 2024-04-27T00:14:10.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.286+0000 7f74fcc00640 1 -- 172.21.15.40:0/3901773334 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ec018980 con 0x7f7500109f80 2024-04-27T00:14:10.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.286+0000 7f7505400640 1 -- 172.21.15.40:0/3901773334 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75001a72b0 con 0x7f7500109f80 2024-04-27T00:14:10.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.287+0000 7f7505400640 1 -- 172.21.15.40:0/3901773334 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7500110050 con 0x7f7500109f80 2024-04-27T00:14:10.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.290+0000 7f74fcc00640 1 -- 172.21.15.40:0/3901773334 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f74ec010430 con 0x7f7500109f80 2024-04-27T00:14:10.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.292+0000 7f74fcc00640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f74e40a3c40 0x7f74e40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:10.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.292+0000 7f74fcc00640 1 -- 172.21.15.40:0/3901773334 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f74ec0d0e90 con 0x7f7500109f80 2024-04-27T00:14:10.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.292+0000 7f74ffe00640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f74e40a3c40 0x7f74e40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:10.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.293+0000 7f74ffe00640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f74e40a3c40 0x7f74e40a6100 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7500109de0 tx=0x7f74f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:10.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.293+0000 7f74fcc00640 1 -- 172.21.15.40:0/3901773334 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f74ec09ccc0 con 0x7f7500109f80 2024-04-27T00:14:10.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.521+0000 7f7505400640 1 -- 172.21.15.40:0/3901773334 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f750002dcf0 con 0x7f7500109f80 2024-04-27T00:14:10.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.522+0000 7f74fcc00640 1 -- 172.21.15.40:0/3901773334 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v94) ==== 81+0+588 (secure 0 0 0) 0x7f74ec0a0fc0 con 0x7f7500109f80 2024-04-27T00:14:10.521 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1959424465 2024-04-27T23:50:44.546469+0000 2024-04-27T00:14:10.521 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/1568598447 2024-04-27T23:50:44.546469+0000 2024-04-27T00:14:10.521 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/29889962 2024-04-27T23:50:44.546469+0000 2024-04-27T00:14:10.521 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/673764112 2024-04-27T23:50:09.399879+0000 2024-04-27T00:14:10.521 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3782482730 2024-04-27T23:50:09.399879+0000 2024-04-27T00:14:10.521 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/1568598447 2024-04-27T23:50:44.546469+0000 2024-04-27T00:14:10.522 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1079307273 2024-04-27T23:50:09.399879+0000 2024-04-27T00:14:10.522 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/673764112 2024-04-27T23:50:09.399879+0000 2024-04-27T00:14:10.522 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4238592841 2024-04-27T23:50:44.546469+0000 2024-04-27T00:14:10.522 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4067589404 2024-04-27T23:50:09.399879+0000 2024-04-27T00:14:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:listed 10 entries 2024-04-27T00:14:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.525+0000 7f7505400640 1 -- 172.21.15.40:0/3901773334 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f74e40a3c40 msgr2=0x7f74e40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.525+0000 7f7505400640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f74e40a3c40 0x7f74e40a6100 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7500109de0 tx=0x7f74f0005f90 comp rx=0 tx=0).stop 2024-04-27T00:14:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.525+0000 7f7505400640 1 -- 172.21.15.40:0/3901773334 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7500109f80 msgr2=0x7f750019f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.525+0000 7f7505400640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7500109f80 0x7f750019f420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00e9f0 tx=0x7f74ec00eeb0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.525+0000 7f7505400640 1 -- 172.21.15.40:0/3901773334 shutdown_connections 2024-04-27T00:14:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.525+0000 7f7505400640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f74e40a3c40 0x7f74e40a6100 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.525+0000 7f7505400640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f750010a940 0x7f75001a6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.525+0000 7f7505400640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f7500109f80 0x7f750019f420 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.525+0000 7f7505400640 1 --2- 172.21.15.40:0/3901773334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7500108d90 0x7f750019eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.525+0000 7f7505400640 1 -- 172.21.15.40:0/3901773334 >> 172.21.15.40:0/3901773334 conn(0x7f7500077d30 msgr2=0x7f750010d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.525+0000 7f7505400640 1 -- 172.21.15.40:0/3901773334 shutdown_connections 2024-04-27T00:14:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:10.525+0000 7f7505400640 1 -- 172.21.15.40:0/3901773334 wait complete. 2024-04-27T00:14:10.544 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-27T00:14:10.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:14:10.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-27T00:14:10.652 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-04-27T00:14:10.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:14:10.653 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-27T00:14:10.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:14:10.712 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-04-27T00:14:10.751 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-27T00:14:10.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:14:10.751 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-27T00:14:10.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:10 smithi080 ceph-mon[32362]: pgmap v786: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:14:10.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:10 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2325469939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:14:10.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:10 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2944920596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:14:10.846 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:10 smithi103 ceph-mon[32952]: pgmap v786: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:14:10.846 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:10 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2325469939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:14:10.846 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:10 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2944920596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:14:10.849 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:10 smithi040 ceph-mon[28095]: pgmap v786: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:14:10.849 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:10 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2325469939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:14:10.849 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:10 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2944920596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:14:10.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:14:10.866 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-04-27T00:14:10.986 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-27T00:14:10.996 INFO:tasks.cephadm:Teardown begin 2024-04-27T00:14:10.997 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-27T00:14:10.997 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-27T00:14:11.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.183+0000 7f5b82400640 1 -- 172.21.15.40:0/1070432515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b7c104360 msgr2=0x7f5b7c10abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:11.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.183+0000 7f5b82400640 1 --2- 172.21.15.40:0/1070432515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b7c104360 0x7f5b7c10abf0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009860 tx=0x7f5b7402a340 comp rx=0 tx=0).stop 2024-04-27T00:14:11.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.183+0000 7f5b82400640 1 -- 172.21.15.40:0/1070432515 shutdown_connections 2024-04-27T00:14:11.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.183+0000 7f5b82400640 1 --2- 172.21.15.40:0/1070432515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b7c104360 0x7f5b7c10abf0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.183+0000 7f5b82400640 1 --2- 172.21.15.40:0/1070432515 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5b7c1039a0 0x7f5b7c103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.183+0000 7f5b82400640 1 --2- 172.21.15.40:0/1070432515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b7c1027b0 0x7f5b7c102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.183+0000 7f5b82400640 1 -- 172.21.15.40:0/1070432515 >> 172.21.15.40:0/1070432515 conn(0x7f5b7c0fdd40 msgr2=0x7f5b7c100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:11.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.183+0000 7f5b82400640 1 -- 172.21.15.40:0/1070432515 shutdown_connections 2024-04-27T00:14:11.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.183+0000 7f5b82400640 1 -- 172.21.15.40:0/1070432515 wait complete. 2024-04-27T00:14:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.184+0000 7f5b82400640 1 Processor -- start 2024-04-27T00:14:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.184+0000 7f5b82400640 1 -- start start 2024-04-27T00:14:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.185+0000 7f5b82400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b7c1027b0 0x7f5b7c19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.185+0000 7f5b82400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b7c1039a0 0x7f5b7c19b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.185+0000 7f5b82400640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5b7c104360 0x7f5b7c1a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.185+0000 7f5b82400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c10e2e0 con 0x7f5b7c1027b0 2024-04-27T00:14:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.185+0000 7f5b82400640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c10e160 con 0x7f5b7c104360 2024-04-27T00:14:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.185+0000 7f5b82400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c10e460 con 0x7f5b7c1039a0 2024-04-27T00:14:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.185+0000 7f5b81000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b7c1027b0 0x7f5b7c19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.185+0000 7f5b81a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5b7c104360 0x7f5b7c1a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.185+0000 7f5b81000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b7c1027b0 0x7f5b7c19ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54104/0 (socket says 172.21.15.40:54104) 2024-04-27T00:14:11.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.185+0000 7f5b81000640 1 -- 172.21.15.40:0/3401972394 learned_addr learned my addr 172.21.15.40:0/3401972394 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:11.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.185+0000 7f5b81a00640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5b7c104360 0x7f5b7c1a20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:36726/0 (socket says 172.21.15.40:36726) 2024-04-27T00:14:11.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.185+0000 7f5b7be00640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b7c1039a0 0x7f5b7c19b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:11.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b7be00640 1 -- 172.21.15.40:0/3401972394 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5b7c104360 msgr2=0x7f5b7c1a20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:11.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b7be00640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5b7c104360 0x7f5b7c1a20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b7be00640 1 -- 172.21.15.40:0/3401972394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b7c1027b0 msgr2=0x7f5b7c19ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:11.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b7be00640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b7c1027b0 0x7f5b7c19ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b7be00640 1 -- 172.21.15.40:0/3401972394 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b7c1a2740 con 0x7f5b7c1039a0 2024-04-27T00:14:11.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b81a00640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5b7c104360 0x7f5b7c1a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:14:11.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b81000640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b7c1027b0 0x7f5b7c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:14:11.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b7be00640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b7c1039a0 0x7f5b7c19b060 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009fd0 tx=0x7f5b6400ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:11.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b79600640 1 -- 172.21.15.40:0/3401972394 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b64004b20 con 0x7f5b7c1039a0 2024-04-27T00:14:11.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b79600640 1 -- 172.21.15.40:0/3401972394 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b64004cc0 con 0x7f5b7c1039a0 2024-04-27T00:14:11.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b79600640 1 -- 172.21.15.40:0/3401972394 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b64010720 con 0x7f5b7c1039a0 2024-04-27T00:14:11.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b82400640 1 -- 172.21.15.40:0/3401972394 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b7c1a2a30 con 0x7f5b7c1039a0 2024-04-27T00:14:11.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.186+0000 7f5b82400640 1 -- 172.21.15.40:0/3401972394 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b7c1a2f30 con 0x7f5b7c1039a0 2024-04-27T00:14:11.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.187+0000 7f5b82400640 1 -- 172.21.15.40:0/3401972394 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b7c1a31c0 con 0x7f5b7c1039a0 2024-04-27T00:14:11.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.190+0000 7f5b79600640 1 -- 172.21.15.40:0/3401972394 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5b640108c0 con 0x7f5b7c1039a0 2024-04-27T00:14:11.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.192+0000 7f5b79600640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5b600a3c40 0x7f5b600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:11.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.192+0000 7f5b79600640 1 -- 172.21.15.40:0/3401972394 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f5b640d15a0 con 0x7f5b7c1039a0 2024-04-27T00:14:11.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.192+0000 7f5b81000640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5b600a3c40 0x7f5b600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:11.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.193+0000 7f5b81000640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5b600a3c40 0x7f5b600a6100 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c011490 tx=0x7f5b6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:11.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.194+0000 7f5b79600640 1 -- 172.21.15.40:0/3401972394 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5b6409da20 con 0x7f5b7c1039a0 2024-04-27T00:14:11.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.443+0000 7f5b82400640 1 -- 172.21.15.40:0/3401972394 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b7c1a3390 con 0x7f5b7c1039a0 2024-04-27T00:14:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.445+0000 7f5b79600640 1 -- 172.21.15.40:0/3401972394 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11872 (secure 0 0 0) 0x7f5b6409d320 con 0x7f5b7c1039a0 2024-04-27T00:14:11.444 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:14:11.445 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/2265538751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":2265538751},{"type":"v1","addr":"172.21.15.40:6841","nonce":2265538751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:58:50.616677+0000","modified":"2024-04-27T00:01:48.043853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24523":{"gid":24523,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.80:6839/3536638444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6838","nonce":3536638444},{"type":"v1","addr":"172.21.15.80:6839","nonce":3536638444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24590,24559,24560]},"id":1}]} 2024-04-27T00:14:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 24 2024-04-27T00:14:11.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.448+0000 7f5b82400640 1 -- 172.21.15.40:0/3401972394 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5b600a3c40 msgr2=0x7f5b600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:11.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.448+0000 7f5b82400640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5b600a3c40 0x7f5b600a6100 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c011490 tx=0x7f5b6c009260 comp rx=0 tx=0).stop 2024-04-27T00:14:11.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.448+0000 7f5b82400640 1 -- 172.21.15.40:0/3401972394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b7c1039a0 msgr2=0x7f5b7c19b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.448+0000 7f5b82400640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b7c1039a0 0x7f5b7c19b060 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009fd0 tx=0x7f5b6400ef90 comp rx=0 tx=0).stop 2024-04-27T00:14:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.448+0000 7f5b82400640 1 -- 172.21.15.40:0/3401972394 shutdown_connections 2024-04-27T00:14:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.448+0000 7f5b82400640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f5b7c104360 0x7f5b7c1a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.448+0000 7f5b82400640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f5b600a3c40 0x7f5b600a6100 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.448+0000 7f5b82400640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b7c1039a0 0x7f5b7c19b060 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.448+0000 7f5b82400640 1 --2- 172.21.15.40:0/3401972394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b7c1027b0 0x7f5b7c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.448+0000 7f5b82400640 1 -- 172.21.15.40:0/3401972394 >> 172.21.15.40:0/3401972394 conn(0x7f5b7c0fdd40 msgr2=0x7f5b7c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.448+0000 7f5b82400640 1 -- 172.21.15.40:0/3401972394 shutdown_connections 2024-04-27T00:14:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.448+0000 7f5b82400640 1 -- 172.21.15.40:0/3401972394 wait complete. 2024-04-27T00:14:11.467 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-27T00:14:11.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.686+0000 7f4744e00640 1 -- 172.21.15.40:0/2478456579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47401027a0 msgr2=0x7f4740102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:11.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.686+0000 7f4744e00640 1 --2- 172.21.15.40:0/2478456579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47401027a0 0x7f4740102bc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4730009900 tx=0x7f473002a410 comp rx=0 tx=0).stop 2024-04-27T00:14:11.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.686+0000 7f4744e00640 1 -- 172.21.15.40:0/2478456579 shutdown_connections 2024-04-27T00:14:11.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.686+0000 7f4744e00640 1 --2- 172.21.15.40:0/2478456579 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4740104350 0x7f474010abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.686+0000 7f4744e00640 1 --2- 172.21.15.40:0/2478456579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4740103990 0x7f4740103e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.686+0000 7f4744e00640 1 --2- 172.21.15.40:0/2478456579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47401027a0 0x7f4740102bc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.686+0000 7f4744e00640 1 -- 172.21.15.40:0/2478456579 >> 172.21.15.40:0/2478456579 conn(0x7f47400fdd10 msgr2=0x7f4740100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:11.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.686+0000 7f4744e00640 1 -- 172.21.15.40:0/2478456579 shutdown_connections 2024-04-27T00:14:11.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.686+0000 7f4744e00640 1 -- 172.21.15.40:0/2478456579 wait complete. 2024-04-27T00:14:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.687+0000 7f4744e00640 1 Processor -- start 2024-04-27T00:14:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.688+0000 7f4744e00640 1 -- start start 2024-04-27T00:14:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.688+0000 7f4744e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47401027a0 0x7f474019ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.688+0000 7f4744e00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4740103990 0x7f474019b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.688+0000 7f4744e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4740104350 0x7f47401a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.688+0000 7f4744e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f474010e310 con 0x7f4740104350 2024-04-27T00:14:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.688+0000 7f4744e00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f474010e190 con 0x7f4740103990 2024-04-27T00:14:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.688+0000 7f4744e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f474010e490 con 0x7f47401027a0 2024-04-27T00:14:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.688+0000 7f4736a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4740103990 0x7f474019b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:11.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.688+0000 7f473fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4740104350 0x7f47401a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:11.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.688+0000 7f473f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47401027a0 0x7f474019ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:11.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.688+0000 7f4736a00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4740103990 0x7f474019b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:36744/0 (socket says 172.21.15.40:36744) 2024-04-27T00:14:11.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.688+0000 7f4736a00640 1 -- 172.21.15.40:0/3553825297 learned_addr learned my addr 172.21.15.40:0/3553825297 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:11.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.689+0000 7f4736a00640 1 -- 172.21.15.40:0/3553825297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47401027a0 msgr2=0x7f474019ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:11.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.689+0000 7f4736a00640 1 --2- 172.21.15.40:0/3553825297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47401027a0 0x7f474019ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.689+0000 7f4736a00640 1 -- 172.21.15.40:0/3553825297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4740104350 msgr2=0x7f47401a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:11.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.689+0000 7f4736a00640 1 --2- 172.21.15.40:0/3553825297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4740104350 0x7f47401a2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.689+0000 7f4736a00640 1 -- 172.21.15.40:0/3553825297 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47401a28d0 con 0x7f4740103990 2024-04-27T00:14:11.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.689+0000 7f4736a00640 1 --2- 172.21.15.40:0/3553825297 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4740103990 0x7f474019b1d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f472800e9f0 tx=0x7f472800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:11.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.689+0000 7f473cc00640 1 -- 172.21.15.40:0/3553825297 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472800cd80 con 0x7f4740103990 2024-04-27T00:14:11.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.689+0000 7f473cc00640 1 -- 172.21.15.40:0/3553825297 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47280054d0 con 0x7f4740103990 2024-04-27T00:14:11.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.689+0000 7f4744e00640 1 -- 172.21.15.40:0/3553825297 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47401a2bc0 con 0x7f4740103990 2024-04-27T00:14:11.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.690+0000 7f473cc00640 1 -- 172.21.15.40:0/3553825297 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4728010690 con 0x7f4740103990 2024-04-27T00:14:11.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.690+0000 7f4744e00640 1 -- 172.21.15.40:0/3553825297 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47401a3090 con 0x7f4740103990 2024-04-27T00:14:11.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.691+0000 7f4744e00640 1 -- 172.21.15.40:0/3553825297 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47401a32c0 con 0x7f4740103990 2024-04-27T00:14:11.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.693+0000 7f473cc00640 1 -- 172.21.15.40:0/3553825297 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4728004e90 con 0x7f4740103990 2024-04-27T00:14:11.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.694+0000 7f473cc00640 1 --2- 172.21.15.40:0/3553825297 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f47240a3c40 0x7f47240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:11.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.695+0000 7f473cc00640 1 -- 172.21.15.40:0/3553825297 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f47280d0e80 con 0x7f4740103990 2024-04-27T00:14:11.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.695+0000 7f473f400640 1 --2- 172.21.15.40:0/3553825297 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f47240a3c40 0x7f47240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:11.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.695+0000 7f473f400640 1 --2- 172.21.15.40:0/3553825297 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f47240a3c40 0x7f47240a6100 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4730005ab0 tx=0x7f4730005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:11.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.697+0000 7f473cc00640 1 -- 172.21.15.40:0/3553825297 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f472809d300 con 0x7f4740103990 2024-04-27T00:14:11.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:11 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3901773334' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-27T00:14:11.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:11 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3401972394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:14:11.846 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:11 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3901773334' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-27T00:14:11.846 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:11 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3401972394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:14:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.944+0000 7f4744e00640 1 -- 172.21.15.40:0/3553825297 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47401a3490 con 0x7f4740103990 2024-04-27T00:14:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.946+0000 7f473cc00640 1 -- 172.21.15.40:0/3553825297 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11872 (secure 0 0 0) 0x7f472809cc00 con 0x7f4740103990 2024-04-27T00:14:11.946 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:14:11.946 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1384127498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1384127498},{"type":"v1","addr":"172.21.15.40:6835","nonce":1384127498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/2265538751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":2265538751},{"type":"v1","addr":"172.21.15.40:6841","nonce":2265538751}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6833/3975801652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6832","nonce":3975801652},{"type":"v1","addr":"172.21.15.80:6833","nonce":3975801652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/970790712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":970790712},{"type":"v1","addr":"172.21.15.103:6833","nonce":970790712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24493,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3497909120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3497909120},{"type":"v1","addr":"172.21.15.40:6837","nonce":3497909120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.80:6835/1031714719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6834","nonce":1031714719},{"type":"v1","addr":"172.21.15.80:6835","nonce":1031714719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:58:50.616677+0000","modified":"2024-04-27T00:01:48.043853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24523":{"gid":24523,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2596993377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2596993377},{"type":"v1","addr":"172.21.15.103:6835","nonce":2596993377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.80:6837/4116866934","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6836","nonce":4116866934},{"type":"v1","addr":"172.21.15.80:6837","nonce":4116866934}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6837/2206925949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":2206925949},{"type":"v1","addr":"172.21.15.103:6837","nonce":2206925949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.103:6839/1527392010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":1527392010},{"type":"v1","addr":"172.21.15.103:6839","nonce":1527392010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6839/2205335602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":2205335602},{"type":"v1","addr":"172.21.15.40:6839","nonce":2205335602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.80:6839/3536638444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6838","nonce":3536638444},{"type":"v1","addr":"172.21.15.80:6839","nonce":3536638444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24590,24559,24560]},"id":1}]} 2024-04-27T00:14:11.946 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 24 2024-04-27T00:14:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.948+0000 7f4744e00640 1 -- 172.21.15.40:0/3553825297 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f47240a3c40 msgr2=0x7f47240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.949+0000 7f4744e00640 1 --2- 172.21.15.40:0/3553825297 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f47240a3c40 0x7f47240a6100 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4730005ab0 tx=0x7f4730005a20 comp rx=0 tx=0).stop 2024-04-27T00:14:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.949+0000 7f4744e00640 1 -- 172.21.15.40:0/3553825297 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4740103990 msgr2=0x7f474019b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.949+0000 7f4744e00640 1 --2- 172.21.15.40:0/3553825297 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4740103990 0x7f474019b1d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f472800e9f0 tx=0x7f472800edf0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.949+0000 7f4744e00640 1 -- 172.21.15.40:0/3553825297 shutdown_connections 2024-04-27T00:14:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.949+0000 7f4744e00640 1 --2- 172.21.15.40:0/3553825297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4740104350 0x7f47401a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.949+0000 7f4744e00640 1 --2- 172.21.15.40:0/3553825297 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f4740103990 0x7f474019b1d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.949+0000 7f4744e00640 1 --2- 172.21.15.40:0/3553825297 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f47240a3c40 0x7f47240a6100 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.949+0000 7f4744e00640 1 --2- 172.21.15.40:0/3553825297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47401027a0 0x7f474019ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.949+0000 7f4744e00640 1 -- 172.21.15.40:0/3553825297 >> 172.21.15.40:0/3553825297 conn(0x7f47400fdd10 msgr2=0x7f4740074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.949+0000 7f4744e00640 1 -- 172.21.15.40:0/3553825297 shutdown_connections 2024-04-27T00:14:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:11.949+0000 7f4744e00640 1 -- 172.21.15.40:0/3553825297 wait complete. 2024-04-27T00:14:11.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:11 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3901773334' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-27T00:14:11.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:11 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3401972394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:14:11.968 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T00:14:12.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.187+0000 7f30cec00640 1 -- 172.21.15.40:0/449362740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8108f90 msgr2=0x7f30c81093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:12.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.187+0000 7f30cec00640 1 --2- 172.21.15.40:0/449362740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8108f90 0x7f30c81093b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009900 tx=0x7f30b802a410 comp rx=0 tx=0).stop 2024-04-27T00:14:12.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.187+0000 7f30cec00640 1 -- 172.21.15.40:0/449362740 shutdown_connections 2024-04-27T00:14:12.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.187+0000 7f30cec00640 1 --2- 172.21.15.40:0/449362740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c810ab40 0x7f30c81113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.187+0000 7f30cec00640 1 --2- 172.21.15.40:0/449362740 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30c810a180 0x7f30c810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.187+0000 7f30cec00640 1 --2- 172.21.15.40:0/449362740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8108f90 0x7f30c81093b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.187+0000 7f30cec00640 1 -- 172.21.15.40:0/449362740 >> 172.21.15.40:0/449362740 conn(0x7f30c8077f40 msgr2=0x7f30c807a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:12.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.187+0000 7f30cec00640 1 -- 172.21.15.40:0/449362740 shutdown_connections 2024-04-27T00:14:12.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.187+0000 7f30cec00640 1 -- 172.21.15.40:0/449362740 wait complete. 2024-04-27T00:14:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.188+0000 7f30cec00640 1 Processor -- start 2024-04-27T00:14:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.188+0000 7f30cec00640 1 -- start start 2024-04-27T00:14:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.188+0000 7f30cec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8108f90 0x7f30c819f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.188+0000 7f30cec00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c810a180 0x7f30c819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30cec00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30c810ab40 0x7f30c81a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30cec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f30c8114ad0 con 0x7f30c8108f90 2024-04-27T00:14:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30cec00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f30c8114950 con 0x7f30c810ab40 2024-04-27T00:14:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30cec00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30c8114c50 con 0x7f30c810a180 2024-04-27T00:14:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30c7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8108f90 0x7f30c819f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30c7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8108f90 0x7f30c819f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54152/0 (socket says 172.21.15.40:54152) 2024-04-27T00:14:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30c7e00640 1 -- 172.21.15.40:0/1117181986 learned_addr learned my addr 172.21.15.40:0/1117181986 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:12.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30ccc00640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30c810ab40 0x7f30c81a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:12.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30c7400640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c810a180 0x7f30c819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:12.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30c7e00640 1 -- 172.21.15.40:0/1117181986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c810a180 msgr2=0x7f30c819f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:12.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30c7e00640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c810a180 0x7f30c819f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30c7e00640 1 -- 172.21.15.40:0/1117181986 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30c810ab40 msgr2=0x7f30c81a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:12.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30c7e00640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30c810ab40 0x7f30c81a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30c7e00640 1 -- 172.21.15.40:0/1117181986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c81a6dd0 con 0x7f30c8108f90 2024-04-27T00:14:12.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.189+0000 7f30c7400640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c810a180 0x7f30c819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:14:12.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.190+0000 7f30c7e00640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8108f90 0x7f30c819f1b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f30b8006010 tx=0x7f30b80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:12.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.190+0000 7f30c4c00640 1 -- 172.21.15.40:0/1117181986 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b803f030 con 0x7f30c8108f90 2024-04-27T00:14:12.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.190+0000 7f30c4c00640 1 -- 172.21.15.40:0/1117181986 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30b8031910 con 0x7f30c8108f90 2024-04-27T00:14:12.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.190+0000 7f30c4c00640 1 -- 172.21.15.40:0/1117181986 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b8035070 con 0x7f30c8108f90 2024-04-27T00:14:12.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.190+0000 7f30cec00640 1 -- 172.21.15.40:0/1117181986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30c81a7060 con 0x7f30c8108f90 2024-04-27T00:14:12.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.190+0000 7f30cec00640 1 -- 172.21.15.40:0/1117181986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30c81a75a0 con 0x7f30c8108f90 2024-04-27T00:14:12.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.192+0000 7f30cec00640 1 -- 172.21.15.40:0/1117181986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30c81103d0 con 0x7f30c8108f90 2024-04-27T00:14:12.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.193+0000 7f30c4c00640 1 -- 172.21.15.40:0/1117181986 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f30b8042030 con 0x7f30c8108f90 2024-04-27T00:14:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.195+0000 7f30c4c00640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f30ac0a3cc0 0x7f30ac0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.195+0000 7f30c4c00640 1 -- 172.21.15.40:0/1117181986 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f30b80f0e90 con 0x7f30c8108f90 2024-04-27T00:14:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.195+0000 7f30c7400640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f30ac0a3cc0 0x7f30ac0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:12.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.196+0000 7f30c7400640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f30ac0a3cc0 0x7f30ac0a6180 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f30c81a06c0 tx=0x7f30b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:12.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.199+0000 7f30c4c00640 1 -- 172.21.15.40:0/1117181986 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f30b80bd310 con 0x7f30c8108f90 2024-04-27T00:14:12.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.427+0000 7f30cec00640 1 -- 172.21.15.40:0/1117181986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f30c8063530 con 0x7f30c8108f90 2024-04-27T00:14:12.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.431+0000 7f30c4c00640 1 -- 172.21.15.40:0/1117181986 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24727 (secure 0 0 0) 0x7f30b80bcc10 con 0x7f30c8108f90 2024-04-27T00:14:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-04-27T00:14:12.431 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":94,"fsid":"78cff6ea-0427-11ef-bc93-c7b262605968","created":"2024-04-26T23:49:26.448890+0000","modified":"2024-04-27T00:01:46.569457+0000","last_up_change":"2024-04-26T23:57:51.516269+0000","last_in_change":"2024-04-26T23:57:32.501276+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:53:32.708818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:58:48.506601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:58:49.540141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:58:53.179820+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54e9c732-e7a5-45f0-be16-dee18cc70647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6803","nonce":1203247598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6805","nonce":1203247598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6809","nonce":1203247598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1203247598},{"type":"v1","addr":"172.21.15.40:6807","nonce":1203247598}]},"public_addr":"172.21.15.40:6803/1203247598","cluster_addr":"172.21.15.40:6805/1203247598","heartbeat_back_addr":"172.21.15.40:6809/1203247598","heartbeat_front_addr":"172.21.15.40:6807/1203247598","state":["exists","up"]},{"osd":1,"uuid":"ee040674-8f99-4406-a7bf-a989815dce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6801","nonce":3368601093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6803","nonce":3368601093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6807","nonce":3368601093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":3368601093},{"type":"v1","addr":"172.21.15.80:6805","nonce":3368601093}]},"public_addr":"172.21.15.80:6801/3368601093","cluster_addr":"172.21.15.80:6803/3368601093","heartbeat_back_addr":"172.21.15.80:6807/3368601093","heartbeat_front_addr":"172.21.15.80:6805/3368601093","state":["exists","up"]},{"osd":2,"uuid":"2ebf4dbf-c70b-44d4-8e4b-602c4fe4e219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6801","nonce":3393467187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6803","nonce":3393467187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6807","nonce":3393467187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":3393467187},{"type":"v1","addr":"172.21.15.103:6805","nonce":3393467187}]},"public_addr":"172.21.15.103:6801/3393467187","cluster_addr":"172.21.15.103:6803/3393467187","heartbeat_back_addr":"172.21.15.103:6807/3393467187","heartbeat_front_addr":"172.21.15.103:6805/3393467187","state":["exists","up"]},{"osd":3,"uuid":"9c08a594-7548-44d3-b6cc-62b25ee078eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6811","nonce":4161271893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6813","nonce":4161271893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6817","nonce":4161271893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":4161271893},{"type":"v1","addr":"172.21.15.40:6815","nonce":4161271893}]},"public_addr":"172.21.15.40:6811/4161271893","cluster_addr":"172.21.15.40:6813/4161271893","heartbeat_back_addr":"172.21.15.40:6817/4161271893","heartbeat_front_addr":"172.21.15.40:6815/4161271893","state":["exists","up"]},{"osd":4,"uuid":"2e645527-e694-4cd0-99d7-035f5c36bed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6809","nonce":3602586790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6811","nonce":3602586790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6815","nonce":3602586790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":3602586790},{"type":"v1","addr":"172.21.15.80:6813","nonce":3602586790}]},"public_addr":"172.21.15.80:6809/3602586790","cluster_addr":"172.21.15.80:6811/3602586790","heartbeat_back_addr":"172.21.15.80:6815/3602586790","heartbeat_front_addr":"172.21.15.80:6813/3602586790","state":["exists","up"]},{"osd":5,"uuid":"9f6e9855-4856-4a8a-a884-1826a9c29327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6809","nonce":63420438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6811","nonce":63420438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6815","nonce":63420438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":63420438},{"type":"v1","addr":"172.21.15.103:6813","nonce":63420438}]},"public_addr":"172.21.15.103:6809/63420438","cluster_addr":"172.21.15.103:6811/63420438","heartbeat_back_addr":"172.21.15.103:6815/63420438","heartbeat_front_addr":"172.21.15.103:6813/63420438","state":["exists","up"]},{"osd":6,"uuid":"6180405f-9925-491b-aee8-ecaa9fd39d15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6819","nonce":3383463516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6821","nonce":3383463516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6825","nonce":3383463516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":3383463516},{"type":"v1","addr":"172.21.15.40:6823","nonce":3383463516}]},"public_addr":"172.21.15.40:6819/3383463516","cluster_addr":"172.21.15.40:6821/3383463516","heartbeat_back_addr":"172.21.15.40:6825/3383463516","heartbeat_front_addr":"172.21.15.40:6823/3383463516","state":["exists","up"]},{"osd":7,"uuid":"9cd5c915-ae45-499a-8910-8619855fec2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6817","nonce":1231184332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6819","nonce":1231184332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6823","nonce":1231184332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":1231184332},{"type":"v1","addr":"172.21.15.80:6821","nonce":1231184332}]},"public_addr":"172.21.15.80:6817/1231184332","cluster_addr":"172.21.15.80:6819/1231184332","heartbeat_back_addr":"172.21.15.80:6823/1231184332","heartbeat_front_addr":"172.21.15.80:6821/1231184332","state":["exists","up"]},{"osd":8,"uuid":"63176b7b-984c-4aac-be69-3fe6d316ac7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6817","nonce":4109830511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6819","nonce":4109830511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6823","nonce":4109830511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4109830511},{"type":"v1","addr":"172.21.15.103:6821","nonce":4109830511}]},"public_addr":"172.21.15.103:6817/4109830511","cluster_addr":"172.21.15.103:6819/4109830511","heartbeat_back_addr":"172.21.15.103:6823/4109830511","heartbeat_front_addr":"172.21.15.103:6821/4109830511","state":["exists","up"]},{"osd":9,"uuid":"0ed158cb-03c5-4930-b529-7d017cf5c31e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6827","nonce":1030365825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6829","nonce":1030365825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6833","nonce":1030365825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":1030365825},{"type":"v1","addr":"172.21.15.40:6831","nonce":1030365825}]},"public_addr":"172.21.15.40:6827/1030365825","cluster_addr":"172.21.15.40:6829/1030365825","heartbeat_back_addr":"172.21.15.40:6833/1030365825","heartbeat_front_addr":"172.21.15.40:6831/1030365825","state":["exists","up"]},{"osd":10,"uuid":"ba1ee814-e69d-492a-9b3c-64d53ba2e22c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6825","nonce":1789667951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6827","nonce":1789667951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6830","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6831","nonce":1789667951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6828","nonce":1789667951},{"type":"v1","addr":"172.21.15.80:6829","nonce":1789667951}]},"public_addr":"172.21.15.80:6825/1789667951","cluster_addr":"172.21.15.80:6827/1789667951","heartbeat_back_addr":"172.21.15.80:6831/1789667951","heartbeat_front_addr":"172.21.15.80:6829/1789667951","state":["exists","up"]},{"osd":11,"uuid":"7a4a8429-ed19-4687-9a13-251dfbf74878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6825","nonce":2714925397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6827","nonce":2714925397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6831","nonce":2714925397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":2714925397},{"type":"v1","addr":"172.21.15.103:6829","nonce":2714925397}]},"public_addr":"172.21.15.103:6825/2714925397","cluster_addr":"172.21.15.103:6827/2714925397","heartbeat_back_addr":"172.21.15.103:6831/2714925397","heartbeat_front_addr":"172.21.15.103:6829/2714925397","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:52:39.474816+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:04.966651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:53:28.870459+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:02.175602+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:28.426915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:54:53.448713+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:28.663186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:55:55.922622+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:20.997956+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:56:56.487465+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:24.720574+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:57:50.284914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9},{"from":5,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/1959424465":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6801/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/29889962":"2024-04-27T23:50:44.546469+0000","172.21.15.40:6800/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/3782482730":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6800/1568598447":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/1079307273":"2024-04-27T23:50:09.399879+0000","172.21.15.40:6801/673764112":"2024-04-27T23:50:09.399879+0000","172.21.15.40:0/4238592841":"2024-04-27T23:50:44.546469+0000","172.21.15.40:0/4067589404":"2024-04-27T23:50:09.399879+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T00:14:12.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.434+0000 7f30cec00640 1 -- 172.21.15.40:0/1117181986 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f30ac0a3cc0 msgr2=0x7f30ac0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:12.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.434+0000 7f30cec00640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f30ac0a3cc0 0x7f30ac0a6180 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f30c81a06c0 tx=0x7f30b4005f90 comp rx=0 tx=0).stop 2024-04-27T00:14:12.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.434+0000 7f30cec00640 1 -- 172.21.15.40:0/1117181986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8108f90 msgr2=0x7f30c819f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:12.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.434+0000 7f30cec00640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8108f90 0x7f30c819f1b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f30b8006010 tx=0x7f30b80042f0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.434+0000 7f30cec00640 1 -- 172.21.15.40:0/1117181986 shutdown_connections 2024-04-27T00:14:12.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.434+0000 7f30cec00640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f30c810ab40 0x7f30c81a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.434+0000 7f30cec00640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f30ac0a3cc0 0x7f30ac0a6180 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.434+0000 7f30cec00640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c810a180 0x7f30c819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.434+0000 7f30cec00640 1 --2- 172.21.15.40:0/1117181986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8108f90 0x7f30c819f1b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.434+0000 7f30cec00640 1 -- 172.21.15.40:0/1117181986 >> 172.21.15.40:0/1117181986 conn(0x7f30c8077f40 msgr2=0x7f30c810d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:12.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.434+0000 7f30cec00640 1 -- 172.21.15.40:0/1117181986 shutdown_connections 2024-04-27T00:14:12.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.434+0000 7f30cec00640 1 -- 172.21.15.40:0/1117181986 wait complete. 2024-04-27T00:14:12.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-27T00:14:12.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.666+0000 7f9d83200640 1 -- 172.21.15.40:0/3492733073 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9d84102770 msgr2=0x7f9d84102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:12.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.666+0000 7f9d83200640 1 --2- 172.21.15.40:0/3492733073 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9d84102770 0x7f9d84102b90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009860 tx=0x7f9d7402a340 comp rx=0 tx=0).stop 2024-04-27T00:14:12.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.666+0000 7f9d83200640 1 -- 172.21.15.40:0/3492733073 shutdown_connections 2024-04-27T00:14:12.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.666+0000 7f9d83200640 1 --2- 172.21.15.40:0/3492733073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84104320 0x7f9d8410abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.666+0000 7f9d83200640 1 --2- 172.21.15.40:0/3492733073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d84103960 0x7f9d84103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.666+0000 7f9d83200640 1 --2- 172.21.15.40:0/3492733073 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9d84102770 0x7f9d84102b90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.666+0000 7f9d83200640 1 -- 172.21.15.40:0/3492733073 >> 172.21.15.40:0/3492733073 conn(0x7f9d840fdcc0 msgr2=0x7f9d84100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:12.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.666+0000 7f9d83200640 1 -- 172.21.15.40:0/3492733073 shutdown_connections 2024-04-27T00:14:12.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.666+0000 7f9d83200640 1 -- 172.21.15.40:0/3492733073 wait complete. 2024-04-27T00:14:12.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.667+0000 7f9d83200640 1 Processor -- start 2024-04-27T00:14:12.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.667+0000 7f9d83200640 1 -- start start 2024-04-27T00:14:12.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.667+0000 7f9d83200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84102770 0x7f9d8419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.667+0000 7f9d83200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d84103960 0x7f9d8419b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.667+0000 7f9d83200640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9d84104320 0x7f9d841a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.667+0000 7f9d83200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8410e290 con 0x7f9d84102770 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.667+0000 7f9d83200640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8410e110 con 0x7f9d84104320 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.667+0000 7f9d83200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8410e410 con 0x7f9d84103960 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.668+0000 7f9d81e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84102770 0x7f9d8419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.668+0000 7f9d81e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84102770 0x7f9d8419abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54174/0 (socket says 172.21.15.40:54174) 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.668+0000 7f9d81e00640 1 -- 172.21.15.40:0/2413176650 learned_addr learned my addr 172.21.15.40:0/2413176650 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.668+0000 7f9d82800640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9d84104320 0x7f9d841a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.668+0000 7f9d81400640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d84103960 0x7f9d8419b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.668+0000 7f9d82800640 1 -- 172.21.15.40:0/2413176650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d84103960 msgr2=0x7f9d8419b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.668+0000 7f9d82800640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d84103960 0x7f9d8419b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.668+0000 7f9d82800640 1 -- 172.21.15.40:0/2413176650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84102770 msgr2=0x7f9d8419abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.668+0000 7f9d82800640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84102770 0x7f9d8419abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.668+0000 7f9d82800640 1 -- 172.21.15.40:0/2413176650 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d841a27d0 con 0x7f9d84104320 2024-04-27T00:14:12.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.668+0000 7f9d81e00640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84102770 0x7f9d8419abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:14:12.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.669+0000 7f9d82800640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9d84104320 0x7f9d841a2160 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c009fd0 tx=0x7f9d7c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:12.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.669+0000 7f9d7aa00640 1 -- 172.21.15.40:0/2413176650 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c00cd60 con 0x7f9d84104320 2024-04-27T00:14:12.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.669+0000 7f9d83200640 1 -- 172.21.15.40:0/2413176650 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d841a2ac0 con 0x7f9d84104320 2024-04-27T00:14:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.669+0000 7f9d7aa00640 1 -- 172.21.15.40:0/2413176650 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d7c005020 con 0x7f9d84104320 2024-04-27T00:14:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.669+0000 7f9d7aa00640 1 -- 172.21.15.40:0/2413176650 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c010690 con 0x7f9d84104320 2024-04-27T00:14:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.669+0000 7f9d83200640 1 -- 172.21.15.40:0/2413176650 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d841a2fc0 con 0x7f9d84104320 2024-04-27T00:14:12.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.670+0000 7f9d83200640 1 -- 172.21.15.40:0/2413176650 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d841a3250 con 0x7f9d84104320 2024-04-27T00:14:12.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.672+0000 7f9d7aa00640 1 -- 172.21.15.40:0/2413176650 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9d7c0054d0 con 0x7f9d84104320 2024-04-27T00:14:12.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.674+0000 7f9d7aa00640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9d680a3c40 0x7f9d680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:12.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.674+0000 7f9d7aa00640 1 -- 172.21.15.40:0/2413176650 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9158+0+0 (secure 0 0 0) 0x7f9d7c0d1060 con 0x7f9d84104320 2024-04-27T00:14:12.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.674+0000 7f9d81e00640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9d680a3c40 0x7f9d680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:12.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.675+0000 7f9d81e00640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9d680a3c40 0x7f9d680a6100 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9d740095f0 tx=0x7f9d74030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:12.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.678+0000 7f9d7aa00640 1 -- 172.21.15.40:0/2413176650 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9d7c09d4e0 con 0x7f9d84104320 2024-04-27T00:14:12.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:12 smithi040 ceph-mon[28095]: pgmap v787: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:14:12.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:12 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/3553825297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:14:12.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:12 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1117181986' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:14:12.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:12 smithi080 ceph-mon[32362]: pgmap v787: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:14:12.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:12 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/3553825297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:14:12.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:12 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1117181986' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:14:12.846 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-mon[32952]: pgmap v787: 129 pgs: 129 active+clean; 4.1 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-27T00:14:12.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/3553825297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:14:12.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1117181986' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:14:12.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.936+0000 7f9d83200640 1 -- 172.21.15.40:0/2413176650 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f9d841a34c0 con 0x7f9d84104320 2024-04-27T00:14:12.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.947+0000 7f9d7aa00640 1 -- 172.21.15.40:0/2413176650 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) ==== 161+0+0 (secure 0 0 0) 0x7f9d7c09cde0 con 0x7f9d84104320 2024-04-27T00:14:12.946 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-27T00:14:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.950+0000 7f9d83200640 1 -- 172.21.15.40:0/2413176650 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9d680a3c40 msgr2=0x7f9d680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.950+0000 7f9d83200640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9d680a3c40 0x7f9d680a6100 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9d740095f0 tx=0x7f9d74030450 comp rx=0 tx=0).stop 2024-04-27T00:14:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.950+0000 7f9d83200640 1 -- 172.21.15.40:0/2413176650 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9d84104320 msgr2=0x7f9d841a2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.950+0000 7f9d83200640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9d84104320 0x7f9d841a2160 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c009fd0 tx=0x7f9d7c00ee50 comp rx=0 tx=0).stop 2024-04-27T00:14:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.950+0000 7f9d83200640 1 -- 172.21.15.40:0/2413176650 shutdown_connections 2024-04-27T00:14:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.950+0000 7f9d83200640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f9d84104320 0x7f9d841a2160 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.950+0000 7f9d83200640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f9d680a3c40 0x7f9d680a6100 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.950+0000 7f9d83200640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d84103960 0x7f9d8419b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.950+0000 7f9d83200640 1 --2- 172.21.15.40:0/2413176650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84102770 0x7f9d8419abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.950+0000 7f9d83200640 1 -- 172.21.15.40:0/2413176650 >> 172.21.15.40:0/2413176650 conn(0x7f9d840fdcc0 msgr2=0x7f9d84074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:12.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.951+0000 7f9d83200640 1 -- 172.21.15.40:0/2413176650 shutdown_connections 2024-04-27T00:14:12.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:12.951+0000 7f9d83200640 1 -- 172.21.15.40:0/2413176650 wait complete. 2024-04-27T00:14:12.974 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-27T00:14:13.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:12 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a[28091]: 2024-04-27T00:14:12.939+0000 7fd9ef200700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T00:14:13.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.168+0000 7feee7800640 1 -- 172.21.15.40:0/3767642775 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feee0108650 msgr2=0x7feee010aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:13.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.168+0000 7feee7800640 1 --2- 172.21.15.40:0/3767642775 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feee0108650 0x7feee010aaa0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7feed8009be0 tx=0x7feed802a6c0 comp rx=0 tx=0).stop 2024-04-27T00:14:13.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.168+0000 7feee7800640 1 -- 172.21.15.40:0/3767642775 shutdown_connections 2024-04-27T00:14:13.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.168+0000 7feee7800640 1 --2- 172.21.15.40:0/3767642775 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feee0108650 0x7feee010aaa0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:13.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.168+0000 7feee7800640 1 --2- 172.21.15.40:0/3767642775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feee0105d20 0x7feee0108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:13.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.168+0000 7feee7800640 1 --2- 172.21.15.40:0/3767642775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feee01033f0 0x7feee01057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:13.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.168+0000 7feee7800640 1 -- 172.21.15.40:0/3767642775 >> 172.21.15.40:0/3767642775 conn(0x7feee00fcde0 msgr2=0x7feee00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:13.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.168+0000 7feee7800640 1 -- 172.21.15.40:0/3767642775 shutdown_connections 2024-04-27T00:14:13.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.168+0000 7feee7800640 1 -- 172.21.15.40:0/3767642775 wait complete. 2024-04-27T00:14:13.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.169+0000 7feee7800640 1 Processor -- start 2024-04-27T00:14:13.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.169+0000 7feee7800640 1 -- start start 2024-04-27T00:14:13.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.170+0000 7feee7800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feee01033f0 0x7feee019adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:13.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.170+0000 7feee7800640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feee0105d20 0x7feee019b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:13.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.170+0000 7feee7800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feee0108650 0x7feee01a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:13.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.170+0000 7feee7800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feee010e4c0 con 0x7feee0108650 2024-04-27T00:14:13.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.170+0000 7feee7800640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7feee010e340 con 0x7feee0105d20 2024-04-27T00:14:13.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.170+0000 7feee7800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feee010e640 con 0x7feee01033f0 2024-04-27T00:14:13.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.170+0000 7feee4e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feee01033f0 0x7feee019adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:13.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.170+0000 7feee4e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feee01033f0 0x7feee019adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:44838/0 (socket says 172.21.15.40:44838) 2024-04-27T00:14:13.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.170+0000 7feee4e00640 1 -- 172.21.15.40:0/877147081 learned_addr learned my addr 172.21.15.40:0/877147081 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:13.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.170+0000 7feedfe00640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feee0105d20 0x7feee019b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:13.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.170+0000 7feee5800640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feee0108650 0x7feee01a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:13.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.171+0000 7feee4e00640 1 -- 172.21.15.40:0/877147081 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feee0105d20 msgr2=0x7feee019b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:13.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.171+0000 7feee4e00640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feee0105d20 0x7feee019b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:13.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.171+0000 7feee4e00640 1 -- 172.21.15.40:0/877147081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feee0108650 msgr2=0x7feee01a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:13.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.171+0000 7feee4e00640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feee0108650 0x7feee01a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:13.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.171+0000 7feee4e00640 1 -- 172.21.15.40:0/877147081 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee01a29c0 con 0x7feee01033f0 2024-04-27T00:14:13.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.171+0000 7feee5800640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feee0108650 0x7feee01a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:14:13.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.171+0000 7feee4e00640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feee01033f0 0x7feee019adb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feed400e9c0 tx=0x7feed400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:13.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.171+0000 7feedd600640 1 -- 172.21.15.40:0/877147081 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed400ccc0 con 0x7feee01033f0 2024-04-27T00:14:13.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.171+0000 7feedd600640 1 -- 172.21.15.40:0/877147081 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feed40054d0 con 0x7feee01033f0 2024-04-27T00:14:13.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.171+0000 7feedd600640 1 -- 172.21.15.40:0/877147081 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed4010430 con 0x7feee01033f0 2024-04-27T00:14:13.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.171+0000 7feee7800640 1 -- 172.21.15.40:0/877147081 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee01a2cb0 con 0x7feee01033f0 2024-04-27T00:14:13.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.171+0000 7feee7800640 1 -- 172.21.15.40:0/877147081 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feee01a31b0 con 0x7feee01033f0 2024-04-27T00:14:13.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.173+0000 7feee7800640 1 -- 172.21.15.40:0/877147081 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee010a140 con 0x7feee01033f0 2024-04-27T00:14:13.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.175+0000 7feedd600640 1 -- 172.21.15.40:0/877147081 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7feed4004f80 con 0x7feee01033f0 2024-04-27T00:14:13.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.176+0000 7feedd600640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feec80a3c40 0x7feec80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:13.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.176+0000 7feedd600640 1 -- 172.21.15.40:0/877147081 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9416+0+0 (secure 0 0 0) 0x7feed40d0ff0 con 0x7feee01033f0 2024-04-27T00:14:13.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.176+0000 7feedfe00640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feec80a3c40 0x7feec80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:13.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.177+0000 7feedfe00640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feec80a3c40 0x7feec80a6100 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7feee006a620 tx=0x7feecc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:13.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.180+0000 7feedd600640 1 -- 172.21.15.40:0/877147081 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7feed409d370 con 0x7feee01033f0 2024-04-27T00:14:13.298 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 27 00:14:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b[63150]: ignoring --setuser ceph since I am not root 2024-04-27T00:14:13.299 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 27 00:14:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b[63150]: ignoring --setgroup ceph since I am not root 2024-04-27T00:14:13.299 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 27 00:14:12 smithi080 ceph-mds[63154]: -- 172.21.15.80:0/2907141444 <== mon.2 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a11126ba40 con 0x55a11123e800 2024-04-27T00:14:13.299 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 27 00:14:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b[63150]: starting mds.b at 2024-04-27T00:14:13.299 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 27 00:14:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-h[62828]: ignoring --setuser ceph since I am not root 2024-04-27T00:14:13.299 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 27 00:14:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-h[62828]: ignoring --setgroup ceph since I am not root 2024-04-27T00:14:13.299 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 27 00:14:12 smithi080 ceph-mds[62832]: -- 172.21.15.80:0/3108011062 <== mon.2 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b8e8ea7a40 con 0x55b8e8e7a800 2024-04-27T00:14:13.299 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 27 00:14:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-h[62828]: starting mds.h at 2024-04-27T00:14:13.349 INFO:journalctl@ceph.mds.c.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-c[63472]: ignoring --setuser ceph since I am not root 2024-04-27T00:14:13.350 INFO:journalctl@ceph.mds.c.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-c[63472]: ignoring --setgroup ceph since I am not root 2024-04-27T00:14:13.350 INFO:journalctl@ceph.mds.c.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-mds[63476]: -- 172.21.15.103:0/71496859 <== mon.2 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559c884fba40 con 0x559c884ce800 2024-04-27T00:14:13.350 INFO:journalctl@ceph.mds.c.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-c[63472]: starting mds.c at 2024-04-27T00:14:13.350 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-f[63162]: ignoring --setuser ceph since I am not root 2024-04-27T00:14:13.350 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-f[63162]: ignoring --setgroup ceph since I am not root 2024-04-27T00:14:13.351 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-mds[63166]: -- 172.21.15.103:0/3344572707 <== mon.1 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56519810da40 con 0x5651980e0800 2024-04-27T00:14:13.351 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 27 00:14:13 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-f[63162]: starting mds.f at 2024-04-27T00:14:13.351 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i[63786]: ignoring --setuser ceph since I am not root 2024-04-27T00:14:13.351 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i[63786]: ignoring --setgroup ceph since I am not root 2024-04-27T00:14:13.351 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 27 00:14:12 smithi103 ceph-mds[63790]: -- 172.21.15.103:0/3857923855 <== mon.1 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56535ec7da40 con 0x56535ec50800 2024-04-27T00:14:13.351 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 27 00:14:13 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i[63786]: starting mds.i at 2024-04-27T00:14:13.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.427+0000 7feee7800640 1 -- 172.21.15.40:0/877147081 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7feee002dd10 con 0x7feee01033f0 2024-04-27T00:14:13.466 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g[67124]: ignoring --setuser ceph since I am not root 2024-04-27T00:14:13.466 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g[67124]: ignoring --setgroup ceph since I am not root 2024-04-27T00:14:13.466 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-mds[67128]: -- 172.21.15.40:0/3757385743 <== mon.2 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5653a075da40 con 0x5653a0730800 2024-04-27T00:14:13.466 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g[67124]: starting mds.g at 2024-04-27T00:14:13.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:13 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/2413176650' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-27T00:14:13.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:13 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-27T00:14:13.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:13 smithi080 ceph-mon[32362]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T00:14:13.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:13 smithi080 ceph-mon[32362]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T00:14:13.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:13 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-27T00:14:13.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:13 smithi080 ceph-mon[32362]: osdmap e95: 12 total, 12 up, 12 in 2024-04-27T00:14:13.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:13 smithi080 ceph-mon[32362]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-27T00:14:13.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:13 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/877147081' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T00:14:13.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:13 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T00:14:13.846 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:13 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/2413176650' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-27T00:14:13.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:13 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-27T00:14:13.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:13 smithi103 ceph-mon[32952]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T00:14:13.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:13 smithi103 ceph-mon[32952]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T00:14:13.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:13 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-27T00:14:13.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:13 smithi103 ceph-mon[32952]: osdmap e95: 12 total, 12 up, 12 in 2024-04-27T00:14:13.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:13 smithi103 ceph-mon[32952]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-27T00:14:13.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:13 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/877147081' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T00:14:13.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:13 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T00:14:13.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.956+0000 7feedd600640 1 -- 172.21.15.40:0/877147081 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) ==== 104+0+0 (secure 0 0 0) 0x7feed409cc70 con 0x7feee01033f0 2024-04-27T00:14:13.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.959+0000 7feee7800640 1 -- 172.21.15.40:0/877147081 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feec80a3c40 msgr2=0x7feec80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:13.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.959+0000 7feee7800640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feec80a3c40 0x7feec80a6100 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7feee006a620 tx=0x7feecc005f90 comp rx=0 tx=0).stop 2024-04-27T00:14:13.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.959+0000 7feee7800640 1 -- 172.21.15.40:0/877147081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feee01033f0 msgr2=0x7feee019adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:13.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.959+0000 7feee7800640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feee01033f0 0x7feee019adb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feed400e9c0 tx=0x7feed400ee80 comp rx=0 tx=0).stop 2024-04-27T00:14:13.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.960+0000 7feee7800640 1 -- 172.21.15.40:0/877147081 shutdown_connections 2024-04-27T00:14:13.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.960+0000 7feee7800640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feee0108650 0x7feee01a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:13.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.960+0000 7feee7800640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7feee0105d20 0x7feee019b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:13.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.960+0000 7feee7800640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7feec80a3c40 0x7feec80a6100 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:13.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.960+0000 7feee7800640 1 --2- 172.21.15.40:0/877147081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feee01033f0 0x7feee019adb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:13.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.960+0000 7feee7800640 1 -- 172.21.15.40:0/877147081 >> 172.21.15.40:0/877147081 conn(0x7feee00fcde0 msgr2=0x7feee0102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:13.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.961+0000 7feee7800640 1 -- 172.21.15.40:0/877147081 shutdown_connections 2024-04-27T00:14:13.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:13.961+0000 7feee7800640 1 -- 172.21.15.40:0/877147081 wait complete. 2024-04-27T00:14:13.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/2413176650' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-27T00:14:13.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-27T00:14:13.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-mon[28095]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T00:14:13.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-mon[28095]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T00:14:13.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-27T00:14:13.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-mon[28095]: osdmap e95: 12 total, 12 up, 12 in 2024-04-27T00:14:13.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-mon[28095]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-27T00:14:13.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/877147081' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T00:14:13.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:13 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T00:14:13.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-27T00:14:14.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.247+0000 7fe76fe00640 1 -- 172.21.15.40:0/1778719736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770109fa0 msgr2=0x7fe77010a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:14.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.247+0000 7fe76fe00640 1 --2- 172.21.15.40:0/1778719736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770109fa0 0x7fe77010a420 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe75c009900 tx=0x7fe75c02a3f0 comp rx=0 tx=0).stop 2024-04-27T00:14:14.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.247+0000 7fe76fe00640 1 -- 172.21.15.40:0/1778719736 shutdown_connections 2024-04-27T00:14:14.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.247+0000 7fe76fe00640 1 --2- 172.21.15.40:0/1778719736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe77010a960 0x7fe770111280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:14.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.247+0000 7fe76fe00640 1 --2- 172.21.15.40:0/1778719736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770109fa0 0x7fe77010a420 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:14.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.247+0000 7fe76fe00640 1 --2- 172.21.15.40:0/1778719736 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe770108e50 0x7fe770109270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:14.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.247+0000 7fe76fe00640 1 -- 172.21.15.40:0/1778719736 >> 172.21.15.40:0/1778719736 conn(0x7fe770077de0 msgr2=0x7fe77007a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:14.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.247+0000 7fe76fe00640 1 -- 172.21.15.40:0/1778719736 shutdown_connections 2024-04-27T00:14:14.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.247+0000 7fe76fe00640 1 -- 172.21.15.40:0/1778719736 wait complete. 2024-04-27T00:14:14.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.248+0000 7fe76fe00640 1 Processor -- start 2024-04-27T00:14:14.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.248+0000 7fe76fe00640 1 -- start start 2024-04-27T00:14:14.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.248+0000 7fe76fe00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe770108e50 0x7fe77019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:14.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.248+0000 7fe76fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770109fa0 0x7fe77019f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:14.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.248+0000 7fe76fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe77010a960 0x7fe7701a6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:14.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.248+0000 7fe76fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe7701149d0 con 0x7fe770109fa0 2024-04-27T00:14:14.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.248+0000 7fe76fe00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7fe770114850 con 0x7fe770108e50 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.248+0000 7fe76fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe770114b50 con 0x7fe77010a960 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.249+0000 7fe76ea00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe770108e50 0x7fe77019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.249+0000 7fe76e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770109fa0 0x7fe77019f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.249+0000 7fe76ea00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe770108e50 0x7fe77019f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.80:3300/0 says I am v2:172.21.15.40:36854/0 (socket says 172.21.15.40:36854) 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.249+0000 7fe76ea00640 1 -- 172.21.15.40:0/1939791701 learned_addr learned my addr 172.21.15.40:0/1939791701 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.249+0000 7fe76f400640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe77010a960 0x7fe7701a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.249+0000 7fe76ea00640 1 -- 172.21.15.40:0/1939791701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe77010a960 msgr2=0x7fe7701a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.249+0000 7fe76ea00640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe77010a960 0x7fe7701a6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.249+0000 7fe76ea00640 1 -- 172.21.15.40:0/1939791701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770109fa0 msgr2=0x7fe77019f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.249+0000 7fe76ea00640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770109fa0 0x7fe77019f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.249+0000 7fe76ea00640 1 -- 172.21.15.40:0/1939791701 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7701a6c70 con 0x7fe770108e50 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.249+0000 7fe76ea00640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe770108e50 0x7fe77019f0a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe76000d950 tx=0x7fe76000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.250+0000 7fe767400640 1 -- 172.21.15.40:0/1939791701 <== mon.2 v2:172.21.15.80:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe760017030 con 0x7fe770108e50 2024-04-27T00:14:14.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.250+0000 7fe76fe00640 1 -- 172.21.15.40:0/1939791701 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7701a6f60 con 0x7fe770108e50 2024-04-27T00:14:14.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.250+0000 7fe767400640 1 -- 172.21.15.40:0/1939791701 <== mon.2 v2:172.21.15.80:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe760004020 con 0x7fe770108e50 2024-04-27T00:14:14.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.250+0000 7fe767400640 1 -- 172.21.15.40:0/1939791701 <== mon.2 v2:172.21.15.80:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7600047a0 con 0x7fe770108e50 2024-04-27T00:14:14.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.250+0000 7fe76fe00640 1 -- 172.21.15.40:0/1939791701 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7701a73d0 con 0x7fe770108e50 2024-04-27T00:14:14.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.251+0000 7fe76fe00640 1 -- 172.21.15.40:0/1939791701 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7701101b0 con 0x7fe770108e50 2024-04-27T00:14:14.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.253+0000 7fe767400640 1 -- 172.21.15.40:0/1939791701 <== mon.2 v2:172.21.15.80:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe760009a30 con 0x7fe770108e50 2024-04-27T00:14:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.255+0000 7fe767400640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe7540a3c40 0x7fe7540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.255+0000 7fe767400640 1 -- 172.21.15.40:0/1939791701 <== mon.2 v2:172.21.15.80:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9416+0+0 (secure 0 0 0) 0x7fe7600d0b10 con 0x7fe770108e50 2024-04-27T00:14:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.255+0000 7fe76e000640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe7540a3c40 0x7fe7540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:14.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.256+0000 7fe76e000640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe7540a3c40 0x7fe7540a6100 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe75c005ab0 tx=0x7fe75c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:14.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.257+0000 7fe767400640 1 -- 172.21.15.40:0/1939791701 <== mon.2 v2:172.21.15.80:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe76009ce90 con 0x7fe770108e50 2024-04-27T00:14:14.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.458+0000 7fe76fe00640 1 -- 172.21.15.40:0/1939791701 --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe77002dad0 con 0x7fe770108e50 2024-04-27T00:14:14.798 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: pgmap v789: 129 pgs: 129 active+clean; 4.1 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-27T00:14:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T00:14:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T00:14:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: Cluster is now healthy 2024-04-27T00:14:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-27T00:14:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.80:6836/3910510870,v1:172.21.15.80:6837/3910510870] up:boot 2024-04-27T00:14:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.80:6838/367041237,v1:172.21.15.80:6839/367041237] up:boot 2024-04-27T00:14:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.103:6834/2168805019,v1:172.21.15.103:6835/2168805019] up:boot 2024-04-27T00:14:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.103:6836/1166877403,v1:172.21.15.103:6838/1166877403] up:boot 2024-04-27T00:14:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.103:6837/2564876493,v1:172.21.15.103:6839/2564876493] up:boot 2024-04-27T00:14:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: mds.? [v2:172.21.15.40:6838/4044303228,v1:172.21.15.40:6839/4044303228] up:boot 2024-04-27T00:14:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: fsmap 12 up:standby 2024-04-27T00:14:14.799 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-27T00:14:14.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-27T00:14:14.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-27T00:14:14.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-27T00:14:14.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-27T00:14:14.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-27T00:14:14.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1939791701' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:14:14.800 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:14 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: pgmap v789: 129 pgs: 129 active+clean; 4.1 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: Cluster is now healthy 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.80:6836/3910510870,v1:172.21.15.80:6837/3910510870] up:boot 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.80:6838/367041237,v1:172.21.15.80:6839/367041237] up:boot 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.103:6834/2168805019,v1:172.21.15.103:6835/2168805019] up:boot 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.103:6836/1166877403,v1:172.21.15.103:6838/1166877403] up:boot 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.103:6837/2564876493,v1:172.21.15.103:6839/2564876493] up:boot 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: mds.? [v2:172.21.15.40:6838/4044303228,v1:172.21.15.40:6839/4044303228] up:boot 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: fsmap 12 up:standby 2024-04-27T00:14:14.847 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-27T00:14:14.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-27T00:14:14.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-27T00:14:14.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-27T00:14:14.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-27T00:14:14.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-27T00:14:14.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1939791701' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:14:14.848 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:14 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:14:14.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.960+0000 7fe767400640 1 -- 172.21.15.40:0/1939791701 <== mon.2 v2:172.21.15.80:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v96) ==== 181+0+0 (secure 0 0 0) 0x7fe76009c790 con 0x7fe770108e50 2024-04-27T00:14:14.960 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-04-27T00:14:14.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.965+0000 7fe76fe00640 1 -- 172.21.15.40:0/1939791701 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe7540a3c40 msgr2=0x7fe7540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:14.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.965+0000 7fe76fe00640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe7540a3c40 0x7fe7540a6100 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe75c005ab0 tx=0x7fe75c005a20 comp rx=0 tx=0).stop 2024-04-27T00:14:14.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.965+0000 7fe76fe00640 1 -- 172.21.15.40:0/1939791701 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe770108e50 msgr2=0x7fe77019f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:14.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.966+0000 7fe76fe00640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe770108e50 0x7fe77019f0a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe76000d950 tx=0x7fe76000de10 comp rx=0 tx=0).stop 2024-04-27T00:14:14.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.966+0000 7fe76fe00640 1 -- 172.21.15.40:0/1939791701 shutdown_connections 2024-04-27T00:14:14.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.966+0000 7fe76fe00640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7fe7540a3c40 0x7fe7540a6100 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:14.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.966+0000 7fe76fe00640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe77010a960 0x7fe7701a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:14.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.966+0000 7fe76fe00640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770109fa0 0x7fe77019f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:14.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.966+0000 7fe76fe00640 1 --2- 172.21.15.40:0/1939791701 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fe770108e50 0x7fe77019f0a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:14.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.966+0000 7fe76fe00640 1 -- 172.21.15.40:0/1939791701 >> 172.21.15.40:0/1939791701 conn(0x7fe770077de0 msgr2=0x7fe77010d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:14.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.966+0000 7fe76fe00640 1 -- 172.21.15.40:0/1939791701 shutdown_connections 2024-04-27T00:14:14.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:14.966+0000 7fe76fe00640 1 -- 172.21.15.40:0/1939791701 wait complete. 2024-04-27T00:14:14.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: pgmap v789: 129 pgs: 129 active+clean; 4.1 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-27T00:14:14.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T00:14:14.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T00:14:14.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: Cluster is now healthy 2024-04-27T00:14:14.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-27T00:14:14.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.80:6836/3910510870,v1:172.21.15.80:6837/3910510870] up:boot 2024-04-27T00:14:14.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.80:6838/367041237,v1:172.21.15.80:6839/367041237] up:boot 2024-04-27T00:14:14.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.103:6834/2168805019,v1:172.21.15.103:6835/2168805019] up:boot 2024-04-27T00:14:14.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.103:6836/1166877403,v1:172.21.15.103:6838/1166877403] up:boot 2024-04-27T00:14:14.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.103:6837/2564876493,v1:172.21.15.103:6839/2564876493] up:boot 2024-04-27T00:14:14.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: mds.? [v2:172.21.15.40:6838/4044303228,v1:172.21.15.40:6839/4044303228] up:boot 2024-04-27T00:14:14.967 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: fsmap 12 up:standby 2024-04-27T00:14:14.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-27T00:14:14.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-27T00:14:14.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-27T00:14:14.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-27T00:14:14.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-27T00:14:14.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: from='mgr.14152 172.21.15.40:0/389302245' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-27T00:14:14.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1939791701' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:14:14.968 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:14 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:14:14.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-27T00:14:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.195+0000 7f0a26600640 1 -- 172.21.15.40:0/1695946620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a2010a980 msgr2=0x7f0a201112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.195+0000 7f0a26600640 1 --2- 172.21.15.40:0/1695946620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a2010a980 0x7f0a201112a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009860 tx=0x7f0a1802a340 comp rx=0 tx=0).stop 2024-04-27T00:14:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.195+0000 7f0a26600640 1 -- 172.21.15.40:0/1695946620 shutdown_connections 2024-04-27T00:14:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.195+0000 7f0a26600640 1 --2- 172.21.15.40:0/1695946620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a2010a980 0x7f0a201112a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.195+0000 7f0a26600640 1 --2- 172.21.15.40:0/1695946620 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a20109fc0 0x7f0a2010a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.195+0000 7f0a26600640 1 --2- 172.21.15.40:0/1695946620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a20108e70 0x7f0a20109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.195+0000 7f0a26600640 1 -- 172.21.15.40:0/1695946620 >> 172.21.15.40:0/1695946620 conn(0x7f0a20077e00 msgr2=0x7f0a2007a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.195+0000 7f0a26600640 1 -- 172.21.15.40:0/1695946620 shutdown_connections 2024-04-27T00:14:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.195+0000 7f0a26600640 1 -- 172.21.15.40:0/1695946620 wait complete. 2024-04-27T00:14:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.197+0000 7f0a26600640 1 Processor -- start 2024-04-27T00:14:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.197+0000 7f0a26600640 1 -- start start 2024-04-27T00:14:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.197+0000 7f0a26600640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a20108e70 0x7f0a2019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.197+0000 7f0a26600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a20109fc0 0x7f0a2019f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.197+0000 7f0a26600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a2010a980 0x7f0a201a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.197+0000 7f0a26600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0a201149d0 con 0x7f0a20109fc0 2024-04-27T00:14:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.197+0000 7f0a26600640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f0a20114850 con 0x7f0a20108e70 2024-04-27T00:14:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.197+0000 7f0a26600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a20114b50 con 0x7f0a2010a980 2024-04-27T00:14:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.197+0000 7f0a1fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a20109fc0 0x7f0a2019f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.198+0000 7f0a1fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a20109fc0 0x7f0a2019f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54220/0 (socket says 172.21.15.40:54220) 2024-04-27T00:14:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.198+0000 7f0a1fe00640 1 -- 172.21.15.40:0/1448744784 learned_addr learned my addr 172.21.15.40:0/1448744784 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.198+0000 7f0a25200640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a20108e70 0x7f0a2019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.198+0000 7f0a25c00640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a2010a980 0x7f0a201a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.198+0000 7f0a1fe00640 1 -- 172.21.15.40:0/1448744784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a2010a980 msgr2=0x7f0a201a65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.198+0000 7f0a1fe00640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a2010a980 0x7f0a201a65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.198+0000 7f0a1fe00640 1 -- 172.21.15.40:0/1448744784 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a20108e70 msgr2=0x7f0a2019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.198+0000 7f0a1fe00640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a20108e70 0x7f0a2019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.198+0000 7f0a1fe00640 1 -- 172.21.15.40:0/1448744784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a201a6c30 con 0x7f0a20109fc0 2024-04-27T00:14:15.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.198+0000 7f0a1fe00640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a20109fc0 0x7f0a2019f550 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800e9c0 tx=0x7f0a0800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:15.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.199+0000 7f0a1d600640 1 -- 172.21.15.40:0/1448744784 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0800cd80 con 0x7f0a20109fc0 2024-04-27T00:14:15.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.199+0000 7f0a1d600640 1 -- 172.21.15.40:0/1448744784 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a080035f0 con 0x7f0a20109fc0 2024-04-27T00:14:15.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.199+0000 7f0a26600640 1 -- 172.21.15.40:0/1448744784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a201a6f20 con 0x7f0a20109fc0 2024-04-27T00:14:15.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.199+0000 7f0a1d600640 1 -- 172.21.15.40:0/1448744784 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08010690 con 0x7f0a20109fc0 2024-04-27T00:14:15.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.199+0000 7f0a26600640 1 -- 172.21.15.40:0/1448744784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a201a7390 con 0x7f0a20109fc0 2024-04-27T00:14:15.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.200+0000 7f0a26600640 1 -- 172.21.15.40:0/1448744784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a20110180 con 0x7f0a20109fc0 2024-04-27T00:14:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.202+0000 7f0a1d600640 1 -- 172.21.15.40:0/1448744784 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0a08004000 con 0x7f0a20109fc0 2024-04-27T00:14:15.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.203+0000 7f0a1d600640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0a040a3c40 0x7f0a040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:15.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.204+0000 7f0a1d600640 1 -- 172.21.15.40:0/1448744784 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8920+0+0 (secure 0 0 0) 0x7f0a080d0d90 con 0x7f0a20109fc0 2024-04-27T00:14:15.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.204+0000 7f0a25200640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0a040a3c40 0x7f0a040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:15.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.204+0000 7f0a25200640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0a040a3c40 0x7f0a040a6100 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0a10005fd0 tx=0x7f0a10010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:15.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.208+0000 7f0a1d600640 1 -- 172.21.15.40:0/1448744784 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0a0809d300 con 0x7f0a20109fc0 2024-04-27T00:14:15.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.434+0000 7f0a26600640 1 -- 172.21.15.40:0/1448744784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0a20063530 con 0x7f0a20109fc0 2024-04-27T00:14:15.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.970+0000 7f0a1d600640 1 -- 172.21.15.40:0/1448744784 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v97) ==== 169+0+0 (secure 0 0 0) 0x7f0a0809cc00 con 0x7f0a20109fc0 2024-04-27T00:14:15.970 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-04-27T00:14:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.974+0000 7f0a26600640 1 -- 172.21.15.40:0/1448744784 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0a040a3c40 msgr2=0x7f0a040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.974+0000 7f0a26600640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0a040a3c40 0x7f0a040a6100 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0a10005fd0 tx=0x7f0a10010040 comp rx=0 tx=0).stop 2024-04-27T00:14:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.974+0000 7f0a26600640 1 -- 172.21.15.40:0/1448744784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a20109fc0 msgr2=0x7f0a2019f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.974+0000 7f0a26600640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a20109fc0 0x7f0a2019f550 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800e9c0 tx=0x7f0a0800ee80 comp rx=0 tx=0).stop 2024-04-27T00:14:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.974+0000 7f0a26600640 1 -- 172.21.15.40:0/1448744784 shutdown_connections 2024-04-27T00:14:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.974+0000 7f0a26600640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f0a040a3c40 0x7f0a040a6100 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.974+0000 7f0a26600640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a2010a980 0x7f0a201a65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.974+0000 7f0a26600640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a20109fc0 0x7f0a2019f550 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.974+0000 7f0a26600640 1 --2- 172.21.15.40:0/1448744784 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f0a20108e70 0x7f0a2019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.974+0000 7f0a26600640 1 -- 172.21.15.40:0/1448744784 >> 172.21.15.40:0/1448744784 conn(0x7f0a20077e00 msgr2=0x7f0a2010d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.975+0000 7f0a26600640 1 -- 172.21.15.40:0/1448744784 shutdown_connections 2024-04-27T00:14:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:15.975+0000 7f0a26600640 1 -- 172.21.15.40:0/1448744784 wait complete. 2024-04-27T00:14:16.000 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-27T00:14:16.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:15 smithi040 ceph-mon[28095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T00:14:16.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:15 smithi040 ceph-mon[28095]: osdmap e96: 12 total, 12 up, 12 in 2024-04-27T00:14:16.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:15 smithi040 ceph-mon[28095]: pgmap v791: 97 pgs: 97 active+clean; 577 KiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-27T00:14:16.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:15 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1448744784' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:14:16.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.220+0000 7f1dc1c00640 1 -- 172.21.15.40:0/923792470 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc10aa40 msgr2=0x7f1dbc10ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:16.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.220+0000 7f1dc1c00640 1 --2- 172.21.15.40:0/923792470 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc10aa40 0x7f1dbc10ce30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1da8009860 tx=0x7f1da802a340 comp rx=0 tx=0).stop 2024-04-27T00:14:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.220+0000 7f1dc1c00640 1 -- 172.21.15.40:0/923792470 shutdown_connections 2024-04-27T00:14:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.220+0000 7f1dc1c00640 1 --2- 172.21.15.40:0/923792470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc10fd80 0x7f1dbc112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.220+0000 7f1dc1c00640 1 --2- 172.21.15.40:0/923792470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dbc10d370 0x7f1dbc10f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.220+0000 7f1dc1c00640 1 --2- 172.21.15.40:0/923792470 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc10aa40 0x7f1dbc10ce30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.220+0000 7f1dc1c00640 1 -- 172.21.15.40:0/923792470 >> 172.21.15.40:0/923792470 conn(0x7f1dbc06fb40 msgr2=0x7f1dbc071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.220+0000 7f1dc1c00640 1 -- 172.21.15.40:0/923792470 shutdown_connections 2024-04-27T00:14:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.220+0000 7f1dc1c00640 1 -- 172.21.15.40:0/923792470 wait complete. 2024-04-27T00:14:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.221+0000 7f1dc1c00640 1 Processor -- start 2024-04-27T00:14:16.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.221+0000 7f1dc1c00640 1 -- start start 2024-04-27T00:14:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.221+0000 7f1dc1c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dbc10aa40 0x7f1dbc1a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.221+0000 7f1dc1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc10d370 0x7f1dbc1a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.221+0000 7f1dc1c00640 1 --2- >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc10fd80 0x7f1dbc1a89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.221+0000 7f1dc1c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc115bb0 con 0x7f1dbc10d370 2024-04-27T00:14:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.221+0000 7f1dc1c00640 1 -- --> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc115a30 con 0x7f1dbc10fd80 2024-04-27T00:14:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.221+0000 7f1dc1c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc115d30 con 0x7f1dbc10aa40 2024-04-27T00:14:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.222+0000 7f1dba000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc10d370 0x7f1dbc1a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.222+0000 7f1dba000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc10d370 0x7f1dbc1a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54244/0 (socket says 172.21.15.40:54244) 2024-04-27T00:14:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.222+0000 7f1dba000640 1 -- 172.21.15.40:0/863425088 learned_addr learned my addr 172.21.15.40:0/863425088 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-04-27T00:14:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.222+0000 7f1dbaa00640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dbc10aa40 0x7f1dbc1a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.222+0000 7f1dbb400640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc10fd80 0x7f1dbc1a89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.222+0000 7f1dba000640 1 -- 172.21.15.40:0/863425088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dbc10aa40 msgr2=0x7f1dbc1a1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.222+0000 7f1dba000640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dbc10aa40 0x7f1dbc1a1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.222+0000 7f1dba000640 1 -- 172.21.15.40:0/863425088 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc10fd80 msgr2=0x7f1dbc1a89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.222+0000 7f1dba000640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc10fd80 0x7f1dbc1a89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.222+0000 7f1dba000640 1 -- 172.21.15.40:0/863425088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dbc1a9020 con 0x7f1dbc10d370 2024-04-27T00:14:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.222+0000 7f1dbaa00640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dbc10aa40 0x7f1dbc1a1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:14:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.223+0000 7f1dba000640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc10d370 0x7f1dbc1a1940 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00e9f0 tx=0x7f1dac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.223+0000 7f1db3400640 1 -- 172.21.15.40:0/863425088 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac00ccc0 con 0x7f1dbc10d370 2024-04-27T00:14:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.223+0000 7f1db3400640 1 -- 172.21.15.40:0/863425088 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dac0054d0 con 0x7f1dbc10d370 2024-04-27T00:14:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.223+0000 7f1dc1c00640 1 -- 172.21.15.40:0/863425088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dbc1a9310 con 0x7f1dbc10d370 2024-04-27T00:14:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.223+0000 7f1db3400640 1 -- 172.21.15.40:0/863425088 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac010430 con 0x7f1dbc10d370 2024-04-27T00:14:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.223+0000 7f1dc1c00640 1 -- 172.21.15.40:0/863425088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dbc1a9810 con 0x7f1dbc10d370 2024-04-27T00:14:16.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.224+0000 7f1dc1c00640 1 -- 172.21.15.40:0/863425088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dbc1a9ad0 con 0x7f1dbc10d370 2024-04-27T00:14:16.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.227+0000 7f1db3400640 1 -- 172.21.15.40:0/863425088 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1dac0046c0 con 0x7f1dbc10d370 2024-04-27T00:14:16.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.228+0000 7f1db3400640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1da00a3c40 0x7f1da00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:14:16.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.228+0000 7f1db3400640 1 -- 172.21.15.40:0/863425088 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8324+0+0 (secure 0 0 0) 0x7f1dac0d0990 con 0x7f1dbc10d370 2024-04-27T00:14:16.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.228+0000 7f1dbaa00640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1da00a3c40 0x7f1da00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:14:16.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.229+0000 7f1dbaa00640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1da00a3c40 0x7f1da00a6100 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1da802a930 tx=0x7f1da8030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:14:16.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.233+0000 7f1db3400640 1 -- 172.21.15.40:0/863425088 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1dac09d150 con 0x7f1dbc10d370 2024-04-27T00:14:16.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:15 smithi080 ceph-mon[32362]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T00:14:16.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:15 smithi080 ceph-mon[32362]: osdmap e96: 12 total, 12 up, 12 in 2024-04-27T00:14:16.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:15 smithi080 ceph-mon[32362]: pgmap v791: 97 pgs: 97 active+clean; 577 KiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-27T00:14:16.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:15 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1448744784' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:14:16.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:15 smithi103 ceph-mon[32952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T00:14:16.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:15 smithi103 ceph-mon[32952]: osdmap e96: 12 total, 12 up, 12 in 2024-04-27T00:14:16.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:15 smithi103 ceph-mon[32952]: pgmap v791: 97 pgs: 97 active+clean; 577 KiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-27T00:14:16.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:15 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1448744784' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:14:16.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.439+0000 7f1dc1c00640 1 -- 172.21.15.40:0/863425088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1dbc1aa1d0 con 0x7f1dbc10d370 2024-04-27T00:14:16.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.973+0000 7f1db3400640 1 -- 172.21.15.40:0/863425088 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v98) ==== 178+0+0 (secure 0 0 0) 0x7f1dac09ca50 con 0x7f1dbc10d370 2024-04-27T00:14:16.973 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data_ec' removed 2024-04-27T00:14:16.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.977+0000 7f1dc1c00640 1 -- 172.21.15.40:0/863425088 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1da00a3c40 msgr2=0x7f1da00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:16.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.977+0000 7f1dc1c00640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1da00a3c40 0x7f1da00a6100 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1da802a930 tx=0x7f1da8030450 comp rx=0 tx=0).stop 2024-04-27T00:14:16.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.977+0000 7f1dc1c00640 1 -- 172.21.15.40:0/863425088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc10d370 msgr2=0x7f1dbc1a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:14:16.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.977+0000 7f1dc1c00640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc10d370 0x7f1dbc1a1940 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00e9f0 tx=0x7f1dac00eeb0 comp rx=0 tx=0).stop 2024-04-27T00:14:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.978+0000 7f1dc1c00640 1 -- 172.21.15.40:0/863425088 shutdown_connections 2024-04-27T00:14:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.978+0000 7f1dc1c00640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7f1dbc10fd80 0x7f1dbc1a89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.978+0000 7f1dc1c00640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dbc10d370 0x7f1dbc1a1940 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.978+0000 7f1dc1c00640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.40:6800/3777720410,v1:172.21.15.40:6801/3777720410] conn(0x7f1da00a3c40 0x7f1da00a6100 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.978+0000 7f1dc1c00640 1 --2- 172.21.15.40:0/863425088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dbc10aa40 0x7f1dbc1a1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:14:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.978+0000 7f1dc1c00640 1 -- 172.21.15.40:0/863425088 >> 172.21.15.40:0/863425088 conn(0x7f1dbc06fb40 msgr2=0x7f1dbc109e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:14:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.978+0000 7f1dc1c00640 1 -- 172.21.15.40:0/863425088 shutdown_connections 2024-04-27T00:14:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-04-27T00:14:16.978+0000 7f1dc1c00640 1 -- 172.21.15.40:0/863425088 wait complete. 2024-04-27T00:14:17.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:14:17.081 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:14:17.115 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:14:17.147 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-27T00:14:17.147 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T00:14:17.205 DEBUG:teuthology.orchestra.run.smithi080:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T00:14:17.223 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T00:14:17.241 INFO:tasks.cephadm:Stopping all daemons... 2024-04-27T00:14:17.241 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-27T00:14:17.241 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.a 2024-04-27T00:14:17.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:16 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/1448744784' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T00:14:17.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:16 smithi080 ceph-mon[32362]: osdmap e97: 12 total, 12 up, 12 in 2024-04-27T00:14:17.298 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:16 smithi080 ceph-mon[32362]: from='client.? 172.21.15.40:0/863425088' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:14:17.321 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:16 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/1448744784' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T00:14:17.321 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:16 smithi040 ceph-mon[28095]: osdmap e97: 12 total, 12 up, 12 in 2024-04-27T00:14:17.322 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:16 smithi040 ceph-mon[28095]: from='client.? 172.21.15.40:0/863425088' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:14:17.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:16 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/1448744784' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T00:14:17.346 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:16 smithi103 ceph-mon[32952]: osdmap e97: 12 total, 12 up, 12 in 2024-04-27T00:14:17.347 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:16 smithi103 ceph-mon[32952]: from='client.? 172.21.15.40:0/863425088' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:14:17.715 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:17 smithi040 systemd[1]: Stopping Ceph mon.a for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:14:18.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:17 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a[28091]: 2024-04-27T00:14:17.897+0000 7fd9f6000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:14:18.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:17 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a[28091]: 2024-04-27T00:14:17.897+0000 7fd9f6000700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-27T00:14:18.219 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:18 smithi040 podman[86539]: 2024-04-27 00:14:18.001808033 +0000 UTC m=+0.290399903 container died 9cf2eb2d66cdc8fe654d31f941f9251881ec8123740de6205165f0d72680cfd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:14:18.849 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:18 smithi040 podman[86539]: 2024-04-27 00:14:18.514117757 +0000 UTC m=+0.802709620 container cleanup 9cf2eb2d66cdc8fe654d31f941f9251881ec8123740de6205165f0d72680cfd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T00:14:18.849 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:18 smithi040 bash[86539]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a 2024-04-27T00:14:19.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Apr 27 00:14:18 smithi040 podman[86553]: 2024-04-27 00:14:18.848955503 +0000 UTC m=+0.835415040 container remove 9cf2eb2d66cdc8fe654d31f941f9251881ec8123740de6205165f0d72680cfd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-27T00:14:19.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.a.service' 2024-04-27T00:14:19.548 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:14:19.548 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-27T00:14:19.548 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-27T00:14:19.548 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.b 2024-04-27T00:14:19.888 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:19 smithi080 systemd[1]: Stopping Ceph mon.b for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:14:20.296 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:19 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-b[32358]: 2024-04-27T00:14:19.887+0000 7f1e3bdfe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:14:20.296 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:19 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-b[32358]: 2024-04-27T00:14:19.888+0000 7f1e3bdfe700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-27T00:14:20.296 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:20 smithi080 podman[67835]: 2024-04-27 00:14:20.010560011 +0000 UTC m=+0.266192755 container died 766b7e4712e2d59e63a3019d0c1331d5dd505d61e538baa270c1c5c8dea7cb39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-27T00:14:20.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:20 smithi080 podman[67835]: 2024-04-27 00:14:20.414297802 +0000 UTC m=+0.669930915 container cleanup 766b7e4712e2d59e63a3019d0c1331d5dd505d61e538baa270c1c5c8dea7cb39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-27T00:14:20.548 INFO:journalctl@ceph.mon.b.smithi080.stdout:Apr 27 00:14:20 smithi080 bash[67835]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-b 2024-04-27T00:14:20.836 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.b.service' 2024-04-27T00:14:20.888 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:14:20.888 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-27T00:14:20.889 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-27T00:14:20.889 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.c 2024-04-27T00:14:21.225 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:20 smithi103 systemd[1]: Stopping Ceph mon.c for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:14:21.596 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:21 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-c[32926]: 2024-04-27T00:14:21.225+0000 7fa483e25700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:14:21.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:21 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-c[32926]: 2024-04-27T00:14:21.225+0000 7fa483e25700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-27T00:14:21.597 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:21 smithi103 podman[69109]: 2024-04-27 00:14:21.27720981 +0000 UTC m=+0.188685823 container died 22caa4266198c439967a131dff07031dfbc911ca8b5a4855ad4f2635cfdeb5a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-27T00:14:21.947 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:21 smithi103 podman[69109]: 2024-04-27 00:14:21.693594975 +0000 UTC m=+0.605070978 container cleanup 22caa4266198c439967a131dff07031dfbc911ca8b5a4855ad4f2635cfdeb5a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-c, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-27T00:14:21.947 INFO:journalctl@ceph.mon.c.smithi103.stdout:Apr 27 00:14:21 smithi103 bash[69109]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mon-c 2024-04-27T00:14:22.080 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mon.c.service' 2024-04-27T00:14:22.122 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:14:22.122 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-27T00:14:22.122 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-27T00:14:22.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.x 2024-04-27T00:14:22.466 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 27 00:14:22 smithi040 systemd[1]: Stopping Ceph mgr.x for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:14:23.158 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 27 00:14:22 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-27T00:14:22.720+0000 7f355b000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:14:23.158 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 27 00:14:22 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x[28338]: 2024-04-27T00:14:22.720+0000 7f355b000700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-27T00:14:23.158 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 27 00:14:22 smithi040 podman[86660]: 2024-04-27 00:14:22.833346803 +0000 UTC m=+0.271942785 container died 24a7832aa9b89649af7e873f241423459df6a1260301487a9d59ed83d836998a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-27T00:14:23.466 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 27 00:14:23 smithi040 podman[86660]: 2024-04-27 00:14:23.369713443 +0000 UTC m=+0.808309409 container cleanup 24a7832aa9b89649af7e873f241423459df6a1260301487a9d59ed83d836998a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-27T00:14:23.466 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 27 00:14:23 smithi040 bash[86660]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x 2024-04-27T00:14:23.843 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 27 00:14:23 smithi040 podman[86673]: 2024-04-27 00:14:23.587671094 +0000 UTC m=+0.744723894 container remove 24a7832aa9b89649af7e873f241423459df6a1260301487a9d59ed83d836998a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-27T00:14:24.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-27T00:14:24.127 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 27 00:14:24 smithi040 systemd[1]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.x.service: Deactivated successfully. 2024-04-27T00:14:24.127 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 27 00:14:24 smithi040 systemd[1]: Stopped Ceph mgr.x for 78cff6ea-0427-11ef-bc93-c7b262605968. 2024-04-27T00:14:24.127 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Apr 27 00:14:24 smithi040 systemd[1]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.x.service: Consumed 2min 31.201s CPU time. 2024-04-27T00:14:24.166 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:14:24.166 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-27T00:14:24.166 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-27T00:14:24.167 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.y 2024-04-27T00:14:24.522 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 27 00:14:24 smithi080 systemd[1]: Stopping Ceph mgr.y for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:14:24.798 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 27 00:14:24 smithi080 podman[67961]: 2024-04-27 00:14:24.523293195 +0000 UTC m=+0.168520936 container died c49e7b5b507ae5d4f82f40755deac3e62e8d016f32794fef2a0913612f5b9d58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-27T00:14:25.296 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 27 00:14:24 smithi080 podman[67961]: 2024-04-27 00:14:24.978510579 +0000 UTC m=+0.623738327 container cleanup c49e7b5b507ae5d4f82f40755deac3e62e8d016f32794fef2a0913612f5b9d58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-04-27T00:14:25.296 INFO:journalctl@ceph.mgr.y.smithi080.stdout:Apr 27 00:14:24 smithi080 bash[67961]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-y 2024-04-27T00:14:25.452 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-27T00:14:25.486 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:14:25.486 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-27T00:14:25.486 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-27T00:14:25.486 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.z 2024-04-27T00:14:25.788 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 27 00:14:25 smithi103 systemd[1]: Stopping Ceph mgr.z for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:14:26.060 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 27 00:14:25 smithi103 podman[69231]: 2024-04-27 00:14:25.788897652 +0000 UTC m=+0.114001735 container died 3a8618c5cbded6abddf27877ecada445365418f0b723065ce8a25f125a19a2ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:14:26.346 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 27 00:14:26 smithi103 podman[69231]: 2024-04-27 00:14:26.152746559 +0000 UTC m=+0.477850642 container cleanup 3a8618c5cbded6abddf27877ecada445365418f0b723065ce8a25f125a19a2ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424) 2024-04-27T00:14:26.347 INFO:journalctl@ceph.mgr.z.smithi103.stdout:Apr 27 00:14:26 smithi103 bash[69231]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mgr-z 2024-04-27T00:14:26.570 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-27T00:14:26.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:14:26.603 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-27T00:14:26.603 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-27T00:14:26.604 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.0 2024-04-27T00:14:26.965 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:26 smithi040 systemd[1]: Stopping Ceph osd.0 for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:14:27.466 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:27 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0[40450]: 2024-04-27T00:14:27.204+0000 7f752b000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:14:27.466 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:27 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0[40450]: 2024-04-27T00:14:27.204+0000 7f752b000700 -1 osd.0 98 *** Got signal Terminated *** 2024-04-27T00:14:27.466 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:27 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0[40450]: 2024-04-27T00:14:27.204+0000 7f752b000700 -1 osd.0 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:14:32.966 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:32 smithi040 podman[86781]: 2024-04-27 00:14:32.696669035 +0000 UTC m=+5.631284309 container died 717b9ecf56d9280a230812e1a89d3acb2c41f27360e77f1d5576003ec177212d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True) 2024-04-27T00:14:33.325 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:33 smithi040 podman[86781]: 2024-04-27 00:14:33.115798131 +0000 UTC m=+6.050413433 container cleanup 717b9ecf56d9280a230812e1a89d3acb2c41f27360e77f1d5576003ec177212d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True) 2024-04-27T00:14:33.325 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:33 smithi040 bash[86781]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0 2024-04-27T00:14:33.715 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:33 smithi040 podman[86800]: 2024-04-27 00:14:33.325305665 +0000 UTC m=+0.608854095 container remove 717b9ecf56d9280a230812e1a89d3acb2c41f27360e77f1d5576003ec177212d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:14:34.465 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:34 smithi040 podman[86887]: 2024-04-27 00:14:34.120843938 +0000 UTC m=+0.038600509 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:14:34.466 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:34 smithi040 podman[86887]: 2024-04-27 00:14:34.242967112 +0000 UTC m=+0.160723716 container create a6f70420d48a1f8e2e40dd3a8d7986c2cae52f6a1ef6b59ed875535bcba79d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-27T00:14:34.965 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:34 smithi040 podman[86887]: 2024-04-27 00:14:34.530178082 +0000 UTC m=+0.447934693 container init a6f70420d48a1f8e2e40dd3a8d7986c2cae52f6a1ef6b59ed875535bcba79d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS) 2024-04-27T00:14:34.966 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:34 smithi040 podman[86887]: 2024-04-27 00:14:34.543411407 +0000 UTC m=+0.461168014 container start a6f70420d48a1f8e2e40dd3a8d7986c2cae52f6a1ef6b59ed875535bcba79d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-27T00:14:34.966 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:34 smithi040 podman[86887]: 2024-04-27 00:14:34.645678144 +0000 UTC m=+0.563434802 container attach a6f70420d48a1f8e2e40dd3a8d7986c2cae52f6a1ef6b59ed875535bcba79d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T00:14:35.379 INFO:journalctl@ceph.osd.0.smithi040.stdout:Apr 27 00:14:35 smithi040 podman[86887]: 2024-04-27 00:14:35.015105261 +0000 UTC m=+0.932861872 container died a6f70420d48a1f8e2e40dd3a8d7986c2cae52f6a1ef6b59ed875535bcba79d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-27T00:14:35.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.0.service' 2024-04-27T00:14:35.924 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:14:35.925 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-27T00:14:35.925 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-27T00:14:35.925 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.1 2024-04-27T00:14:36.232 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:35 smithi080 systemd[1]: Stopping Ceph osd.1 for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:14:36.548 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:36 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1[36955]: 2024-04-27T00:14:36.231+0000 7f00e7a15700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:14:36.548 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:36 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1[36955]: 2024-04-27T00:14:36.231+0000 7f00e7a15700 -1 osd.1 98 *** Got signal Terminated *** 2024-04-27T00:14:36.549 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:36 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1[36955]: 2024-04-27T00:14:36.231+0000 7f00e7a15700 -1 osd.1 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:14:42.048 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:41 smithi080 podman[68088]: 2024-04-27 00:14:41.77129618 +0000 UTC m=+5.649330683 container died 204147e7ecab7140f338d554655778de1db2f0be9a01c36207f7115eabd2170f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-27T00:14:42.402 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:42 smithi080 podman[68088]: 2024-04-27 00:14:42.149541062 +0000 UTC m=+6.027575573 container cleanup 204147e7ecab7140f338d554655778de1db2f0be9a01c36207f7115eabd2170f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-27T00:14:42.402 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:42 smithi080 bash[68088]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1 2024-04-27T00:14:42.798 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:42 smithi080 podman[68107]: 2024-04-27 00:14:42.453530135 +0000 UTC m=+0.681906140 container remove 204147e7ecab7140f338d554655778de1db2f0be9a01c36207f7115eabd2170f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-27T00:14:43.105 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:42 smithi080 podman[68206]: 2024-04-27 00:14:42.731941696 +0000 UTC m=+0.019309971 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:14:43.105 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:42 smithi080 podman[68206]: 2024-04-27 00:14:42.860931166 +0000 UTC m=+0.148299420 container create c977ab938f1684b95d549644ae0c48a42d69ea917bab79704a0f8efcda5b17f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:14:43.407 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:43 smithi080 podman[68206]: 2024-04-27 00:14:43.105816841 +0000 UTC m=+0.393185116 container init c977ab938f1684b95d549644ae0c48a42d69ea917bab79704a0f8efcda5b17f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0) 2024-04-27T00:14:43.407 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:43 smithi080 podman[68206]: 2024-04-27 00:14:43.109225779 +0000 UTC m=+0.396594043 container start c977ab938f1684b95d549644ae0c48a42d69ea917bab79704a0f8efcda5b17f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-04-27T00:14:43.407 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:43 smithi080 podman[68206]: 2024-04-27 00:14:43.189294406 +0000 UTC m=+0.476662688 container attach c977ab938f1684b95d549644ae0c48a42d69ea917bab79704a0f8efcda5b17f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-27T00:14:43.759 INFO:journalctl@ceph.osd.1.smithi080.stdout:Apr 27 00:14:43 smithi080 podman[68206]: 2024-04-27 00:14:43.407782439 +0000 UTC m=+0.695150701 container died c977ab938f1684b95d549644ae0c48a42d69ea917bab79704a0f8efcda5b17f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-27T00:14:44.088 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.1.service' 2024-04-27T00:14:44.127 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:14:44.128 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-27T00:14:44.128 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-27T00:14:44.128 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.2 2024-04-27T00:14:44.450 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:44 smithi103 systemd[1]: Stopping Ceph osd.2 for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:14:44.846 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:44 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2[38087]: 2024-04-27T00:14:44.450+0000 7f58f1670700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:14:44.846 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:44 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2[38087]: 2024-04-27T00:14:44.450+0000 7f58f1670700 -1 osd.2 98 *** Got signal Terminated *** 2024-04-27T00:14:44.847 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:44 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2[38087]: 2024-04-27T00:14:44.450+0000 7f58f1670700 -1 osd.2 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:14:50.059 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:49 smithi103 podman[69367]: 2024-04-27 00:14:49.770843505 +0000 UTC m=+5.446064028 container died 6740ea3c416425236f7ee08ba0c2378409a753fcb17f0bb8b950068c6083746e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=) 2024-04-27T00:14:50.346 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:50 smithi103 podman[69367]: 2024-04-27 00:14:50.15890585 +0000 UTC m=+5.834126372 container cleanup 6740ea3c416425236f7ee08ba0c2378409a753fcb17f0bb8b950068c6083746e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:14:50.347 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:50 smithi103 bash[69367]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2 2024-04-27T00:14:50.772 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:50 smithi103 podman[69384]: 2024-04-27 00:14:50.419274837 +0000 UTC m=+0.643831058 container remove 6740ea3c416425236f7ee08ba0c2378409a753fcb17f0bb8b950068c6083746e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-27T00:14:51.039 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:50 smithi103 podman[69475]: 2024-04-27 00:14:50.675594595 +0000 UTC m=+0.017868499 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:14:51.040 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:50 smithi103 podman[69475]: 2024-04-27 00:14:50.778559312 +0000 UTC m=+0.120833216 container create e55dc7bef58674ed14a1f095718e51a9fcd102485c57acd19b3d50218d5cb4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0) 2024-04-27T00:14:51.306 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:51 smithi103 podman[69475]: 2024-04-27 00:14:51.040339195 +0000 UTC m=+0.382613104 container init e55dc7bef58674ed14a1f095718e51a9fcd102485c57acd19b3d50218d5cb4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-27T00:14:51.307 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:51 smithi103 podman[69475]: 2024-04-27 00:14:51.044387236 +0000 UTC m=+0.386661141 container start e55dc7bef58674ed14a1f095718e51a9fcd102485c57acd19b3d50218d5cb4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-27T00:14:51.307 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:51 smithi103 podman[69475]: 2024-04-27 00:14:51.139996349 +0000 UTC m=+0.482270282 container attach e55dc7bef58674ed14a1f095718e51a9fcd102485c57acd19b3d50218d5cb4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-27T00:14:51.596 INFO:journalctl@ceph.osd.2.smithi103.stdout:Apr 27 00:14:51 smithi103 podman[69475]: 2024-04-27 00:14:51.351563898 +0000 UTC m=+0.693837805 container died e55dc7bef58674ed14a1f095718e51a9fcd102485c57acd19b3d50218d5cb4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:14:51.905 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.2.service' 2024-04-27T00:14:51.942 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:14:51.943 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-27T00:14:51.943 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-27T00:14:51.943 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.3 2024-04-27T00:14:52.436 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:52 smithi040 systemd[1]: Stopping Ceph osd.3 for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:14:52.716 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:52 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3[47395]: 2024-04-27T00:14:52.683+0000 7fc609a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:14:52.716 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:52 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3[47395]: 2024-04-27T00:14:52.683+0000 7fc609a00700 -1 osd.3 98 *** Got signal Terminated *** 2024-04-27T00:14:52.716 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:52 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3[47395]: 2024-04-27T00:14:52.683+0000 7fc609a00700 -1 osd.3 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:14:54.162 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:53 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3[47395]: 2024-04-27T00:14:53.749+0000 7fc600200700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:29.695847+0000 front 2024-04-27T00:14:29.695002+0000 (oldest deadline 2024-04-27T00:14:53.696230+0000) 2024-04-27T00:14:54.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:14:53 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:14:53.906+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:14:54.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:14:54 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:14:54.187+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:14:54.788 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:14:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:14:54.517+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:14:55.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:14:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:14:54.788+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:14:55.104 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:54 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3[47395]: 2024-04-27T00:14:54.718+0000 7fc600200700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:29.695847+0000 front 2024-04-27T00:14:29.695002+0000 (oldest deadline 2024-04-27T00:14:53.696230+0000) 2024-04-27T00:14:55.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:14:54 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:14:54.899+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:14:55.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:14:55 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:14:55.183+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:14:55.824 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:14:55 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:14:55.470+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:14:55.965 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:55 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3[47395]: 2024-04-27T00:14:55.676+0000 7fc600200700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:29.695847+0000 front 2024-04-27T00:14:29.695002+0000 (oldest deadline 2024-04-27T00:14:53.696230+0000) 2024-04-27T00:14:56.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:14:55 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:14:55.823+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:14:56.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:14:55 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:14:55.870+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:14:56.627 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:14:56 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:14:56.220+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:14:56.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:14:56 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:14:56.433+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:14:56.965 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:56 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3[47395]: 2024-04-27T00:14:56.627+0000 7fc600200700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:29.695847+0000 front 2024-04-27T00:14:29.695002+0000 (oldest deadline 2024-04-27T00:14:53.696230+0000) 2024-04-27T00:14:57.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:14:56 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:14:56.897+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:14:57.346 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:14:56 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:14:56.858+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:14:57.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:14:57 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:14:57.188+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:14:57.814 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:14:57 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:14:57.460+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:14:57.965 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:57 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3[47395]: 2024-04-27T00:14:57.602+0000 7fc600200700 -1 osd.3 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:29.695847+0000 front 2024-04-27T00:14:29.695002+0000 (oldest deadline 2024-04-27T00:14:53.696230+0000) 2024-04-27T00:14:58.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:14:57 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:14:57.814+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:14:58.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:14:57 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:14:57.887+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:14:58.406 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:14:58 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:14:58.206+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:14:58.407 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:58 smithi040 podman[86991]: 2024-04-27 00:14:58.102882245 +0000 UTC m=+5.674924293 container died 1072722a8a0ee996f511b64c8fce8555794bfdfa91a04412f05819c50493711f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-27T00:14:58.715 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:58 smithi040 podman[86991]: 2024-04-27 00:14:58.535242709 +0000 UTC m=+6.107284791 container cleanup 1072722a8a0ee996f511b64c8fce8555794bfdfa91a04412f05819c50493711f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-27T00:14:58.716 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:58 smithi040 bash[86991]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3 2024-04-27T00:14:58.822 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:14:58 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:14:58.480+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:14:59.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:14:58 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:14:58.822+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:14:59.215 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:58 smithi040 podman[87021]: 2024-04-27 00:14:58.795116116 +0000 UTC m=+0.683933520 container remove 1072722a8a0ee996f511b64c8fce8555794bfdfa91a04412f05819c50493711f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:14:59.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:14:58 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:14:58.842+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:14:59.617 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:14:59 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:14:59.231+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:14:59.773 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:14:59 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:14:59.476+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:14:59.928 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:59 smithi040 podman[87113]: 2024-04-27 00:14:59.525629967 +0000 UTC m=+0.044396266 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:14:59.928 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:59 smithi040 podman[87113]: 2024-04-27 00:14:59.646243434 +0000 UTC m=+0.165009771 container create 85afd454a08f2ae3a7fe2921ed30bbb4bab7fa84aa71fa9f8607bb11be0553f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-04-27T00:15:00.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:14:59 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:14:59.772+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:00.216 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:59 smithi040 podman[87113]: 2024-04-27 00:14:59.926595195 +0000 UTC m=+0.445361562 container init 85afd454a08f2ae3a7fe2921ed30bbb4bab7fa84aa71fa9f8607bb11be0553f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-27T00:15:00.216 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:14:59 smithi040 podman[87113]: 2024-04-27 00:14:59.939061967 +0000 UTC m=+0.457828246 container start 85afd454a08f2ae3a7fe2921ed30bbb4bab7fa84aa71fa9f8607bb11be0553f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True) 2024-04-27T00:15:00.216 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:15:00 smithi040 podman[87113]: 2024-04-27 00:15:00.023687687 +0000 UTC m=+0.542454054 container attach 85afd454a08f2ae3a7fe2921ed30bbb4bab7fa84aa71fa9f8607bb11be0553f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-27T00:15:00.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:14:59 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:14:59.832+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:00.715 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:00 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:00.228+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:00.716 INFO:journalctl@ceph.osd.3.smithi040.stdout:Apr 27 00:15:00 smithi040 podman[87113]: 2024-04-27 00:15:00.408183121 +0000 UTC m=+0.926949438 container died 85afd454a08f2ae3a7fe2921ed30bbb4bab7fa84aa71fa9f8607bb11be0553f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-27T00:15:00.734 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:00 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:00.468+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:01.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:00 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:00.734+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:01.196 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.3.service' 2024-04-27T00:15:01.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:00 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:00.806+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:01.314 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:15:01.314 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-27T00:15:01.314 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-27T00:15:01.314 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.4 2024-04-27T00:15:01.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:01 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:01.264+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:01.621 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:01 smithi080 systemd[1]: Stopping Ceph osd.4 for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:15:01.753 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:01 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:01.448+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:02.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:01 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:01.766+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:02.048 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:01 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4[43898]: 2024-04-27T00:15:01.620+0000 7f5bd8b10700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:15:02.049 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:01 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4[43898]: 2024-04-27T00:15:01.620+0000 7f5bd8b10700 -1 osd.4 98 *** Got signal Terminated *** 2024-04-27T00:15:02.049 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:01 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4[43898]: 2024-04-27T00:15:01.620+0000 7f5bd8b10700 -1 osd.4 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:15:02.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:01 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:01.752+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:02.596 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:02 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:02.460+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:02.715 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:02 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:02.282+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:03.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:02 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:02.721+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:03.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:02 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:02.721+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:03.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:02 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:02.747+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:03.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:03 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:03.099+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:03.549 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:03 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4[43898]: 2024-04-27T00:15:03.280+0000 7f5bd10fc700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:37.895292+0000 front 2024-04-27T00:14:37.895260+0000 (oldest deadline 2024-04-27T00:15:02.595467+0000) 2024-04-27T00:15:03.715 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:03 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:03.235+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:03.715 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:03 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:03.235+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:03.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:03 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:03.479+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:03.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:03 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:03.724+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:03.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:03 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:03.724+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:04.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:03 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:03.738+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:04.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:03 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:03.738+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:04.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:04 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:04.218+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:04.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:04 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:04.218+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:04.466 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:03 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:03.982+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:04.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:04 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:04.147+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:04.549 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:04 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4[43898]: 2024-04-27T00:15:04.271+0000 7f5bd10fc700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:37.895292+0000 front 2024-04-27T00:14:37.895260+0000 (oldest deadline 2024-04-27T00:15:02.595467+0000) 2024-04-27T00:15:04.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:04 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:04.742+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:04.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:04 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:04.742+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:04.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:04 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:04.526+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:04.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:04 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:04.526+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:05.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:04 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:04.765+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:05.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:04 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:04.765+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:05.465 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:05 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:05.014+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:05.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:05 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:05.176+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:05.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:05 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:05.176+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:05.536 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:05 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:05.152+0000 7f4ad13f6700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.798447+0000 front 2024-04-27T00:14:39.798514+0000 (oldest deadline 2024-04-27T00:15:05.098470+0000) 2024-04-27T00:15:05.548 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:05 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4[43898]: 2024-04-27T00:15:05.311+0000 7f5bd10fc700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:37.895292+0000 front 2024-04-27T00:14:37.895260+0000 (oldest deadline 2024-04-27T00:15:02.595467+0000) 2024-04-27T00:15:05.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:05 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:05.137+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:05.846 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:05 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:05.753+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:05.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:05 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:05.753+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:05.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:05 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:05.536+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:05.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:05 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:05.536+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:06.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:05 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:05.778+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:06.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:05 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:05.778+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:06.465 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:05 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:05.977+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:06.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:06 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:06.159+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:06.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:06 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:06.159+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:06.529 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:06 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:06.130+0000 7f4ad13f6700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.798447+0000 front 2024-04-27T00:14:39.798514+0000 (oldest deadline 2024-04-27T00:15:05.098470+0000) 2024-04-27T00:15:06.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:06 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:06.108+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:06.597 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:06 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4[43898]: 2024-04-27T00:15:06.346+0000 7f5bd10fc700 -1 osd.4 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:37.895292+0000 front 2024-04-27T00:14:37.895260+0000 (oldest deadline 2024-04-27T00:15:02.595467+0000) 2024-04-27T00:15:06.846 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:06 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:06.736+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:06.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:06 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:06.736+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:06.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:06 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:06.529+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:06.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:06 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:06.529+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:07.012 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:06 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:06.743+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:07.012 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:06 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:06.743+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:07.215 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:06 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:06.963+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:07.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:07 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:07.154+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:07.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:07 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:07.154+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:07.298 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:07 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:07.149+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:07.299 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:07 smithi080 podman[68312]: 2024-04-27 00:15:07.01342173 +0000 UTC m=+5.505875069 container died db0fa9c13963979cc650f026d8e0ff7769ef96f1626ac673c750972ec48830e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-27T00:15:07.567 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:07 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:07.121+0000 7f4ad13f6700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.798447+0000 front 2024-04-27T00:14:39.798514+0000 (oldest deadline 2024-04-27T00:15:05.098470+0000) 2024-04-27T00:15:07.699 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:07 smithi080 podman[68312]: 2024-04-27 00:15:07.422634452 +0000 UTC m=+5.915087790 container cleanup db0fa9c13963979cc650f026d8e0ff7769ef96f1626ac673c750972ec48830e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-04-27T00:15:07.699 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:07 smithi080 bash[68312]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4 2024-04-27T00:15:07.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:07 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:07.567+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:07.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:07 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:07.567+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:07.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:07 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:07.715+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:07.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:07 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:07.715+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:08.031 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:07 smithi080 podman[68329]: 2024-04-27 00:15:07.699742144 +0000 UTC m=+0.680263412 container remove db0fa9c13963979cc650f026d8e0ff7769ef96f1626ac673c750972ec48830e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-27T00:15:08.031 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:07 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:07.788+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:08.032 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:07 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:07.788+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:08.215 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:08 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:08.145+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:08.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:08 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:08.145+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:08.216 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:07 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:07.922+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:08.299 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:08 smithi080 podman[68423]: 2024-04-27 00:15:07.935029551 +0000 UTC m=+0.017890266 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:15:08.299 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:08 smithi080 podman[68423]: 2024-04-27 00:15:08.050818121 +0000 UTC m=+0.133678822 container create 9c914aa6cb4c8356724ff7f140ea7170d968337565b26bd1cca305f414fdad3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:15:08.299 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:08 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:08.181+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:08.521 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:08 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:08.123+0000 7f4ad13f6700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.798447+0000 front 2024-04-27T00:14:39.798514+0000 (oldest deadline 2024-04-27T00:15:05.098470+0000) 2024-04-27T00:15:08.632 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:08 smithi080 podman[68423]: 2024-04-27 00:15:08.333999056 +0000 UTC m=+0.416859768 container init 9c914aa6cb4c8356724ff7f140ea7170d968337565b26bd1cca305f414fdad3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-27T00:15:08.633 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:08 smithi080 podman[68423]: 2024-04-27 00:15:08.337268591 +0000 UTC m=+0.420129301 container start 9c914aa6cb4c8356724ff7f140ea7170d968337565b26bd1cca305f414fdad3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-27T00:15:08.633 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:08 smithi080 podman[68423]: 2024-04-27 00:15:08.420364241 +0000 UTC m=+0.503224973 container attach 9c914aa6cb4c8356724ff7f140ea7170d968337565b26bd1cca305f414fdad3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-27T00:15:08.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:08 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:08.521+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:08.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:08 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:08.521+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:08.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:08 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:08.733+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:08.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:08 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:08.733+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:08.922 INFO:journalctl@ceph.osd.4.smithi080.stdout:Apr 27 00:15:08 smithi080 podman[68423]: 2024-04-27 00:15:08.633574336 +0000 UTC m=+0.716435046 container died 9c914aa6cb4c8356724ff7f140ea7170d968337565b26bd1cca305f414fdad3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-27T00:15:08.923 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:08 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:08.811+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:08.923 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:08 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:08.811+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:09.215 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:08 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:08.944+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:09.217 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:09 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:09.156+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:09.217 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:09 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:09.156+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:09.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.4.service' 2024-04-27T00:15:09.286 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:15:09.286 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-27T00:15:09.286 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-27T00:15:09.286 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.5 2024-04-27T00:15:09.346 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:09 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:09.089+0000 7f4ad13f6700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.798447+0000 front 2024-04-27T00:14:39.798514+0000 (oldest deadline 2024-04-27T00:15:05.098470+0000) 2024-04-27T00:15:09.347 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:09 smithi103 systemd[1]: Stopping Ceph osd.5 for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:15:09.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:09 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:09.185+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:09.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:09 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:09.735+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:09.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:09 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:09.735+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:09.848 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:09 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:09.568+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:09.848 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:09 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:09.568+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:09.848 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:09 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:09.589+0000 7f4ad4e02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:15:09.848 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:09 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:09.589+0000 7f4ad4e02700 -1 osd.5 98 *** Got signal Terminated *** 2024-04-27T00:15:09.849 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:09 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:09.589+0000 7f4ad4e02700 -1 osd.5 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:15:10.182 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:09 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:09.829+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:10.183 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:09 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:09.829+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:10.192 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:09 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:09.921+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:10.346 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:10 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:10.088+0000 7f4ad13f6700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.798447+0000 front 2024-04-27T00:14:39.798514+0000 (oldest deadline 2024-04-27T00:15:05.098470+0000) 2024-04-27T00:15:10.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:10 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:10.192+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:10.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:10 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:10.192+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:10.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:10 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:10.183+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:10.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:10 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:10.519+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:10.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:10 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:10.519+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:10.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:10 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:10.723+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:10.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:10 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:10.723+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:10.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:10 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:10.723+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:11.193 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:10 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:10.801+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:11.193 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:10 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:10.801+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:11.215 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:10 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:10.896+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:11.215 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:10 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:10.896+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.572035+0000 front 2024-04-27T00:14:46.571932+0000 (oldest deadline 2024-04-27T00:15:10.573048+0000) 2024-04-27T00:15:11.346 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:11 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:11.057+0000 7f4ad13f6700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.798447+0000 front 2024-04-27T00:14:39.798514+0000 (oldest deadline 2024-04-27T00:15:05.098470+0000) 2024-04-27T00:15:11.458 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:11 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:11.193+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:11.715 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:11 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:11.217+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:11.716 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:11 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:11.217+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:11.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:11 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:11.550+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:11.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:11 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:11.550+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:11.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:11 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:11.707+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:11.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:11 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:11.707+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:11.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:11 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:11.707+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:12.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:11 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:11.757+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:12.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:11 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:11.757+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:12.201 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:11 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:11.878+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:12.201 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:11 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:11.878+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.572035+0000 front 2024-04-27T00:14:46.571932+0000 (oldest deadline 2024-04-27T00:15:10.573048+0000) 2024-04-27T00:15:12.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:12 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:12.201+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:12.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:12 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:12.201+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:12.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:12 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:12.201+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:12.516 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:12.106+0000 7f4ad13f6700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.798447+0000 front 2024-04-27T00:14:39.798514+0000 (oldest deadline 2024-04-27T00:15:05.098470+0000) 2024-04-27T00:15:12.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:12.174+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:12.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:12.738+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:12.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:12.738+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:12.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:12.738+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:12.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:12.516+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:12.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:12 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:12.516+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:13.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:12.796+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:13.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:12 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:12.796+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:13.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:12 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:12.923+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:13.181 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:12 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:12.923+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.572035+0000 front 2024-04-27T00:14:46.571932+0000 (oldest deadline 2024-04-27T00:15:10.573048+0000) 2024-04-27T00:15:13.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:13 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:13.180+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:13.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:13 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:13.180+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:13.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:13 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:13.180+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:13.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:13 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:13.141+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:13.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:13 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:13.141+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:13.549 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:13 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:13.124+0000 7f4ad13f6700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.798447+0000 front 2024-04-27T00:14:39.798514+0000 (oldest deadline 2024-04-27T00:15:05.098470+0000) 2024-04-27T00:15:13.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:13 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:13.549+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:13.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:13 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:13.549+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:13.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:13 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:13.754+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:13.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:13 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:13.754+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:13.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:13 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:13.754+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:14.168 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:13 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:13.803+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:14.168 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:13 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:13.803+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:14.215 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:14 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:14.171+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:14.215 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:14 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:14.171+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:14.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:14 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:14.171+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:14.216 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:13 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:13.935+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:14.216 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:13 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:13.935+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.572035+0000 front 2024-04-27T00:14:46.571932+0000 (oldest deadline 2024-04-27T00:15:10.573048+0000) 2024-04-27T00:15:14.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:14 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:14.168+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:14.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:14 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:14.168+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:14.592 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:14 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:14.157+0000 7f4ad13f6700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.798447+0000 front 2024-04-27T00:14:39.798514+0000 (oldest deadline 2024-04-27T00:15:05.098470+0000) 2024-04-27T00:15:14.592 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:14 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5[44880]: 2024-04-27T00:15:14.157+0000 7f4ad13f6700 -1 osd.5 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.500064+0000 front 2024-04-27T00:14:48.500013+0000 (oldest deadline 2024-04-27T00:15:13.200057+0000) 2024-04-27T00:15:14.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:14 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:14.592+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:14.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:14 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:14.592+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:14.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:14 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:14.592+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:14.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:14 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:14.726+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:14.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:14 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:14.726+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:14.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:14 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:14.726+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:15.182 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:14 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:14.848+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:15.182 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:14 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:14.848+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:15.215 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:14 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:14.946+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:15.216 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:14 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:14.946+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.572035+0000 front 2024-04-27T00:14:46.571932+0000 (oldest deadline 2024-04-27T00:15:10.573048+0000) 2024-04-27T00:15:15.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:15.143+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:15.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:15.143+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:15.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:15.143+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:15.399 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:15 smithi103 podman[69593]: 2024-04-27 00:15:15.107252107 +0000 UTC m=+5.629174705 container died 3006c0859c1d66da4884dc63446786a9918d7120f5e6ea9785e66d0c423f39ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-27T00:15:15.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:15 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:15.182+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:15.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:15 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:15.182+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:15.666 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:15 smithi103 podman[69593]: 2024-04-27 00:15:15.481989534 +0000 UTC m=+6.003912121 container cleanup 3006c0859c1d66da4884dc63446786a9918d7120f5e6ea9785e66d0c423f39ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:15:15.666 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:15 smithi103 bash[69593]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5 2024-04-27T00:15:15.667 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:15 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:15.550+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:15.667 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:15 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:15.550+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:15.667 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:15 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:15.550+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:15.975 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:15 smithi103 podman[69610]: 2024-04-27 00:15:15.667122212 +0000 UTC m=+0.553807003 container remove 3006c0859c1d66da4884dc63446786a9918d7120f5e6ea9785e66d0c423f39ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-27T00:15:15.975 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:15 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:15.681+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:15.976 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:15 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:15.681+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:15.976 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:15 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:15.681+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:16.145 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:15 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:15.874+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:16.145 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:15 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:15.874+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:16.145 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:15 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:15.874+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:16.303 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:15 smithi103 podman[69701]: 2024-04-27 00:15:15.976345181 +0000 UTC m=+0.102799261 container create 347a102dd3bffd71cb4da1d958cc29a913de524d4bdfcfdfea546da8e8573f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:15:16.303 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:15 smithi103 podman[69701]: 2024-04-27 00:15:15.891726097 +0000 UTC m=+0.018180175 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:15:16.304 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:16 smithi103 podman[69701]: 2024-04-27 00:15:16.212360106 +0000 UTC m=+0.338814184 container init 347a102dd3bffd71cb4da1d958cc29a913de524d4bdfcfdfea546da8e8573f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-27T00:15:16.304 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:16 smithi103 podman[69701]: 2024-04-27 00:15:16.215378908 +0000 UTC m=+0.341832999 container start 347a102dd3bffd71cb4da1d958cc29a913de524d4bdfcfdfea546da8e8573f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-27T00:15:16.304 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:16 smithi103 podman[69701]: 2024-04-27 00:15:16.304151657 +0000 UTC m=+0.430605770 container attach 347a102dd3bffd71cb4da1d958cc29a913de524d4bdfcfdfea546da8e8573f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-27T00:15:16.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:16 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:16.103+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:16.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:16 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:16.103+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:16.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:16 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:16.103+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:16.466 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:15.974+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:16.466 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:15 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:15.974+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.572035+0000 front 2024-04-27T00:14:46.571932+0000 (oldest deadline 2024-04-27T00:15:10.573048+0000) 2024-04-27T00:15:16.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:16 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:16.145+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:16.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:16 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:16.145+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:16.596 INFO:journalctl@ceph.osd.5.smithi103.stdout:Apr 27 00:15:16 smithi103 podman[69701]: 2024-04-27 00:15:16.504638568 +0000 UTC m=+0.631092646 container died 347a102dd3bffd71cb4da1d958cc29a913de524d4bdfcfdfea546da8e8573f4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-27T00:15:16.597 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:16 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:16.543+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:16.597 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:16 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:16.543+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:16.597 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:16 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:16.543+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:17.074 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:16 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:16.680+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:17.074 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:16 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:16.680+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:17.074 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:16 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:16.680+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:17.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.5.service' 2024-04-27T00:15:17.125 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:16 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:16.836+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:17.125 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:16 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:16.836+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:17.125 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:16 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:16.836+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:17.135 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:15:17.135 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-27T00:15:17.135 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-27T00:15:17.135 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.6 2024-04-27T00:15:17.215 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:16 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:16.945+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:17.216 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:16 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:16.945+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.572035+0000 front 2024-04-27T00:14:46.571932+0000 (oldest deadline 2024-04-27T00:15:10.573048+0000) 2024-04-27T00:15:17.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:17 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:17.079+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:17.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:17 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:17.079+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:17.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:17 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:17.079+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:17.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:17 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:17.125+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:17.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:17 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:17.125+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:17.715 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:17 smithi040 systemd[1]: Stopping Ceph osd.6 for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:15:17.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:17 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:17.496+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:17.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:17 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:17.496+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:17.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:17 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:17.496+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:17.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:17 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:17.694+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:17.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:17 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:17.694+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:17.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:17 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:17.694+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:18.096 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:17 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:17.736+0000 7f87db200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:15:18.096 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:17 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:17.736+0000 7f87db200700 -1 osd.6 98 *** Got signal Terminated *** 2024-04-27T00:15:18.096 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:17 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:17.736+0000 7f87db200700 -1 osd.6 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:15:18.096 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:17 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:17.907+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:18.096 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:17 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:17.907+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.572035+0000 front 2024-04-27T00:14:46.571932+0000 (oldest deadline 2024-04-27T00:15:10.573048+0000) 2024-04-27T00:15:18.298 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:18 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:18.084+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:18.299 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:18 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:18.084+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:18.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:17 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:17.878+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:18.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:17 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:17.878+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:18.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:17 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:17.878+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:18.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:18 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:18.095+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:18.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:18 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:18.095+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:18.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:18 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:18.095+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:18.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:18 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:18.512+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:18.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:18 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:18.512+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:18.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:18 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:18.512+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:18.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:18 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:18.733+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:18.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:18 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:18.733+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:18.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:18 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:18.733+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:19.216 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:18 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:18.898+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:19.216 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:18 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:18.898+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.572035+0000 front 2024-04-27T00:14:46.571932+0000 (oldest deadline 2024-04-27T00:15:10.573048+0000) 2024-04-27T00:15:19.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:19 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:19.127+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:19.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:19 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:19.127+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:19.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:19 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:19.127+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:19.298 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:19 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:19.082+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:19.299 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:19 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:19.082+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:19.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:18 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:18.841+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:19.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:18 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:18.841+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:19.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:18 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:18.841+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:19.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:18 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:18.841+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:19.770 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:19 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:19.475+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:19.770 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:19 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:19.475+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:19.770 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:19 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:19.475+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:19.770 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:19 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:19.475+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:20.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:19 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:19.770+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:20.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:19 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:19.770+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:20.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:19 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:19.770+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:20.162 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:19 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:19.868+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:20.162 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:19 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:19.868+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.572035+0000 front 2024-04-27T00:14:46.571932+0000 (oldest deadline 2024-04-27T00:15:10.573048+0000) 2024-04-27T00:15:20.298 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:20 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:20.104+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:20.299 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:20 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:20.104+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:20.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:19 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:19.858+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:20.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:19 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:19.858+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:20.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:19 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:19.858+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:20.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:19 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:19.858+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:20.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:20 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:20.161+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:20.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:20 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:20.161+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:20.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:20 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:20.161+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:20.846 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:20 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:20.731+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:20.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:20 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:20.731+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:20.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:20 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:20.731+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:20.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:20 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:20.518+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:20.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:20 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:20.518+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:20.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:20 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:20.518+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:20.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:20 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:20.518+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:21.132 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:20 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:20.826+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:21.132 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:20 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:20.826+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:21.132 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:20 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:20.826+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:21.132 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:20 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:20.826+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:21.215 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:20 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:20.909+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:21.216 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:20 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:20.909+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.572035+0000 front 2024-04-27T00:14:46.571932+0000 (oldest deadline 2024-04-27T00:15:10.573048+0000) 2024-04-27T00:15:21.216 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:20 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:20.909+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:57.577381+0000 front 2024-04-27T00:14:57.577418+0000 (oldest deadline 2024-04-27T00:15:19.877422+0000) 2024-04-27T00:15:21.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:21 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:21.120+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:21.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:21 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:21.120+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:21.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:21 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:21.120+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:21.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:21 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:21.132+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:21.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:21 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:21.132+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:21.846 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:21 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:21.691+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:21.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:21 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:21.691+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:21.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:21 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:21.691+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:21.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:21 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:21.532+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:21.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:21 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:21.532+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:21.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:21 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:21.532+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:21.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:21 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:21.532+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:22.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:21 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:21.798+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:22.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:21 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:21.798+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:22.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:21 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:21.798+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:22.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:21 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:21.798+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:22.119 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:21 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:21.868+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.969425+0000 front 2024-04-27T00:14:38.969491+0000 (oldest deadline 2024-04-27T00:15:03.669663+0000) 2024-04-27T00:15:22.119 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:21 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:21.868+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.572035+0000 front 2024-04-27T00:14:46.571932+0000 (oldest deadline 2024-04-27T00:15:10.573048+0000) 2024-04-27T00:15:22.119 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:21 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6[54259]: 2024-04-27T00:15:21.868+0000 7f87d1a00700 -1 osd.6 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:57.577381+0000 front 2024-04-27T00:14:57.577418+0000 (oldest deadline 2024-04-27T00:15:19.877422+0000) 2024-04-27T00:15:22.452 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:22 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:22.084+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:22.452 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:22 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:22.084+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:22.452 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:22 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:22.084+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.785198+0000 front 2024-04-27T00:14:56.785417+0000 (oldest deadline 2024-04-27T00:15:21.485060+0000) 2024-04-27T00:15:22.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:22 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:22.136+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:22.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:22 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:22.136+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:22.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:22 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:22.136+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:22.846 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:22 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:22.701+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:22.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:22 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:22.701+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:22.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:22 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:22.701+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:22.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:22 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:22.543+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:22.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:22 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:22.543+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:22.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:22 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:22.543+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:22.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:22 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:22.543+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:23.038 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:22 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:22.758+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:23.038 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:22 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:22.758+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:23.038 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:22 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:22.758+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:23.038 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:22 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:22.758+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:23.039 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:23 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:23.038+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:23.039 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:23 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:23.038+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:23.465 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:23 smithi040 podman[87220]: 2024-04-27 00:15:23.270229667 +0000 UTC m=+5.692856927 container died 22d62700457930419838dce3be1bdae8f20785201e97e7afb277047084f75001 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:15:23.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:23 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:23.161+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:23.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:23 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:23.161+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:23.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:23 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:23.161+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:23.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:23 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:23.161+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:23.838 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:23 smithi040 podman[87220]: 2024-04-27 00:15:23.730025245 +0000 UTC m=+6.152652562 container cleanup 22d62700457930419838dce3be1bdae8f20785201e97e7afb277047084f75001 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-27T00:15:23.838 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:23 smithi040 bash[87220]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6 2024-04-27T00:15:23.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:23 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:23.533+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:23.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:23 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:23.533+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:23.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:23 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:23.533+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:23.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:23 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:23.533+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:23.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:23 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:23.746+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:23.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:23 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:23.746+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:23.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:23 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:23.746+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:24.048 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:23 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:23.038+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.785198+0000 front 2024-04-27T00:14:56.785417+0000 (oldest deadline 2024-04-27T00:15:21.485060+0000) 2024-04-27T00:15:24.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:23 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:23.797+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:24.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:23 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:23.797+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:24.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:23 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:23.797+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:24.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:23 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:23.797+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:24.172 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:23 smithi040 podman[87241]: 2024-04-27 00:15:23.948030332 +0000 UTC m=+0.668227084 container remove 22d62700457930419838dce3be1bdae8f20785201e97e7afb277047084f75001 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-27T00:15:24.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:24 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:24.171+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:24.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:24 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:24.171+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:24.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:24 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:24.171+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:24.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:24 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:24.171+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:24.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:24 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:24.076+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:24.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:24 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:24.076+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:24.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:24 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:24.076+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.785198+0000 front 2024-04-27T00:14:56.785417+0000 (oldest deadline 2024-04-27T00:15:21.485060+0000) 2024-04-27T00:15:24.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:24 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:24.759+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:24.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:24 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:24.759+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:24.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:24 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:24.759+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:24.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:24 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:24.539+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:24.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:24 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:24.539+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:24.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:24 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:24.539+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:24.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:24 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:24.539+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:25.153 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:24 smithi040 podman[87334]: 2024-04-27 00:15:24.70122364 +0000 UTC m=+0.042817381 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:15:25.153 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:24 smithi040 podman[87334]: 2024-04-27 00:15:24.849036246 +0000 UTC m=+0.190630022 container create 4d5b8792d0a40c6c62b506ea8ae5b002e64cec061c02d5658b7970dc033cc6a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:15:25.153 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:25 smithi040 podman[87334]: 2024-04-27 00:15:25.152673746 +0000 UTC m=+0.494267526 container init 4d5b8792d0a40c6c62b506ea8ae5b002e64cec061c02d5658b7970dc033cc6a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-27T00:15:25.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:24 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:24.806+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:25.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:24 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:24.806+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:25.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:24 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:24.806+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:25.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:24 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:24.806+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:25.299 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:25 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:25.030+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:25.299 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:25 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:25.030+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:25.299 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:25 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:25.030+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.785198+0000 front 2024-04-27T00:14:56.785417+0000 (oldest deadline 2024-04-27T00:15:21.485060+0000) 2024-04-27T00:15:25.465 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:25 smithi040 podman[87334]: 2024-04-27 00:15:25.165912611 +0000 UTC m=+0.507506418 container start 4d5b8792d0a40c6c62b506ea8ae5b002e64cec061c02d5658b7970dc033cc6a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-27T00:15:25.465 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:25 smithi040 podman[87334]: 2024-04-27 00:15:25.260060188 +0000 UTC m=+0.601654004 container attach 4d5b8792d0a40c6c62b506ea8ae5b002e64cec061c02d5658b7970dc033cc6a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-27T00:15:25.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:25 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:25.159+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:25.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:25 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:25.159+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:25.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:25 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:25.159+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:25.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:25 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:25.159+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:25.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:25 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:25.741+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:25.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:25 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:25.741+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:25.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:25 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:25.741+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:25.848 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:25 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:25.547+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:25.848 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:25 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:25.547+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:25.848 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:25 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:25.547+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:25.848 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:25 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:25.547+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:25.966 INFO:journalctl@ceph.osd.6.smithi040.stdout:Apr 27 00:15:25 smithi040 podman[87334]: 2024-04-27 00:15:25.634928951 +0000 UTC m=+0.976522731 container died 4d5b8792d0a40c6c62b506ea8ae5b002e64cec061c02d5658b7970dc033cc6a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:15:26.049 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:25 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:25.986+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:26.049 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:25 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:25.986+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:26.049 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:25 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:25.986+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.785198+0000 front 2024-04-27T00:14:56.785417+0000 (oldest deadline 2024-04-27T00:15:21.485060+0000) 2024-04-27T00:15:26.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:25 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:25.768+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:26.050 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:25 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:25.768+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:26.050 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:25 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:25.768+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:26.050 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:25 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:25.768+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:26.414 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:26 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:26.121+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:26.414 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:26 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:26.121+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:26.414 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:26 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:26.121+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:26.414 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:26 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:26.121+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:26.459 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.6.service' 2024-04-27T00:15:26.574 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:15:26.575 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-27T00:15:26.575 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-27T00:15:26.575 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.7 2024-04-27T00:15:26.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:26 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:26.766+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:26.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:26 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:26.766+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:26.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:26 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:26.766+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:26.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:26 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:26.549+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:26.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:26 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:26.549+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:26.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:26 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:26.549+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:26.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:26 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:26.549+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:26.918 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:26 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:26.809+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:26.918 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:26 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:26.809+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:26.918 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:26 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:26.809+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:26.918 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:26 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:26.809+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:26.918 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:26 smithi080 systemd[1]: Stopping Ceph osd.7 for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:15:26.918 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:26 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:26.918+0000 7f7bf076d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:15:26.919 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:26 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:26.918+0000 7f7bf076d700 -1 osd.7 98 *** Got signal Terminated *** 2024-04-27T00:15:26.919 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:26 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:26.918+0000 7f7bf076d700 -1 osd.7 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:15:27.298 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:27 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:27.034+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:27.298 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:27 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:27.034+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:27.298 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:27 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:27.034+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.785198+0000 front 2024-04-27T00:14:56.785417+0000 (oldest deadline 2024-04-27T00:15:21.485060+0000) 2024-04-27T00:15:27.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:27 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:27.122+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:27.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:27 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:27.122+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:27.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:27 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:27.122+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:27.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:27 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:27.122+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:27.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:27 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:27.578+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:27.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:27 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:27.578+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:27.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:27 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:27.578+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:27.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:27 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:27.578+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:27.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:27 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:27.744+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:27.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:27 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:27.744+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:27.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:27 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:27.744+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:27.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:27 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:27.744+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:28.048 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:28 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:28.011+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:28.048 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:28 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:28.011+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:28.049 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:28 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:28.011+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.785198+0000 front 2024-04-27T00:14:56.785417+0000 (oldest deadline 2024-04-27T00:15:21.485060+0000) 2024-04-27T00:15:28.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:27 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:27.794+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:28.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:27 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:27.794+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:28.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:27 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:27.794+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:28.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:27 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:27.794+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:28.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:28 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:28.155+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:28.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:28 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:28.155+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:28.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:28 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:28.155+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:28.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:28 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:28.155+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:28.846 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:28 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:28.558+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:28.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:28 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:28.558+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:28.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:28 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:28.558+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:28.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:28 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:28.558+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:28.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:28 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:28.724+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:28.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:28 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:28.724+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:28.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:28 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:28.724+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:28.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:28 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:28.724+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:29.048 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:28 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:28.995+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:29.048 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:28 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:28.995+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:29.049 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:28 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:28.995+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.785198+0000 front 2024-04-27T00:14:56.785417+0000 (oldest deadline 2024-04-27T00:15:21.485060+0000) 2024-04-27T00:15:29.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:28 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:28.782+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:29.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:28 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:28.782+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:29.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:28 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:28.782+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:29.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:28 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:28.782+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:29.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:28 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:28.782+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:29.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:29 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:29.164+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:29.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:29 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:29.164+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:29.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:29 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:29.164+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:29.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:29 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:29.164+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:29.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:29 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:29.164+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:29.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:29 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:29.555+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:29.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:29 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:29.555+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:29.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:29 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:29.555+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:29.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:29 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:29.555+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:29.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:29 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:29.748+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:29.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:29 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:29.748+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:29.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:29 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:29.748+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:29.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:29 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:29.748+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:30.048 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:29 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:29.950+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:30.049 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:29 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:29.950+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:30.049 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:29 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:29.950+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.785198+0000 front 2024-04-27T00:14:56.785417+0000 (oldest deadline 2024-04-27T00:15:21.485060+0000) 2024-04-27T00:15:30.049 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:29 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:29.950+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.185756+0000 front 2024-04-27T00:15:03.185781+0000 (oldest deadline 2024-04-27T00:15:29.085864+0000) 2024-04-27T00:15:30.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:29 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:29.791+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:30.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:29 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:29.791+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:30.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:29 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:29.791+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:30.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:29 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:29.791+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:30.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:29 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:29.791+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:30.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:30 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:30.151+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:30.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:30 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:30.151+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:30.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:30 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:30.151+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:30.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:30 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:30.151+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:30.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:30 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:30.151+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:30.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:30 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:30.592+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:30.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:30 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:30.592+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:30.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:30 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:30.592+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:30.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:30 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:30.592+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:30.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:30 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:30.702+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:30.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:30 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:30.702+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:30.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:30 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:30.702+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:30.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:30 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:30.702+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:31.298 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:30.917+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:31.298 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:30.917+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:31.298 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:30.917+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.785198+0000 front 2024-04-27T00:14:56.785417+0000 (oldest deadline 2024-04-27T00:15:21.485060+0000) 2024-04-27T00:15:31.298 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:30.917+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.185756+0000 front 2024-04-27T00:15:03.185781+0000 (oldest deadline 2024-04-27T00:15:29.085864+0000) 2024-04-27T00:15:31.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:30.835+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:31.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:30.835+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:31.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:30.835+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:31.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:30.835+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:31.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:30.835+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:31.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:31 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:31.106+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:31.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:31 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:31.106+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:31.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:31 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:31.106+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:31.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:31 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:31.106+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:31.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:31 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:31.106+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:32.096 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:31 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:31.637+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:32.096 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:31 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:31.637+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:32.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:31 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:31.637+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:32.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:31 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:31.637+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:32.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:31 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:31.692+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:32.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:31 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:31.692+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:32.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:31 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:31.692+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:32.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:31 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:31.692+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:32.293 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:31 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:31.867+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:40.582187+0000 front 2024-04-27T00:14:40.582220+0000 (oldest deadline 2024-04-27T00:15:02.282329+0000) 2024-04-27T00:15:32.294 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:31 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:31.867+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:46.883739+0000 front 2024-04-27T00:14:46.883823+0000 (oldest deadline 2024-04-27T00:15:12.183725+0000) 2024-04-27T00:15:32.294 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:31 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:31.867+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.785198+0000 front 2024-04-27T00:14:56.785417+0000 (oldest deadline 2024-04-27T00:15:21.485060+0000) 2024-04-27T00:15:32.294 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:31 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7[50889]: 2024-04-27T00:15:31.867+0000 7f7becd61700 -1 osd.7 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.185756+0000 front 2024-04-27T00:15:03.185781+0000 (oldest deadline 2024-04-27T00:15:29.085864+0000) 2024-04-27T00:15:32.294 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:31 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:31.806+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:32.294 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:31 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:31.806+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:32.294 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:31 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:31.806+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:32.294 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:31 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:31.806+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:32.294 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:31 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:31.806+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:32.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:32 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:32.143+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:32.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:32 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:32.143+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:32.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:32 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:32.143+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:32.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:32 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:32.143+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:32.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:32 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:32.143+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:32.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:32 smithi080 podman[68541]: 2024-04-27 00:15:32.293959099 +0000 UTC m=+5.518386963 container died 3e7a6ef652d21beac2f6e25d866a11f3422c5881b4d1af2f4a29b8ee6f35114e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-27T00:15:32.927 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:32 smithi080 podman[68541]: 2024-04-27 00:15:32.677018455 +0000 UTC m=+5.901446329 container cleanup 3e7a6ef652d21beac2f6e25d866a11f3422c5881b4d1af2f4a29b8ee6f35114e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-27T00:15:32.928 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:32 smithi080 bash[68541]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7 2024-04-27T00:15:32.928 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:32 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:32.828+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:32.928 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:32 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:32.828+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:32.928 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:32 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:32.828+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:32.928 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:32 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:32.828+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:32.928 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:32 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:32.828+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:33.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:32 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:32.662+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:33.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:32 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:32.662+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:33.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:32 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:32.662+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:33.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:32 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:32.662+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:33.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:32 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:32.701+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:33.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:32 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:32.701+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:33.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:32 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:32.701+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:33.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:32 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:32.701+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:33.292 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:32 smithi080 podman[68559]: 2024-04-27 00:15:32.928622433 +0000 UTC m=+0.627941412 container remove 3e7a6ef652d21beac2f6e25d866a11f3422c5881b4d1af2f4a29b8ee6f35114e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-27T00:15:33.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:33 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:33.176+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:33.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:33 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:33.176+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:33.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:33 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:33.176+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:33.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:33 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:33.176+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:33.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:33 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:33.176+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:33.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:33 smithi080 podman[68646]: 2024-04-27 00:15:33.195483952 +0000 UTC m=+0.023952001 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:15:33.548 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:33 smithi080 podman[68646]: 2024-04-27 00:15:33.312962301 +0000 UTC m=+0.141430343 container create 8bc0481406ba025f47d02f8cc69e0553c4d8416a5395f18fa3c4ab138a2b0640 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:15:33.816 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:33 smithi080 podman[68646]: 2024-04-27 00:15:33.591892597 +0000 UTC m=+0.420360643 container init 8bc0481406ba025f47d02f8cc69e0553c4d8416a5395f18fa3c4ab138a2b0640 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux ) 2024-04-27T00:15:33.816 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:33 smithi080 podman[68646]: 2024-04-27 00:15:33.595099554 +0000 UTC m=+0.423567599 container start 8bc0481406ba025f47d02f8cc69e0553c4d8416a5395f18fa3c4ab138a2b0640 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-27T00:15:33.816 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:33 smithi080 podman[68646]: 2024-04-27 00:15:33.675060032 +0000 UTC m=+0.503528075 container attach 8bc0481406ba025f47d02f8cc69e0553c4d8416a5395f18fa3c4ab138a2b0640 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-27T00:15:33.998 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:33 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:33.621+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:33.998 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:33 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:33.621+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:33.998 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:33 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:33.621+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:33.998 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:33 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:33.621+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:33.999 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:33 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:33.732+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:33.999 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:33 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:33.732+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:33.999 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:33 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:33.732+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:33.999 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:33 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:33.732+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:34.236 INFO:journalctl@ceph.osd.7.smithi080.stdout:Apr 27 00:15:33 smithi080 podman[68646]: 2024-04-27 00:15:33.889198624 +0000 UTC m=+0.717666659 container died 8bc0481406ba025f47d02f8cc69e0553c4d8416a5395f18fa3c4ab138a2b0640 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T00:15:34.236 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:33 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:33.815+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:34.236 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:33 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:33.815+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:34.236 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:33 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:33.815+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:34.237 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:33 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:33.815+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:34.237 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:33 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:33.815+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:34.556 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.7.service' 2024-04-27T00:15:34.594 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:15:34.594 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-27T00:15:34.594 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-27T00:15:34.594 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.8 2024-04-27T00:15:34.716 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:34 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:34.219+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:34.716 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:34 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:34.219+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:34.716 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:34 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:34.219+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:34.716 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:34 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:34.219+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:34.716 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:34 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:34.219+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:34.912 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:34 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:34.705+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:34.912 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:34 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:34.705+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:34.912 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:34 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:34.705+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:34.912 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:34 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:34.705+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:34.912 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:34 smithi103 systemd[1]: Stopping Ceph osd.8 for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:15:34.912 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:34 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:34.667+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:34.912 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:34 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:34.667+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:34.912 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:34 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:34.667+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:34.912 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:34 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:34.667+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:34.913 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:34 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:34.911+0000 7ff2f2de5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:15:34.913 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:34 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:34.911+0000 7ff2f2de5700 -1 osd.8 98 *** Got signal Terminated *** 2024-04-27T00:15:34.913 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:34 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:34.911+0000 7ff2f2de5700 -1 osd.8 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:15:35.171 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:35 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:35.171+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:35.171 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:35 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:35.171+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:35.171 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:35 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:35.171+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:35.171 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:35 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:35.171+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:35.171 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:35 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:35.171+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:35.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:34 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:34.799+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:35.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:34 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:34.799+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:35.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:34 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:34.799+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:35.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:34 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:34.799+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:35.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:34 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:34.799+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:36.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:35 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:35.757+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:36.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:35 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:35.757+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:36.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:35 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:35.757+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:36.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:35 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:35.757+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:36.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:35 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:35.757+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:36.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:35 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:35.757+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:36.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:35 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:35.647+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:36.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:35 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:35.647+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:36.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:35 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:35.647+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:36.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:35 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:35.647+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:36.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:35 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:35.669+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:36.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:35 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:35.669+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:36.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:35 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:35.669+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:36.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:35 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:35.669+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:36.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:36 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:36.144+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:36.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:36 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:36.144+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:36.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:36 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:36.144+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:36.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:36 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:36.144+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:36.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:36 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:36.144+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:37.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:36 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:36.793+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:37.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:36 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:36.793+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:37.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:36 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:36.793+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:37.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:36 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:36.793+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:37.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:36 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:36.793+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:37.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:36 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:36.793+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:37.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:36 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:36.653+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:37.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:36 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:36.653+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:37.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:36 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:36.653+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:37.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:36 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:36.653+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:37.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:36 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:36.654+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:37.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:36 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:36.654+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:37.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:36 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:36.654+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:37.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:36 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:36.654+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:37.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:36 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:36.654+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:09.952914+0000 front 2024-04-27T00:15:09.952848+0000 (oldest deadline 2024-04-27T00:15:35.753505+0000) 2024-04-27T00:15:37.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:37 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:37.097+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:37.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:37 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:37.097+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:37.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:37 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:37.097+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:37.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:37 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:37.097+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:37.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:37 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:37.097+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:38.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:37 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:37.651+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:38.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:37 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:37.651+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:38.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:37 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:37.651+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:38.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:37 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:37.651+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:38.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:37 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:37.640+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:38.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:37 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:37.640+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:38.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:37 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:37.640+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:38.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:37 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:37.640+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:38.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:37 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:37.640+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:09.952914+0000 front 2024-04-27T00:15:09.952848+0000 (oldest deadline 2024-04-27T00:15:35.753505+0000) 2024-04-27T00:15:38.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:37 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:37.808+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:38.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:37 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:37.808+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:38.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:37 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:37.808+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:38.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:37 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:37.808+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:38.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:37 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:37.808+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:38.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:37 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:37.808+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:38.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:38 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:38.099+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:38.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:38 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:38.099+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:38.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:38 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:38.099+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:38.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:38 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:38.099+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:38.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:38 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:38.099+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:38.846 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:38 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:38.682+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:38.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:38 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:38.682+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:38.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:38 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:38.682+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:38.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:38 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:38.682+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:38.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:38 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:38.591+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:38.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:38 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:38.591+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:38.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:38 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:38.591+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:38.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:38 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:38.591+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:38.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:38 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:38.591+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:09.952914+0000 front 2024-04-27T00:15:09.952848+0000 (oldest deadline 2024-04-27T00:15:35.753505+0000) 2024-04-27T00:15:39.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:38 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:38.773+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:39.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:38 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:38.773+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:39.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:38 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:38.773+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:39.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:38 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:38.773+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:39.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:38 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:38.773+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:39.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:38 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:38.773+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:39.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:39 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:39.138+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:39.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:39 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:39.138+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:39.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:39 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:39.138+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:39.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:39 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:39.138+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:39.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:39 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:39.138+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:39.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:39 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:39.558+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.948188+0000 front 2024-04-27T00:14:31.948699+0000 (oldest deadline 2024-04-27T00:14:53.648056+0000) 2024-04-27T00:15:39.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:39 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:39.558+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.549173+0000 front 2024-04-27T00:14:39.549190+0000 (oldest deadline 2024-04-27T00:15:04.249213+0000) 2024-04-27T00:15:39.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:39 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:39.558+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:48.950024+0000 front 2024-04-27T00:14:48.950096+0000 (oldest deadline 2024-04-27T00:15:13.650230+0000) 2024-04-27T00:15:39.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:39 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:39.558+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.551365+0000 front 2024-04-27T00:14:56.551509+0000 (oldest deadline 2024-04-27T00:15:18.851207+0000) 2024-04-27T00:15:39.847 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:39 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8[51420]: 2024-04-27T00:15:39.558+0000 7ff2eb3d1700 -1 osd.8 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:09.952914+0000 front 2024-04-27T00:15:09.952848+0000 (oldest deadline 2024-04-27T00:15:35.753505+0000) 2024-04-27T00:15:39.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:39 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:39.681+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:39.848 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:39 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:39.681+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:39.848 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:39 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:39.681+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:39.848 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:39 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:39.681+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:39.848 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:39 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:39.681+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:40.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:39 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:39.748+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:40.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:39 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:39.748+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:40.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:39 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:39.748+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:40.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:39 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:39.748+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:40.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:39 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:39.748+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:40.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:39 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:39.748+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:40.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:40 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:40.103+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:40.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:40 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:40.103+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:40.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:40 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:40.103+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:40.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:40 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:40.103+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:40.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:40 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:40.103+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:40.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:40 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:40.103+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.698856+0000 front 2024-04-27T00:15:13.698955+0000 (oldest deadline 2024-04-27T00:15:39.599123+0000) 2024-04-27T00:15:40.597 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:40 smithi103 podman[69815]: 2024-04-27 00:15:40.324987618 +0000 UTC m=+5.531454333 container died 4811ec89670a98f80e0651d84b390a913e30a77c25c6d5e88c4372e81c38e22c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-27T00:15:40.930 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:40 smithi103 podman[69815]: 2024-04-27 00:15:40.741513958 +0000 UTC m=+5.947980671 container cleanup 4811ec89670a98f80e0651d84b390a913e30a77c25c6d5e88c4372e81c38e22c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:15:40.930 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:40 smithi103 bash[69815]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8 2024-04-27T00:15:40.930 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:40 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:40.649+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:40.930 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:40 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:40.649+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:40.930 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:40 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:40.649+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:40.930 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:40 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:40.649+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:40.930 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:40 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:40.649+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:41.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:40 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:40.767+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:41.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:40 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:40.767+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:41.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:40 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:40.767+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:41.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:40 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:40.767+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:41.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:40 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:40.767+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:41.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:40 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:40.767+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:41.314 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:40 smithi103 podman[69832]: 2024-04-27 00:15:40.930691132 +0000 UTC m=+0.602362097 container remove 4811ec89670a98f80e0651d84b390a913e30a77c25c6d5e88c4372e81c38e22c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-27T00:15:41.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:41 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:41.097+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:41.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:41 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:41.097+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:41.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:41 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:41.097+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:41.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:41 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:41.097+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:41.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:41 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:41.097+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:41.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:41 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:41.097+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.698856+0000 front 2024-04-27T00:15:13.698955+0000 (oldest deadline 2024-04-27T00:15:39.599123+0000) 2024-04-27T00:15:41.596 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:41 smithi103 podman[69926]: 2024-04-27 00:15:41.314792213 +0000 UTC m=+0.089358670 container create 552b9878ad8870ae7a085d0807a28cab8fba73ee6a5903f31f940c3aaac5a6f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-27T00:15:41.596 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:41 smithi103 podman[69926]: 2024-04-27 00:15:41.243401505 +0000 UTC m=+0.017967985 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:15:41.596 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:41 smithi103 podman[69926]: 2024-04-27 00:15:41.534340311 +0000 UTC m=+0.308906772 container init 552b9878ad8870ae7a085d0807a28cab8fba73ee6a5903f31f940c3aaac5a6f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-27T00:15:41.597 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:41 smithi103 podman[69926]: 2024-04-27 00:15:41.538183041 +0000 UTC m=+0.312749500 container start 552b9878ad8870ae7a085d0807a28cab8fba73ee6a5903f31f940c3aaac5a6f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-27T00:15:42.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:41 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:41.753+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:42.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:41 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:41.753+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:42.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:41 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:41.753+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:42.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:41 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:41.753+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:42.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:41 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:41.753+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:42.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:41 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:41.753+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:42.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:41 smithi103 podman[69926]: 2024-04-27 00:15:41.634280324 +0000 UTC m=+0.408846802 container attach 552b9878ad8870ae7a085d0807a28cab8fba73ee6a5903f31f940c3aaac5a6f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-27T00:15:42.097 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:41 smithi103 podman[69926]: 2024-04-27 00:15:41.833705112 +0000 UTC m=+0.608271561 container died 552b9878ad8870ae7a085d0807a28cab8fba73ee6a5903f31f940c3aaac5a6f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:15:42.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:41 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:41.693+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:42.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:41 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:41.693+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:42.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:41 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:41.693+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:42.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:41 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:41.693+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:42.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:41 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:41.693+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:42.398 INFO:journalctl@ceph.osd.8.smithi103.stdout:Apr 27 00:15:42 smithi103 podman[69952]: 2024-04-27 00:15:42.380119526 +0000 UTC m=+0.536165084 container remove 552b9878ad8870ae7a085d0807a28cab8fba73ee6a5903f31f940c3aaac5a6f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True) 2024-04-27T00:15:42.404 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.8.service' 2024-04-27T00:15:42.441 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:15:42.442 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-27T00:15:42.442 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-27T00:15:42.442 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.9 2024-04-27T00:15:42.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:42 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:42.140+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:42.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:42 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:42.140+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:42.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:42 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:42.140+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:42.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:42 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:42.140+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:42.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:42 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:42.140+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:42.465 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:42 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:42.140+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.698856+0000 front 2024-04-27T00:15:13.698955+0000 (oldest deadline 2024-04-27T00:15:39.599123+0000) 2024-04-27T00:15:42.965 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:42 smithi040 systemd[1]: Stopping Ceph osd.9 for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:15:43.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:42 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:42.746+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:43.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:42 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:42.746+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:43.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:42 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:42.746+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:43.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:42 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:42.746+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:43.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:42 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:42.746+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:43.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:42 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:42.746+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:43.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:42 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:42.719+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:43.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:42 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:42.719+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:43.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:42 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:42.719+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:43.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:42 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:42.719+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:43.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:42 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:42.719+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:43.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:43 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:43.072+0000 7f6ac4c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:15:43.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:43 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:43.072+0000 7f6ac4c00700 -1 osd.9 98 *** Got signal Terminated *** 2024-04-27T00:15:43.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:43 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:43.072+0000 7f6ac4c00700 -1 osd.9 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:15:43.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:43 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:43.113+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:43.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:43 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:43.113+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:43.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:43 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:43.113+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:43.467 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:43 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:43.113+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:43.467 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:43 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:43.113+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:43.467 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:43 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:43.113+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.698856+0000 front 2024-04-27T00:15:13.698955+0000 (oldest deadline 2024-04-27T00:15:39.599123+0000) 2024-04-27T00:15:44.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:43 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:43.758+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:44.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:43 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:43.758+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:44.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:43 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:43.758+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:44.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:43 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:43.758+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:44.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:43 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:43.758+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:44.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:43 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:43.758+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:44.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:43 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:43.671+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:44.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:43 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:43.671+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:44.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:43 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:43.671+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:44.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:43 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:43.671+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:44.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:43 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:43.671+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:44.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:44 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:44.120+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:44.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:44 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:44.120+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:44.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:44 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:44.120+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:44.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:44 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:44.120+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:44.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:44 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:44.120+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:44.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:44 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:44.120+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.698856+0000 front 2024-04-27T00:15:13.698955+0000 (oldest deadline 2024-04-27T00:15:39.599123+0000) 2024-04-27T00:15:45.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:44 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:44.779+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:45.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:44 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:44.779+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:45.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:44 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:44.779+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:45.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:44 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:44.779+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:45.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:44 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:44.779+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:45.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:44 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:44.779+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:45.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:44 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:44.665+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:45.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:44 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:44.665+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:45.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:44 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:44.665+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:45.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:44 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:44.665+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:45.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:44 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:44.665+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:45.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:44 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:44.665+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:45.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:45 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:45.094+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:45.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:45 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:45.094+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:45.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:45 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:45.094+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:45.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:45 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:45.094+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:45.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:45 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:45.094+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:45.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:45 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:45.094+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.698856+0000 front 2024-04-27T00:15:13.698955+0000 (oldest deadline 2024-04-27T00:15:39.599123+0000) 2024-04-27T00:15:46.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:45 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:45.752+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:46.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:45 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:45.752+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:46.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:45 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:45.752+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:46.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:45 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:45.752+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:46.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:45 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:45.752+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:46.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:45 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:45.752+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:46.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:45 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:45.752+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:22.441309+0000 front 2024-04-27T00:15:22.441138+0000 (oldest deadline 2024-04-27T00:15:45.241359+0000) 2024-04-27T00:15:46.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:45 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:45.702+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:46.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:45 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:45.702+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:46.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:45 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:45.702+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:46.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:45 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:45.702+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:46.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:45 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:45.702+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:46.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:45 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:45.702+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:46.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:46 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:46.088+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:46.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:46 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:46.088+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:46.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:46 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:46.088+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:46.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:46 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:46.088+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:46.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:46 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:46.088+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:46.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:46 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:46.088+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.698856+0000 front 2024-04-27T00:15:13.698955+0000 (oldest deadline 2024-04-27T00:15:39.599123+0000) 2024-04-27T00:15:47.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:46 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:46.735+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:47.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:46 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:46.735+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:47.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:46 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:46.735+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:47.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:46 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:46.735+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:47.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:46 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:46.735+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:47.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:46 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:46.735+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:47.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:46 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:46.735+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:22.441309+0000 front 2024-04-27T00:15:22.441138+0000 (oldest deadline 2024-04-27T00:15:45.241359+0000) 2024-04-27T00:15:47.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:46 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:46.710+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:47.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:46 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:46.710+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:47.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:46 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:46.710+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:47.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:46 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:46.710+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:47.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:46 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:46.710+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:47.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:46 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:46.710+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:47.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:47 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:47.102+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.882543+0000 front 2024-04-27T00:14:31.882451+0000 (oldest deadline 2024-04-27T00:14:54.182580+0000) 2024-04-27T00:15:47.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:47 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:47.102+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.986464+0000 front 2024-04-27T00:14:39.986573+0000 (oldest deadline 2024-04-27T00:15:02.886733+0000) 2024-04-27T00:15:47.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:47 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:47.102+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.289889+0000 front 2024-04-27T00:14:49.289952+0000 (oldest deadline 2024-04-27T00:15:12.189874+0000) 2024-04-27T00:15:47.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:47 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:47.102+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:55.493757+0000 front 2024-04-27T00:14:55.493909+0000 (oldest deadline 2024-04-27T00:15:23.095381+0000) 2024-04-27T00:15:47.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:47 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:47.102+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.096799+0000 front 2024-04-27T00:15:03.096486+0000 (oldest deadline 2024-04-27T00:15:28.996441+0000) 2024-04-27T00:15:47.466 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:47 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9[61004]: 2024-04-27T00:15:47.102+0000 7f6abb400700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.698856+0000 front 2024-04-27T00:15:13.698955+0000 (oldest deadline 2024-04-27T00:15:39.599123+0000) 2024-04-27T00:15:48.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:47 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:47.781+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:48.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:47 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:47.781+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:48.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:47 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:47.781+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:48.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:47 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:47.781+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:48.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:47 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:47.781+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:48.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:47 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:47.781+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:48.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:47 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:47.781+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:22.441309+0000 front 2024-04-27T00:15:22.441138+0000 (oldest deadline 2024-04-27T00:15:45.241359+0000) 2024-04-27T00:15:48.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:47 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:47.704+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:48.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:47 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:47.704+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:48.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:47 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:47.704+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:48.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:47 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:47.704+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:48.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:47 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:47.704+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:48.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:47 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:47.704+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:48.854 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:48 smithi040 podman[87439]: 2024-04-27 00:15:48.563077048 +0000 UTC m=+5.675121812 container died b9ac475e79cfc2cffc3126771148d1305eb1a8092f54ec1f50a2f2c0264ac835 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-27T00:15:49.048 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:48 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:48.775+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:49.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:48 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:48.775+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:49.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:48 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:48.775+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:49.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:48 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:48.775+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:49.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:48 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:48.775+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:49.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:48 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:48.775+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:49.050 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:48 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:48.775+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:22.441309+0000 front 2024-04-27T00:15:22.441138+0000 (oldest deadline 2024-04-27T00:15:45.241359+0000) 2024-04-27T00:15:49.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:48 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:48.725+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:49.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:48 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:48.725+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:49.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:48 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:48.725+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:49.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:48 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:48.725+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:49.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:48 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:48.725+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:49.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:48 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:48.725+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:49.215 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:48 smithi040 podman[87439]: 2024-04-27 00:15:48.973991422 +0000 UTC m=+6.086036182 container cleanup b9ac475e79cfc2cffc3126771148d1305eb1a8092f54ec1f50a2f2c0264ac835 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-27T00:15:49.216 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:48 smithi040 bash[87439]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9 2024-04-27T00:15:49.715 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:49 smithi040 podman[87457]: 2024-04-27 00:15:49.266674717 +0000 UTC m=+0.688722392 container remove b9ac475e79cfc2cffc3126771148d1305eb1a8092f54ec1f50a2f2c0264ac835 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:15:50.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:49 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:49.697+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:50.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:49 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:49.697+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:50.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:49 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:49.697+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:50.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:49 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:49.697+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:50.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:49 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:49.697+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:50.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:49 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:49.697+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:50.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:49 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:49.823+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:50.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:49 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:49.823+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:50.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:49 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:49.823+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:50.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:49 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:49.823+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:50.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:49 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:49.823+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:50.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:49 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:49.823+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:50.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:49 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:49.823+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:22.441309+0000 front 2024-04-27T00:15:22.441138+0000 (oldest deadline 2024-04-27T00:15:45.241359+0000) 2024-04-27T00:15:50.420 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:50 smithi040 podman[87544]: 2024-04-27 00:15:50.059472504 +0000 UTC m=+0.038809204 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:15:50.420 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:50 smithi040 podman[87544]: 2024-04-27 00:15:50.159218291 +0000 UTC m=+0.138555012 container create 2e3967b32f6b598bd39379708dc14649d4995b00a6094a56fa8c13af53d40b9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-27T00:15:50.420 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:50 smithi040 podman[87544]: 2024-04-27 00:15:50.42003177 +0000 UTC m=+0.399368485 container init 2e3967b32f6b598bd39379708dc14649d4995b00a6094a56fa8c13af53d40b9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-27T00:15:50.715 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:50 smithi040 podman[87544]: 2024-04-27 00:15:50.432293882 +0000 UTC m=+0.411630588 container start 2e3967b32f6b598bd39379708dc14649d4995b00a6094a56fa8c13af53d40b9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T00:15:50.715 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:50 smithi040 podman[87544]: 2024-04-27 00:15:50.519878157 +0000 UTC m=+0.499214871 container attach 2e3967b32f6b598bd39379708dc14649d4995b00a6094a56fa8c13af53d40b9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True) 2024-04-27T00:15:51.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:50 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:50.673+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:51.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:50 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:50.673+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:51.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:50 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:50.673+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:51.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:50 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:50.673+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:51.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:50 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:50.673+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:51.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:50 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:50.673+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:51.208 INFO:journalctl@ceph.osd.9.smithi040.stdout:Apr 27 00:15:50 smithi040 podman[87544]: 2024-04-27 00:15:50.905055499 +0000 UTC m=+0.884392233 container died 2e3967b32f6b598bd39379708dc14649d4995b00a6094a56fa8c13af53d40b9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-27T00:15:51.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:50 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:50.834+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:51.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:50 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:50.834+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:51.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:50 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:50.834+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:51.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:50 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:50.834+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:51.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:50 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:50.834+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:51.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:50 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:50.834+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:51.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:50 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:50.834+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:22.441309+0000 front 2024-04-27T00:15:22.441138+0000 (oldest deadline 2024-04-27T00:15:45.241359+0000) 2024-04-27T00:15:51.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.9.service' 2024-04-27T00:15:51.768 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:15:51.768 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-27T00:15:51.768 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-27T00:15:51.768 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.10 2024-04-27T00:15:52.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:51 smithi080 systemd[1]: Stopping Ceph osd.10 for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:15:52.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:51 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:51.825+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:52.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:51 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:51.825+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:52.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:51 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:51.825+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:52.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:51 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:51.825+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:52.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:51 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:51.825+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:52.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:51 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:51.825+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:52.049 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:51 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:51.825+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:22.441309+0000 front 2024-04-27T00:15:22.441138+0000 (oldest deadline 2024-04-27T00:15:45.241359+0000) 2024-04-27T00:15:52.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:51 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:51.659+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:52.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:51 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:51.659+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:52.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:51 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:51.659+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:52.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:51 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:51.659+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:52.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:51 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:51.659+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:52.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:51 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:51.659+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:52.548 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:52 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:52.097+0000 7f0e08da2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:15:52.549 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:52 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:52.097+0000 7f0e08da2700 -1 osd.10 98 *** Got signal Terminated *** 2024-04-27T00:15:52.549 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:52 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:52.097+0000 7f0e08da2700 -1 osd.10 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:15:52.846 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:52 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:52.671+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:52.846 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:52 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:52.671+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:52.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:52 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:52.671+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:52.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:52 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:52.671+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:52.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:52 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:52.671+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:52.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:52 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:52.671+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:53.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:52 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:52.869+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:53.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:52 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:52.869+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:53.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:52 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:52.869+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:53.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:52 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:52.869+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:53.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:52 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:52.869+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:53.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:52 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:52.869+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:53.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:52 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:52.869+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:22.441309+0000 front 2024-04-27T00:15:22.441138+0000 (oldest deadline 2024-04-27T00:15:45.241359+0000) 2024-04-27T00:15:54.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:53 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:53.625+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:54.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:53 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:53.625+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:54.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:53 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:53.625+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:54.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:53 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:53.625+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:54.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:53 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:53.625+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:54.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:53 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:53.625+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:54.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:53 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:53.900+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:54.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:53 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:53.900+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:54.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:53 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:53.900+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:54.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:53 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:53.900+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:54.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:53 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:53.900+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:54.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:53 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:53.900+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:54.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:53 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:53.900+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:22.441309+0000 front 2024-04-27T00:15:22.441138+0000 (oldest deadline 2024-04-27T00:15:45.241359+0000) 2024-04-27T00:15:55.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:54.667+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:55.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:54.667+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:55.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:54.667+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:55.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:54.667+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:55.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:54.667+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:55.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:54.667+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:55.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:54.667+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6820 osd.7 since back 2024-04-27T00:15:31.100204+0000 front 2024-04-27T00:15:31.100160+0000 (oldest deadline 2024-04-27T00:15:53.999954+0000) 2024-04-27T00:15:55.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:54 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:54.904+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:55.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:54 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:54.904+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:55.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:54 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:54.904+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:55.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:54 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:54.904+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:55.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:54 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:54.904+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:55.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:54 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:54.904+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:55.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:54 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:54.904+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:22.441309+0000 front 2024-04-27T00:15:22.441138+0000 (oldest deadline 2024-04-27T00:15:45.241359+0000) 2024-04-27T00:15:56.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:55 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:55.706+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:56.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:55 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:55.706+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:56.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:55 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:55.706+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:56.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:55 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:55.706+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:56.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:55 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:55.706+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:56.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:55 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:55.706+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:56.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:55 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:55.706+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6820 osd.7 since back 2024-04-27T00:15:31.100204+0000 front 2024-04-27T00:15:31.100160+0000 (oldest deadline 2024-04-27T00:15:53.999954+0000) 2024-04-27T00:15:56.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:55 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:55.887+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:56.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:55 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:55.887+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:56.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:55 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:55.887+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:56.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:55 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:55.887+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:56.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:55 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:55.887+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:56.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:55 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:55.887+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:56.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:55 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:55.887+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:22.441309+0000 front 2024-04-27T00:15:22.441138+0000 (oldest deadline 2024-04-27T00:15:45.241359+0000) 2024-04-27T00:15:57.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:56 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:56.738+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:57.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:56 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:56.738+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:57.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:56 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:56.738+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:57.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:56 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:56.738+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:57.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:56 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:56.738+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:57.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:56 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:56.738+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:57.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:56 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:56.738+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6820 osd.7 since back 2024-04-27T00:15:31.100204+0000 front 2024-04-27T00:15:31.100160+0000 (oldest deadline 2024-04-27T00:15:53.999954+0000) 2024-04-27T00:15:57.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:56 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:56.903+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:31.232954+0000 front 2024-04-27T00:14:31.232912+0000 (oldest deadline 2024-04-27T00:14:53.532390+0000) 2024-04-27T00:15:57.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:56 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:56.903+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:38.734190+0000 front 2024-04-27T00:14:38.734371+0000 (oldest deadline 2024-04-27T00:15:02.234311+0000) 2024-04-27T00:15:57.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:56 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:56.903+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:49.136432+0000 front 2024-04-27T00:14:49.136473+0000 (oldest deadline 2024-04-27T00:15:15.036512+0000) 2024-04-27T00:15:57.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:56 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:56.903+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2024-04-27T00:14:56.737993+0000 front 2024-04-27T00:14:56.737881+0000 (oldest deadline 2024-04-27T00:15:18.437647+0000) 2024-04-27T00:15:57.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:56 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:56.903+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:03.738395+0000 front 2024-04-27T00:15:03.738435+0000 (oldest deadline 2024-04-27T00:15:28.438543+0000) 2024-04-27T00:15:57.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:56 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:56.903+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:10.739764+0000 front 2024-04-27T00:15:10.739709+0000 (oldest deadline 2024-04-27T00:15:35.439487+0000) 2024-04-27T00:15:57.299 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:56 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10[57768]: 2024-04-27T00:15:56.903+0000 7f0e05396700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:22.441309+0000 front 2024-04-27T00:15:22.441138+0000 (oldest deadline 2024-04-27T00:15:45.241359+0000) 2024-04-27T00:15:57.672 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:57 smithi080 podman[68764]: 2024-04-27 00:15:57.396598458 +0000 UTC m=+5.427526305 container died d3742a708cc22691433310c13f6fd482fc8c74063586e3ed9b506b7847931176 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-27T00:15:58.023 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:57 smithi080 podman[68764]: 2024-04-27 00:15:57.771991847 +0000 UTC m=+5.802919683 container cleanup d3742a708cc22691433310c13f6fd482fc8c74063586e3ed9b506b7847931176 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-27T00:15:58.023 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:57 smithi080 bash[68764]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10 2024-04-27T00:15:58.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:57 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:57.786+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:58.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:57 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:57.786+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:58.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:57 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:57.786+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:58.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:57 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:57.786+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:58.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:57 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:57.786+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:58.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:57 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:57.786+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:58.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:57 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:57.786+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6820 osd.7 since back 2024-04-27T00:15:31.100204+0000 front 2024-04-27T00:15:31.100160+0000 (oldest deadline 2024-04-27T00:15:53.999954+0000) 2024-04-27T00:15:58.298 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:58 smithi080 podman[68782]: 2024-04-27 00:15:58.023959786 +0000 UTC m=+0.623665242 container remove d3742a708cc22691433310c13f6fd482fc8c74063586e3ed9b506b7847931176 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:15:58.635 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:58 smithi080 podman[68875]: 2024-04-27 00:15:58.283610799 +0000 UTC m=+0.017936542 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:15:58.635 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:58 smithi080 podman[68875]: 2024-04-27 00:15:58.39975318 +0000 UTC m=+0.134078909 container create 08833cf2a65f185d4edb94afb89a63a12047f3c8ad6ab40e6aefffcce776efd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:15:58.635 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:58 smithi080 podman[68875]: 2024-04-27 00:15:58.635689838 +0000 UTC m=+0.370015567 container init 08833cf2a65f185d4edb94afb89a63a12047f3c8ad6ab40e6aefffcce776efd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-27T00:15:58.931 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:58 smithi080 podman[68875]: 2024-04-27 00:15:58.63879787 +0000 UTC m=+0.373123608 container start 08833cf2a65f185d4edb94afb89a63a12047f3c8ad6ab40e6aefffcce776efd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-27T00:15:58.932 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:58 smithi080 podman[68875]: 2024-04-27 00:15:58.73576159 +0000 UTC m=+0.470087337 container attach 08833cf2a65f185d4edb94afb89a63a12047f3c8ad6ab40e6aefffcce776efd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:15:59.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:58 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:58.752+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:59.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:58 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:58.752+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:59.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:58 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:58.752+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:59.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:58 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:58.752+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:59.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:58 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:58.752+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:59.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:58 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:58.752+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:59.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:58 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:58.752+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6820 osd.7 since back 2024-04-27T00:15:31.100204+0000 front 2024-04-27T00:15:31.100160+0000 (oldest deadline 2024-04-27T00:15:53.999954+0000) 2024-04-27T00:15:59.221 INFO:journalctl@ceph.osd.10.smithi080.stdout:Apr 27 00:15:58 smithi080 podman[68875]: 2024-04-27 00:15:58.932788358 +0000 UTC m=+0.667114094 container died 08833cf2a65f185d4edb94afb89a63a12047f3c8ad6ab40e6aefffcce776efd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-27T00:15:59.519 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.10.service' 2024-04-27T00:15:59.557 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:15:59.558 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-27T00:15:59.558 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-27T00:15:59.558 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.11 2024-04-27T00:15:59.846 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:59 smithi103 systemd[1]: Stopping Ceph osd.11 for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:15:59.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:59 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:59.727+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:15:59.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:59 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:59.727+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:15:59.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:59 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:59.727+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:15:59.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:59 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:59.727+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:15:59.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:59 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:59.727+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:15:59.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:59 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:59.727+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:15:59.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:59 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:59.727+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6820 osd.7 since back 2024-04-27T00:15:31.100204+0000 front 2024-04-27T00:15:31.100160+0000 (oldest deadline 2024-04-27T00:15:53.999954+0000) 2024-04-27T00:16:00.346 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:59 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:59.848+0000 7f644456b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:00.347 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:59 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:59.848+0000 7f644456b700 -1 osd.11 98 *** Got signal Terminated *** 2024-04-27T00:16:00.347 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:15:59 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:15:59.848+0000 7f644456b700 -1 osd.11 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:16:01.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:00 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:00.766+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:16:01.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:00 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:00.766+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:16:01.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:00 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:00.766+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:16:01.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:00 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:00.766+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:16:01.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:00 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:00.766+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:16:01.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:00 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:00.766+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:16:01.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:00 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:00.766+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6820 osd.7 since back 2024-04-27T00:15:31.100204+0000 front 2024-04-27T00:15:31.100160+0000 (oldest deadline 2024-04-27T00:15:53.999954+0000) 2024-04-27T00:16:02.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:01 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:01.721+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:16:02.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:01 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:01.721+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:16:02.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:01 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:01.721+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:16:02.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:01 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:01.721+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:16:02.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:01 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:01.721+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:16:02.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:01 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:01.721+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:16:02.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:01 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:01.721+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6820 osd.7 since back 2024-04-27T00:15:31.100204+0000 front 2024-04-27T00:15:31.100160+0000 (oldest deadline 2024-04-27T00:15:53.999954+0000) 2024-04-27T00:16:02.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:01 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:01.721+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6820 osd.8 since back 2024-04-27T00:15:37.500830+0000 front 2024-04-27T00:15:37.500845+0000 (oldest deadline 2024-04-27T00:16:01.600759+0000) 2024-04-27T00:16:03.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:02 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:02.745+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:16:03.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:02 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:02.745+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:16:03.096 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:02 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:02.745+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:16:03.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:02 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:02.745+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:16:03.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:02 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:02.745+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:16:03.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:02 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:02.745+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:16:03.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:02 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:02.745+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6820 osd.7 since back 2024-04-27T00:15:31.100204+0000 front 2024-04-27T00:15:31.100160+0000 (oldest deadline 2024-04-27T00:15:53.999954+0000) 2024-04-27T00:16:03.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:02 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:02.745+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6820 osd.8 since back 2024-04-27T00:15:37.500830+0000 front 2024-04-27T00:15:37.500845+0000 (oldest deadline 2024-04-27T00:16:01.600759+0000) 2024-04-27T00:16:03.998 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:03 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:03.747+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:16:03.998 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:03 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:03.747+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:16:03.998 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:03 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:03.747+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:16:03.998 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:03 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:03.747+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:16:03.998 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:03 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:03.747+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:16:03.998 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:03 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:03.747+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:16:03.998 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:03 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:03.747+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6820 osd.7 since back 2024-04-27T00:15:31.100204+0000 front 2024-04-27T00:15:31.100160+0000 (oldest deadline 2024-04-27T00:15:53.999954+0000) 2024-04-27T00:16:03.999 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:03 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:03.747+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6820 osd.8 since back 2024-04-27T00:15:37.500830+0000 front 2024-04-27T00:15:37.500845+0000 (oldest deadline 2024-04-27T00:16:01.600759+0000) 2024-04-27T00:16:05.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:04 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:04.736+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-04-27T00:14:30.591164+0000 front 2024-04-27T00:14:30.591248+0000 (oldest deadline 2024-04-27T00:14:54.690745+0000) 2024-04-27T00:16:05.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:04 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:04.736+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6804 osd.1 since back 2024-04-27T00:14:39.991719+0000 front 2024-04-27T00:14:39.991679+0000 (oldest deadline 2024-04-27T00:15:02.891571+0000) 2024-04-27T00:16:05.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:04 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:04.736+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6804 osd.2 since back 2024-04-27T00:14:47.492795+0000 front 2024-04-27T00:14:47.492959+0000 (oldest deadline 2024-04-27T00:15:10.393005+0000) 2024-04-27T00:16:05.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:04 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:04.736+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6812 osd.4 since back 2024-04-27T00:15:05.996222+0000 front 2024-04-27T00:15:05.996122+0000 (oldest deadline 2024-04-27T00:15:27.696162+0000) 2024-04-27T00:16:05.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:04 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:04.736+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6812 osd.5 since back 2024-04-27T00:15:13.597240+0000 front 2024-04-27T00:15:13.597272+0000 (oldest deadline 2024-04-27T00:15:39.497400+0000) 2024-04-27T00:16:05.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:04 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:04.736+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2024-04-27T00:15:19.498090+0000 front 2024-04-27T00:15:19.498074+0000 (oldest deadline 2024-04-27T00:15:44.197716+0000) 2024-04-27T00:16:05.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:04 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:04.736+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.80:6820 osd.7 since back 2024-04-27T00:15:31.100204+0000 front 2024-04-27T00:15:31.100160+0000 (oldest deadline 2024-04-27T00:15:53.999954+0000) 2024-04-27T00:16:05.097 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:04 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11[58034]: 2024-04-27T00:16:04.736+0000 7f6440b5f700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.103:6820 osd.8 since back 2024-04-27T00:15:37.500830+0000 front 2024-04-27T00:15:37.500845+0000 (oldest deadline 2024-04-27T00:16:01.600759+0000) 2024-04-27T00:16:05.466 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:05 smithi103 podman[70032]: 2024-04-27 00:16:05.188067354 +0000 UTC m=+5.443169736 container died e8b53b452967454e19a47a0457cb0c702db40a2c12066820bf3f95a35f3c74a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-27T00:16:05.846 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:05 smithi103 podman[70032]: 2024-04-27 00:16:05.550246364 +0000 UTC m=+5.805348800 container cleanup e8b53b452967454e19a47a0457cb0c702db40a2c12066820bf3f95a35f3c74a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-27T00:16:05.847 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:05 smithi103 bash[70032]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11 2024-04-27T00:16:06.168 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:05 smithi103 podman[70060]: 2024-04-27 00:16:05.848555032 +0000 UTC m=+0.655414270 container remove e8b53b452967454e19a47a0457cb0c702db40a2c12066820bf3f95a35f3c74a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-27T00:16:06.421 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:06 smithi103 podman[70153]: 2024-04-27 00:16:06.168910311 +0000 UTC m=+0.106213398 container create 57c7e18e8a38c6f50ced7ba920684f24003483dc42238319df17e332c60a68ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-27T00:16:06.421 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:06 smithi103 podman[70153]: 2024-04-27 00:16:06.080658586 +0000 UTC m=+0.017961685 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:16:06.421 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:06 smithi103 podman[70153]: 2024-04-27 00:16:06.347060165 +0000 UTC m=+0.284363252 container init 57c7e18e8a38c6f50ced7ba920684f24003483dc42238319df17e332c60a68ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-27T00:16:06.421 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:06 smithi103 podman[70153]: 2024-04-27 00:16:06.35023576 +0000 UTC m=+0.287538851 container start 57c7e18e8a38c6f50ced7ba920684f24003483dc42238319df17e332c60a68ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-27T00:16:06.421 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:06 smithi103 podman[70153]: 2024-04-27 00:16:06.42178112 +0000 UTC m=+0.359084221 container attach 57c7e18e8a38c6f50ced7ba920684f24003483dc42238319df17e332c60a68ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-27T00:16:06.846 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:06 smithi103 podman[70153]: 2024-04-27 00:16:06.647831868 +0000 UTC m=+0.585134963 container died 57c7e18e8a38c6f50ced7ba920684f24003483dc42238319df17e332c60a68ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424) 2024-04-27T00:16:07.191 INFO:journalctl@ceph.osd.11.smithi103.stdout:Apr 27 00:16:07 smithi103 podman[70179]: 2024-04-27 00:16:07.175130583 +0000 UTC m=+0.522209301 container remove 57c7e18e8a38c6f50ced7ba920684f24003483dc42238319df17e332c60a68ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, ceph=True) 2024-04-27T00:16:07.197 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@osd.11.service' 2024-04-27T00:16:07.234 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:07.234 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-27T00:16:07.235 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-27T00:16:07.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.a 2024-04-27T00:16:07.715 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 27 00:16:07 smithi040 systemd[1]: Stopping Ceph mds.a for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:16:08.215 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 27 00:16:07 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-a[67508]: 2024-04-27T00:16:07.856+0000 7fd79d000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:08.215 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 27 00:16:07 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-a[67508]: 2024-04-27T00:16:07.856+0000 7fd79d000700 -1 mds.a *** got signal Terminated *** 2024-04-27T00:16:10.196 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 27 00:16:09 smithi040 podman[87660]: 2024-04-27 00:16:09.809010196 +0000 UTC m=+2.080198282 container died fab18cca5df1502457b49677eb9352f2ab2741f29425a19678cd16ef5ac2269d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T00:16:10.465 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 27 00:16:10 smithi040 podman[87660]: 2024-04-27 00:16:10.307984528 +0000 UTC m=+2.579172631 container cleanup fab18cca5df1502457b49677eb9352f2ab2741f29425a19678cd16ef5ac2269d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-27T00:16:10.465 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 27 00:16:10 smithi040 bash[87660]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-a 2024-04-27T00:16:10.811 INFO:journalctl@ceph.mds.a.smithi040.stdout:Apr 27 00:16:10 smithi040 podman[87674]: 2024-04-27 00:16:10.550759716 +0000 UTC m=+0.725073872 container remove fab18cca5df1502457b49677eb9352f2ab2741f29425a19678cd16ef5ac2269d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-27T00:16:10.852 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.a.service' 2024-04-27T00:16:10.966 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:10.967 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-27T00:16:10.967 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-27T00:16:10.967 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.d 2024-04-27T00:16:11.465 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 27 00:16:11 smithi040 systemd[1]: Stopping Ceph mds.d for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:16:11.965 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 27 00:16:11 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-d[66480]: 2024-04-27T00:16:11.533+0000 7f42bc800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:11.965 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 27 00:16:11 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-d[66480]: 2024-04-27T00:16:11.533+0000 7f42bc800700 -1 mds.d *** got signal Terminated *** 2024-04-27T00:16:15.167 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 27 00:16:14 smithi040 podman[87779]: 2024-04-27 00:16:14.875947995 +0000 UTC m=+3.452176635 container died 885a4a0934399dea542a5acda7365c7a6cc528e0aba4ca0ea0c36e181ac9a70c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:16:15.465 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 27 00:16:15 smithi040 podman[87779]: 2024-04-27 00:16:15.286511284 +0000 UTC m=+3.862739951 container cleanup 885a4a0934399dea542a5acda7365c7a6cc528e0aba4ca0ea0c36e181ac9a70c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:16:15.465 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 27 00:16:15 smithi040 bash[87779]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-d 2024-04-27T00:16:15.965 INFO:journalctl@ceph.mds.d.smithi040.stdout:Apr 27 00:16:15 smithi040 podman[87793]: 2024-04-27 00:16:15.470679573 +0000 UTC m=+0.574314208 container remove 885a4a0934399dea542a5acda7365c7a6cc528e0aba4ca0ea0c36e181ac9a70c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-27T00:16:16.032 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.d.service' 2024-04-27T00:16:16.145 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:16.145 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-27T00:16:16.145 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-27T00:16:16.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.g 2024-04-27T00:16:16.716 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 27 00:16:16 smithi040 systemd[1]: Stopping Ceph mds.g for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:16:17.215 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 27 00:16:16 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g[67124]: 2024-04-27T00:16:16.737+0000 7f899e200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:17.215 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 27 00:16:16 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g[67124]: 2024-04-27T00:16:16.737+0000 7f899e200700 -1 mds.g *** got signal Terminated *** 2024-04-27T00:16:21.029 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 27 00:16:20 smithi040 podman[87898]: 2024-04-27 00:16:20.7550257 +0000 UTC m=+4.152776929 container died 337ff61668b7e1ef1dab8d55472fe9aa38786d645b937886c1fcd361467eb3b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-27T00:16:21.307 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 27 00:16:21 smithi040 podman[87898]: 2024-04-27 00:16:21.139782472 +0000 UTC m=+4.537533697 container cleanup 337ff61668b7e1ef1dab8d55472fe9aa38786d645b937886c1fcd361467eb3b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:16:21.307 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 27 00:16:21 smithi040 bash[87898]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g 2024-04-27T00:16:21.717 INFO:journalctl@ceph.mds.g.smithi040.stdout:Apr 27 00:16:21 smithi040 podman[87912]: 2024-04-27 00:16:21.307392763 +0000 UTC m=+0.532284191 container remove 337ff61668b7e1ef1dab8d55472fe9aa38786d645b937886c1fcd361467eb3b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-27T00:16:21.816 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.g.service' 2024-04-27T00:16:21.929 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:21.929 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-27T00:16:21.930 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-27T00:16:21.930 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.j 2024-04-27T00:16:22.215 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 27 00:16:22 smithi040 systemd[1]: Stopping Ceph mds.j for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:16:22.965 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 27 00:16:22 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-j[66137]: 2024-04-27T00:16:22.482+0000 7f84fc200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:22.965 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 27 00:16:22 smithi040 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-j[66137]: 2024-04-27T00:16:22.482+0000 7f84fc200700 -1 mds.j *** got signal Terminated *** 2024-04-27T00:16:26.799 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 27 00:16:26 smithi040 podman[88019]: 2024-04-27 00:16:26.509937131 +0000 UTC m=+4.149728300 container died 9261d7109e4aa434376070eae0cd36bf828d9ceda3d89ca863be57389340157e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-27T00:16:27.077 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 27 00:16:26 smithi040 podman[88019]: 2024-04-27 00:16:26.909895899 +0000 UTC m=+4.549687078 container cleanup 9261d7109e4aa434376070eae0cd36bf828d9ceda3d89ca863be57389340157e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-27T00:16:27.077 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 27 00:16:26 smithi040 bash[88019]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-j 2024-04-27T00:16:27.465 INFO:journalctl@ceph.mds.j.smithi040.stdout:Apr 27 00:16:27 smithi040 podman[88044]: 2024-04-27 00:16:27.077419508 +0000 UTC m=+0.554236079 container remove 9261d7109e4aa434376070eae0cd36bf828d9ceda3d89ca863be57389340157e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:16:27.537 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.j.service' 2024-04-27T00:16:27.641 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:27.642 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-27T00:16:27.642 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-27T00:16:27.642 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.b 2024-04-27T00:16:27.971 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 27 00:16:27 smithi080 systemd[1]: Stopping Ceph mds.b for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:16:28.298 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 27 00:16:27 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b[63150]: 2024-04-27T00:16:27.971+0000 7fe7492ae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:28.298 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 27 00:16:27 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b[63150]: 2024-04-27T00:16:27.971+0000 7fe7492ae700 -1 mds.b *** got signal Terminated *** 2024-04-27T00:16:29.296 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 27 00:16:29 smithi080 podman[68981]: 2024-04-27 00:16:29.009915967 +0000 UTC m=+1.173136312 container died cb060557aa61d7c2dcc70a3c8eda64c5f9cb595091f65b9358c1805a682d6737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True) 2024-04-27T00:16:29.548 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 27 00:16:29 smithi080 podman[68981]: 2024-04-27 00:16:29.380187969 +0000 UTC m=+1.543408309 container cleanup cb060557aa61d7c2dcc70a3c8eda64c5f9cb595091f65b9358c1805a682d6737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-27T00:16:29.548 INFO:journalctl@ceph.mds.b.smithi080.stdout:Apr 27 00:16:29 smithi080 bash[68981]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-b 2024-04-27T00:16:29.799 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.b.service' 2024-04-27T00:16:29.835 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:29.835 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-27T00:16:29.835 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-27T00:16:29.835 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.e 2024-04-27T00:16:30.048 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 27 00:16:29 smithi080 systemd[1]: Stopping Ceph mds.e for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:16:30.548 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 27 00:16:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-e[62519]: 2024-04-27T00:16:30.142+0000 7f1ede541700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:30.548 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 27 00:16:30 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-e[62519]: 2024-04-27T00:16:30.142+0000 7f1ede541700 -1 mds.e *** got signal Terminated *** 2024-04-27T00:16:34.034 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 27 00:16:33 smithi080 podman[69113]: 2024-04-27 00:16:33.777456485 +0000 UTC m=+3.725831448 container died 908d3b1b27f62ce0f95f6b2d869147527a92190da66ad9da56692f24dd6756bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-27T00:16:34.298 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 27 00:16:34 smithi080 podman[69113]: 2024-04-27 00:16:34.125920624 +0000 UTC m=+4.074295587 container cleanup 908d3b1b27f62ce0f95f6b2d869147527a92190da66ad9da56692f24dd6756bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-27T00:16:34.298 INFO:journalctl@ceph.mds.e.smithi080.stdout:Apr 27 00:16:34 smithi080 bash[69113]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-e 2024-04-27T00:16:34.605 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.e.service' 2024-04-27T00:16:34.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:34.640 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-27T00:16:34.640 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-27T00:16:34.640 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.h 2024-04-27T00:16:34.954 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 27 00:16:34 smithi080 systemd[1]: Stopping Ceph mds.h for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:16:35.298 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 27 00:16:34 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-h[62828]: 2024-04-27T00:16:34.954+0000 7f301417c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:35.298 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 27 00:16:34 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-h[62828]: 2024-04-27T00:16:34.954+0000 7f301417c700 -1 mds.h *** got signal Terminated *** 2024-04-27T00:16:37.301 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 27 00:16:37 smithi080 podman[69233]: 2024-04-27 00:16:37.008224125 +0000 UTC m=+2.151603046 container died 19f051a4dcd9d84cbd4584cec0f3bba323db2f778bd2de9dbf001de18705e7f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:16:37.746 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 27 00:16:37 smithi080 podman[69233]: 2024-04-27 00:16:37.463207194 +0000 UTC m=+2.606586103 container cleanup 19f051a4dcd9d84cbd4584cec0f3bba323db2f778bd2de9dbf001de18705e7f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True) 2024-04-27T00:16:37.747 INFO:journalctl@ceph.mds.h.smithi080.stdout:Apr 27 00:16:37 smithi080 bash[69233]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-h 2024-04-27T00:16:37.883 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.h.service' 2024-04-27T00:16:37.914 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:37.914 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-27T00:16:37.914 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-27T00:16:37.914 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.k 2024-04-27T00:16:38.048 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 27 00:16:37 smithi080 systemd[1]: Stopping Ceph mds.k for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:16:38.548 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 27 00:16:38 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-k[62213]: 2024-04-27T00:16:38.233+0000 7f28afa71700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:38.548 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 27 00:16:38 smithi080 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-k[62213]: 2024-04-27T00:16:38.233+0000 7f28afa71700 -1 mds.k *** got signal Terminated *** 2024-04-27T00:16:42.548 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 27 00:16:42 smithi080 podman[69352]: 2024-04-27 00:16:42.241434418 +0000 UTC m=+4.116984153 container died bf6655a67f9f1e8ffea16a5a78106e5090e36eed1d73cfd5d354b40d7db92077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-27T00:16:42.935 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 27 00:16:42 smithi080 podman[69352]: 2024-04-27 00:16:42.566721217 +0000 UTC m=+4.442270936 container cleanup bf6655a67f9f1e8ffea16a5a78106e5090e36eed1d73cfd5d354b40d7db92077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-27T00:16:42.935 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 27 00:16:42 smithi080 bash[69352]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-k 2024-04-27T00:16:42.935 INFO:journalctl@ceph.mds.k.smithi080.stdout:Apr 27 00:16:42 smithi080 podman[69366]: 2024-04-27 00:16:42.734607328 +0000 UTC m=+0.486842537 container remove bf6655a67f9f1e8ffea16a5a78106e5090e36eed1d73cfd5d354b40d7db92077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-27T00:16:42.952 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.k.service' 2024-04-27T00:16:42.982 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:42.983 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-27T00:16:42.983 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-27T00:16:42.983 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.c 2024-04-27T00:16:43.284 INFO:journalctl@ceph.mds.c.smithi103.stdout:Apr 27 00:16:43 smithi103 systemd[1]: Stopping Ceph mds.c for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:16:43.596 INFO:journalctl@ceph.mds.c.smithi103.stdout:Apr 27 00:16:43 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-c[63472]: 2024-04-27T00:16:43.283+0000 7f6012a57700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:43.596 INFO:journalctl@ceph.mds.c.smithi103.stdout:Apr 27 00:16:43 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-c[63472]: 2024-04-27T00:16:43.283+0000 7f6012a57700 -1 mds.c *** got signal Terminated *** 2024-04-27T00:16:45.276 INFO:journalctl@ceph.mds.c.smithi103.stdout:Apr 27 00:16:45 smithi103 podman[70258]: 2024-04-27 00:16:45.00827359 +0000 UTC m=+1.834744890 container died 6356b71257033b9e6e2e1b28f2e31f30ebcbd06afcbd666b92f5652cc32f3b12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:16:45.596 INFO:journalctl@ceph.mds.c.smithi103.stdout:Apr 27 00:16:45 smithi103 podman[70258]: 2024-04-27 00:16:45.384471413 +0000 UTC m=+2.210942722 container cleanup 6356b71257033b9e6e2e1b28f2e31f30ebcbd06afcbd666b92f5652cc32f3b12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:16:45.596 INFO:journalctl@ceph.mds.c.smithi103.stdout:Apr 27 00:16:45 smithi103 bash[70258]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-c 2024-04-27T00:16:45.803 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.c.service' 2024-04-27T00:16:45.838 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:45.838 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-27T00:16:45.838 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-27T00:16:45.839 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.f 2024-04-27T00:16:46.347 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 27 00:16:45 smithi103 systemd[1]: Stopping Ceph mds.f for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:16:46.347 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 27 00:16:46 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-f[63162]: 2024-04-27T00:16:46.146+0000 7f9e095c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:46.347 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 27 00:16:46 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-f[63162]: 2024-04-27T00:16:46.146+0000 7f9e095c1700 -1 mds.f *** got signal Terminated *** 2024-04-27T00:16:49.296 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 27 00:16:49 smithi103 podman[70391]: 2024-04-27 00:16:49.015795342 +0000 UTC m=+2.959255693 container died 642aacff1d54ce4ff7f4a3671bd0faaebd4f73891e59b69b9354e149fabc53bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-f, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-27T00:16:49.296 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 27 00:16:49 smithi103 podman[70391]: 2024-04-27 00:16:49.297159923 +0000 UTC m=+3.240620264 container cleanup 642aacff1d54ce4ff7f4a3671bd0faaebd4f73891e59b69b9354e149fabc53bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-27T00:16:49.564 INFO:journalctl@ceph.mds.f.smithi103.stdout:Apr 27 00:16:49 smithi103 bash[70391]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-f 2024-04-27T00:16:49.720 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.f.service' 2024-04-27T00:16:49.752 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:49.753 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-27T00:16:49.753 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-27T00:16:49.753 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.i 2024-04-27T00:16:50.096 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 27 00:16:49 smithi103 systemd[1]: Stopping Ceph mds.i for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:16:50.096 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 27 00:16:50 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i[63786]: 2024-04-27T00:16:50.058+0000 7f41e328e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:50.097 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 27 00:16:50 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i[63786]: 2024-04-27T00:16:50.058+0000 7f41e328e700 -1 mds.i *** got signal Terminated *** 2024-04-27T00:16:53.342 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 27 00:16:53 smithi103 podman[70512]: 2024-04-27 00:16:53.021850649 +0000 UTC m=+3.054090118 container died ee1cb2f0787cdc4c8d08be88788e287a02ef4f3cfe438cd2152c6b6ad7c61d0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.license=GPLv2) 2024-04-27T00:16:53.342 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 27 00:16:53 smithi103 podman[70512]: 2024-04-27 00:16:53.342730708 +0000 UTC m=+3.374970183 container cleanup ee1cb2f0787cdc4c8d08be88788e287a02ef4f3cfe438cd2152c6b6ad7c61d0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:16:53.596 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 27 00:16:53 smithi103 bash[70512]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i 2024-04-27T00:16:53.596 INFO:journalctl@ceph.mds.i.smithi103.stdout:Apr 27 00:16:53 smithi103 podman[70526]: 2024-04-27 00:16:53.561528959 +0000 UTC m=+0.541017803 container remove ee1cb2f0787cdc4c8d08be88788e287a02ef4f3cfe438cd2152c6b6ad7c61d0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-27T00:16:53.774 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.i.service' 2024-04-27T00:16:53.806 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:53.806 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-27T00:16:53.806 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-27T00:16:53.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.l 2024-04-27T00:16:54.096 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 27 00:16:53 smithi103 systemd[1]: Stopping Ceph mds.l for 78cff6ea-0427-11ef-bc93-c7b262605968... 2024-04-27T00:16:54.596 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 27 00:16:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-l[62850]: 2024-04-27T00:16:54.121+0000 7f8bb529c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:16:54.596 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 27 00:16:54 smithi103 ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-l[62850]: 2024-04-27T00:16:54.121+0000 7f8bb529c700 -1 mds.l *** got signal Terminated *** 2024-04-27T00:16:56.143 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 27 00:16:55 smithi103 podman[70633]: 2024-04-27 00:16:55.883438841 +0000 UTC m=+1.866208179 container died 646bac59a43542dcc10839dba24ca592b95be584b5cc32daf80f11d852ca7aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:16:56.490 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 27 00:16:56 smithi103 podman[70633]: 2024-04-27 00:16:56.239708775 +0000 UTC m=+2.222478102 container cleanup 646bac59a43542dcc10839dba24ca592b95be584b5cc32daf80f11d852ca7aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS) 2024-04-27T00:16:56.490 INFO:journalctl@ceph.mds.l.smithi103.stdout:Apr 27 00:16:56 smithi103 bash[70633]: ceph-78cff6ea-0427-11ef-bc93-c7b262605968-mds-l 2024-04-27T00:16:56.627 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-78cff6ea-0427-11ef-bc93-c7b262605968@mds.l.service' 2024-04-27T00:16:56.658 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:16:56.659 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-27T00:16:56.659 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-27T00:16:56.991 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-27T00:17:02.212 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-27T00:17:02.402 INFO:teuthology.orchestra.run.smithi080.stdout:Deleting cluster with fsid: 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-27T00:17:04.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-27T00:17:05.057 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-27T00:17:07.350 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:17:07.448 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:17:07.480 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:17:07.506 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-27T00:17:07.508 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327/remote/smithi040/crash 2024-04-27T00:17:07.509 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/crash -- . 2024-04-27T00:17:07.607 INFO:teuthology.orchestra.run.smithi040.stderr:tar: /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-27T00:17:07.607 INFO:teuthology.orchestra.run.smithi040.stderr:tar: Error is not recoverable: exiting now 2024-04-27T00:17:07.612 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327/remote/smithi080/crash 2024-04-27T00:17:07.613 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/crash -- . 2024-04-27T00:17:07.717 INFO:teuthology.orchestra.run.smithi080.stderr:tar: /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-27T00:17:07.717 INFO:teuthology.orchestra.run.smithi080.stderr:tar: Error is not recoverable: exiting now 2024-04-27T00:17:07.719 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327/remote/smithi103/crash 2024-04-27T00:17:07.720 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/crash -- . 2024-04-27T00:17:07.779 INFO:teuthology.orchestra.run.smithi103.stderr:tar: /var/lib/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-27T00:17:07.780 INFO:teuthology.orchestra.run.smithi103.stderr:tar: Error is not recoverable: exiting now 2024-04-27T00:17:07.781 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-27T00:17:07.782 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-27T00:17:07.890 INFO:tasks.cephadm:Compressing logs... 2024-04-27T00:17:07.890 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T00:17:07.895 DEBUG:teuthology.orchestra.run.smithi080:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T00:17:07.898 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T00:17:07.922 INFO:teuthology.orchestra.run.smithi103.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T00:17:07.922 INFO:teuthology.orchestra.run.smithi103.stderr:: No such file or directory 2024-04-27T00:17:07.922 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.log 2024-04-27T00:17:07.922 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5/var/log/ceph/cephadm.log: --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-27T00:17:07.922 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-27T00:17:07.922 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T00:17:07.922 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-27T00:17:07.928 INFO:teuthology.orchestra.run.smithi103.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T00:17:07.928 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-27T00:17:07.929 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-27T00:17:07.929 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-27T00:17:07.929 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mon.c.log: 87.8%/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mgr.z.log: -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T00:17:07.929 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-27T00:17:07.929 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T00:17:07.929 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.3% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T00:17:07.931 INFO:teuthology.orchestra.run.smithi080.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T00:17:07.931 INFO:teuthology.orchestra.run.smithi080.stderr:: No such file or directory 2024-04-27T00:17:07.931 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.log 2024-04-27T00:17:07.931 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T00:17:07.931 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-27T00:17:07.931 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-27T00:17:07.932 INFO:teuthology.orchestra.run.smithi080.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T00:17:07.932 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-27T00:17:07.932 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.7.log: /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-27T00:17:07.932 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-27T00:17:07.932 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-27T00:17:07.932 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-27T00:17:07.932 INFO:teuthology.orchestra.run.smithi080.stderr: 90.3% 87.8% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T00:17:07.932 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.h.log: -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T00:17:07.932 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T00:17:07.932 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-27T00:17:07.932 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.2% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T00:17:07.932 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T00:17:07.934 INFO:teuthology.orchestra.run.smithi103.stderr: 93.1% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-27T00:17:07.935 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-27T00:17:07.935 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-27T00:17:07.935 INFO:teuthology.orchestra.run.smithi103.stderr: 82.2% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T00:17:07.936 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.e.log: 93.0% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-27T00:17:07.936 INFO:teuthology.orchestra.run.smithi080.stderr: 94.5% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-27T00:17:07.936 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-27T00:17:07.936 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-27T00:17:07.945 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T00:17:07.953 INFO:teuthology.orchestra.run.smithi040.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-04-27T00:17:07.954 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/cephadm.log 2024-04-27T00:17:07.954 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-27T00:17:07.955 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.log 2024-04-27T00:17:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.log: -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T00:17:07.957 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.k.log: 91.0% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T00:17:07.957 INFO:teuthology.orchestra.run.smithi080.stderr: 94.5% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-27T00:17:07.958 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-27T00:17:07.958 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-27T00:17:07.959 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-27T00:17:07.959 INFO:teuthology.orchestra.run.smithi040.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-27T00:17:07.960 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.g.log: 87.7% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T00:17:07.960 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-27T00:17:07.961 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-27T00:17:07.961 INFO:teuthology.orchestra.run.smithi040.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T00:17:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-27T00:17:07.963 INFO:teuthology.orchestra.run.smithi040.stderr: 90.2% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T00:17:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-27T00:17:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-27T00:17:07.967 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T00:17:07.969 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.6.log: 94.5% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-27T00:17:07.969 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.cephadm.log: /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-27T00:17:07.970 INFO:teuthology.orchestra.run.smithi040.stderr: 82.6% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T00:17:07.974 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T00:17:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-27T00:17:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-27T00:17:07.980 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.l.log: 94.6% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-27T00:17:07.984 INFO:teuthology.orchestra.run.smithi103.stderr: 91.0% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T00:17:07.984 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.5% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-27T00:17:07.988 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.3% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-27T00:17:07.993 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T00:17:07.993 INFO:teuthology.orchestra.run.smithi103.stderr: 94.5% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-27T00:17:07.994 INFO:teuthology.orchestra.run.smithi080.stderr: 94.5% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-27T00:17:08.008 INFO:teuthology.orchestra.run.smithi103.stderr: 93.0% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-27T00:17:08.055 INFO:teuthology.orchestra.run.smithi040.stderr: 93.2% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-27T00:17:08.213 INFO:teuthology.orchestra.run.smithi103.stderr: 94.2% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-27T00:17:08.316 INFO:teuthology.orchestra.run.smithi040.stderr: 90.4% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-27T00:17:08.367 INFO:teuthology.orchestra.run.smithi103.stderr: 91.8% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-27T00:17:08.459 INFO:teuthology.orchestra.run.smithi080.stderr: 91.9% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-27T00:17:08.667 INFO:teuthology.orchestra.run.smithi103.stderr: 92.8% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-27T00:17:08.929 INFO:teuthology.orchestra.run.smithi080.stderr: 92.2% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-27T00:17:08.935 INFO:teuthology.orchestra.run.smithi040.stderr: 90.4% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-27T00:17:09.330 INFO:teuthology.orchestra.run.smithi080.stderr: 93.3% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-27T00:17:09.342 INFO:teuthology.orchestra.run.smithi040.stderr: 93.2% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-27T00:17:09.389 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-27T00:17:09.442 INFO:teuthology.orchestra.run.smithi103.stderr: 91.6% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-27T00:17:09.487 INFO:teuthology.orchestra.run.smithi080.stderr: 91.5% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-27T00:17:09.547 INFO:teuthology.orchestra.run.smithi080.stderr: 91.7% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-27T00:17:09.583 INFO:teuthology.orchestra.run.smithi040.stderr: 93.3% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-27T00:17:09.621 INFO:teuthology.orchestra.run.smithi040.stderr: 92.7% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-27T00:17:09.912 INFO:teuthology.orchestra.run.smithi103.stderr: 92.6% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-27T00:17:09.914 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-27T00:17:09.914 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m2.002s 2024-04-27T00:17:09.914 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m6.379s 2024-04-27T00:17:09.914 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m0.227s 2024-04-27T00:17:09.915 INFO:teuthology.orchestra.run.smithi080.stderr: 92.8% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-27T00:17:09.917 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-04-27T00:17:09.917 INFO:teuthology.orchestra.run.smithi080.stderr:real 0m2.005s 2024-04-27T00:17:09.917 INFO:teuthology.orchestra.run.smithi080.stderr:user 0m7.976s 2024-04-27T00:17:09.917 INFO:teuthology.orchestra.run.smithi080.stderr:sys 0m0.251s 2024-04-27T00:17:10.242 INFO:teuthology.orchestra.run.smithi040.stderr: 92.4% -- replaced with /var/log/ceph/78cff6ea-0427-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-27T00:17:10.246 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-04-27T00:17:10.246 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m2.318s 2024-04-27T00:17:10.246 INFO:teuthology.orchestra.run.smithi040.stderr:user 0m7.790s 2024-04-27T00:17:10.246 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m0.679s 2024-04-27T00:17:10.247 INFO:tasks.cephadm:Archiving logs... 2024-04-27T00:17:10.247 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327/remote/smithi040/log 2024-04-27T00:17:10.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T00:17:11.487 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327/remote/smithi080/log 2024-04-27T00:17:11.488 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T00:17:12.694 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327/remote/smithi103/log 2024-04-27T00:17:12.695 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T00:17:13.668 INFO:tasks.cephadm:Removing cluster... 2024-04-27T00:17:13.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 --force 2024-04-27T00:17:13.999 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-27T00:17:14.587 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 --force 2024-04-27T00:17:14.771 INFO:teuthology.orchestra.run.smithi080.stdout:Deleting cluster with fsid: 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-27T00:17:15.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 78cff6ea-0427-11ef-bc93-c7b262605968 --force 2024-04-27T00:17:15.223 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 78cff6ea-0427-11ef-bc93-c7b262605968 2024-04-27T00:17:15.482 INFO:tasks.cephadm:Removing cephadm ... 2024-04-27T00:17:15.483 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-27T00:17:15.544 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-27T00:17:15.560 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-27T00:17:15.575 INFO:tasks.cephadm:Teardown complete 2024-04-27T00:17:15.576 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-27T00:17:15.588 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-27T00:17:15.588 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T00:17:15.596 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T00:17:15.603 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T00:17:15.746 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-27T00:17:15.758 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-27T00:17:15.758 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T00:17:15.766 DEBUG:teuthology.orchestra.run.smithi080:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T00:17:15.768 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T00:17:15.780 INFO:teuthology.orchestra.run.smithi080.stderr:bash: line 1: ntpq: command not found 2024-04-27T00:17:15.782 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-04-27T00:17:15.784 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T00:17:15.784 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2024-04-27T00:17:15.784 INFO:teuthology.orchestra.run.smithi080.stdout:^* hv01.front.sepia.ceph.com 3 7 377 50 +1291us[+1360us] +/- 38ms 2024-04-27T00:17:15.784 INFO:teuthology.orchestra.run.smithi080.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 49 -814us[ -814us] +/- 57ms 2024-04-27T00:17:15.784 INFO:teuthology.orchestra.run.smithi080.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 -1075us[-1075us] +/- 67ms 2024-04-27T00:17:15.784 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:17:15.786 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T00:17:15.786 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-04-27T00:17:15.786 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv01.front.sepia.ceph.com 3 6 377 47 +1241us[+1249us] +/- 38ms 2024-04-27T00:17:15.786 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 53 -820us[ -812us] +/- 57ms 2024-04-27T00:17:15.786 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -1085us[-1077us] +/- 66ms 2024-04-27T00:17:15.786 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:17:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-04-27T00:17:15.828 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T00:17:15.828 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-04-27T00:17:15.828 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv01.front.sepia.ceph.com 3 6 377 50 +1245us[+1242us] +/- 38ms 2024-04-27T00:17:15.829 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 50 -827us[ -830us] +/- 57ms 2024-04-27T00:17:15.829 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -1098us[-1098us] +/- 67ms 2024-04-27T00:17:15.829 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:17:15.831 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-27T00:17:15.843 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-27T00:17:15.843 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-27T00:17:15.855 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T00:17:16.074 DEBUG:teuthology.orchestra.run.smithi080:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T00:17:16.138 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T00:17:16.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T00:17:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:17:16.296 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T00:17:16.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:17:16.323 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T00:17:16.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:17:16.349 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-27T00:17:16.360 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-27T00:17:16.371 INFO:teuthology.task.internal:Duration was 2183.229024 seconds 2024-04-27T00:17:16.371 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-27T00:17:16.383 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-27T00:17:16.383 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T00:17:16.391 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T00:17:16.394 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T00:17:16.428 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T00:17:16.430 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T00:17:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T00:17:16.884 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-27T00:17:16.884 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-04-27T00:17:16.885 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T00:17:16.964 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi080.front.sepia.ceph.com 2024-04-27T00:17:16.964 DEBUG:teuthology.orchestra.run.smithi080:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T00:17:16.985 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-04-27T00:17:16.985 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T00:17:17.004 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-27T00:17:17.004 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T00:17:17.012 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T00:17:17.028 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T00:17:17.096 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-27T00:17:17.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T00:17:17.101 DEBUG:teuthology.orchestra.run.smithi080:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T00:17:17.103 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T00:17:17.608 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-27T00:17:17.619 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-27T00:17:17.619 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T00:17:17.697 DEBUG:teuthology.orchestra.run.smithi080:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T00:17:17.721 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T00:17:17.746 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-27T00:17:17.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T00:17:17.766 DEBUG:teuthology.orchestra.run.smithi080:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T00:17:17.768 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T00:17:17.788 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = core 2024-04-27T00:17:17.809 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-04-27T00:17:17.863 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-04-27T00:17:17.908 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T00:17:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:17:17.958 DEBUG:teuthology.orchestra.run.smithi080:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T00:17:17.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:17:17.972 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T00:17:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:17:17.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-27T00:17:18.000 INFO:teuthology.task.internal:Transferring archived files... 2024-04-27T00:17:18.001 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327/remote/smithi040 2024-04-27T00:17:18.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T00:17:18.102 DEBUG:teuthology.misc:Transferring archived files from smithi080:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327/remote/smithi080 2024-04-27T00:17:18.102 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T00:17:18.135 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675327/remote/smithi103 2024-04-27T00:17:18.136 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T00:17:18.169 INFO:teuthology.task.internal:Removing archive directory... 2024-04-27T00:17:18.169 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T00:17:18.177 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T00:17:18.179 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T00:17:18.230 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-27T00:17:18.244 INFO:teuthology.task.internal:Not uploading archives. 2024-04-27T00:17:18.245 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-27T00:17:18.283 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-27T00:17:18.283 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T00:17:18.289 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T00:17:18.292 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T00:17:18.303 INFO:teuthology.orchestra.run.smithi080.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 00:17 /home/ubuntu/cephtest 2024-04-27T00:17:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 00:17 /home/ubuntu/cephtest 2024-04-27T00:17:18.338 INFO:teuthology.orchestra.run.smithi040.stdout: 393251 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 00:17 /home/ubuntu/cephtest 2024-04-27T00:17:18.341 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-27T00:17:18.354 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-27T00:17:18.382 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} duration: 2183.229024171829 flavor: default owner: scheduled_rishabh@teuthology success: true 2024-04-27T00:17:18.382 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T00:17:18.459 INFO:teuthology.run:pass